Deprecate raw images #227

AbdelrahmanElawady · 2024-04-24T11:14:15Z

Description

Our full-vm images are currently done as a raw images (single image.raw file). It's been deprecated for a while and now it's removed from zos documentation. We should move to normal flist with kernel as part of it as mentioned here.

Ubuntu 22 & 24 image #237

xmonader · 2024-04-24T11:15:43Z

@sabrinasadik core images like ubuntu need to get updated please give it priority

AbdelrahmanElawady · 2024-04-24T11:19:34Z

Also, I should mention that we need virtiofs module to be part of the kernel in new images and especially loaded with initramfs.

muhamadazmy · 2024-04-24T11:23:44Z

The procedure in documentation can be easily followed to build flists for all ubuntu flavours. Similar steps can be followed to also build arch images and others.

PeterNashaat · 2024-04-24T13:14:42Z

Getting error with new jammy image
Step i followed to create the image

mkdir ubuntu-jammy
sudo debootstrap jammy ubuntu-jammy  http://archive.ubuntu.com/ubuntu
sudo mount -o bind /proc ubuntu-jammy/proc
sudo mount -o bind /sys ubuntu-jammy/sys
sudo mount -o bind /dev ubuntu-jammy/dev
sudo mount -o bind /run ubuntu-jammy/run
chroot ubuntu-jammy/
export PATH=/usr/local/sbin/:/usr/local/bin/:/usr/sbin/:/usr/bin/:/sbin:/bin
rm /etc/resolv.conf
echo 'nameserver 1.1.1.1' > /etc/resolv.conf
apt-get update
apt-get install cloud-init openssh-server curl
cloud-init clean
apt-get install linux-modules-extra-5.15.0-25-generic
echo 'fs-virtiofs' >> /etc/initramfs-tools/modules
update-initramfs -c -k all
apt-get clean
sudo umount ubuntu-jammy/run
sudo umount -l ubuntu-jammy/dev
sudo umount ubuntu-jammy/sys
sudo umount ubuntu-jammy/proc
rm -rf ubuntu-jammy/dev/*
tar -czf ubuntu-jammy.tar.gz -C  ubuntu-jammy .

/boot directory

root@imagecreator:~/tf-images/tfgrid3# ls -la ubuntu-jammy/boot/
total 57612
drwxr-xr-x  3 root root     4096 Apr 24 12:46 .
drwxr-xr-x 17 root root     4096 Apr 24 12:42 ..
-rw-------  1 root root  6246119 Mar 30  2022 System.map-5.15.0-25-generic
-rw-r--r--  1 root root   260489 Mar 30  2022 config-5.15.0-25-generic
drwxr-xr-x  2 root root     4096 Apr 24 12:45 grub
lrwxrwxrwx  1 root root       28 Apr 24 12:45 initrd.img -> initrd.img-5.15.0-25-generic
-rw-r--r--  1 root root 41394934 Apr 24 12:46 initrd.img-5.15.0-25-generic
lrwxrwxrwx  1 root root       28 Apr 24 12:45 initrd.img.old -> initrd.img-5.15.0-25-generic
lrwxrwxrwx  1 root root       25 Apr 24 12:45 vmlinuz -> vmlinuz-5.15.0-25-generic
-rw-------  1 root root 11073600 Mar 30  2022 vmlinuz-5.15.0-25-generic
lrwxrwxrwx  1 root root       25 Apr 24 12:45 vmlinuz.old -> vmlinuz-5.15.0-25-generic

Link to flist

PeterNashaat · 2024-04-24T14:45:06Z

Used micro VM and tried installing new vm with the flist again
- Getting this error, tried it on multiple nodes man times and got the same error

WorkloadDeployError: Failed to deploy zmachine with name test12 on node 13 due to: cloud-hypervisor: 44.92169ms: <payload_loader> WARN:vmm/src/vm.rs:1011 -- Loading of legacy (non-PVH) firmware is deprecated and will be removed in a future version. Error booting VM: VmBoot(FirmwareTooLarge) : failed to spawn vm machine process '604-407362-test12'.

AbdelrahmanElawady · 2024-04-24T15:10:26Z

This works with me on devnet:

Can you share vm configuration?

PeterNashaat · 2024-04-24T15:12:42Z

PeterNashaat · 2024-04-24T15:14:25Z

Getting the same error on testnet

AbdelrahmanElawady · 2024-04-24T15:35:33Z

I think it's because the kernel decompression feature didn't reach other networks yet, if you tried readelf -n <vmlinuz> it should return an error.

you can extract the kernel by doing something similar to this:

curl -O https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux
chmod +x extract-vmlinux

sudo ./extract-vmlinux ubuntu-jammy/boot/vmlinuz | sudo tee ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf > /dev/null
# then replace original kernel
sudo mv ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic

PeterNashaat · 2024-04-28T11:30:44Z

Ubuntu 22 flist created and tested
Kernel 5.15.0-25-generic is not available in the package repository for Ubuntu versions 18 and 20
- in arch-chroot env :
  ubuntu 18

root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic

ubuntu 20

root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

AbdelrahmanElawady · 2024-05-02T14:21:29Z

I believe kernel 5.15 was specified as an example, so other versions can be used too.

PeterNashaat · 2024-05-07T12:26:07Z

Ubuntu 24 flist
Tested

PeterNashaat · 2024-05-07T12:28:23Z

Ubuntu 18 flist created but can's ssh to vms, investigating

ssh root@185.69.167.182 -A
ssh: connect to host 185.69.167.182 port 22: Connection refused

PeterNashaat · 2024-05-08T07:03:36Z

Focal flist Link
but had issue with extracting kernel from vmlinuz file
- Issue extracting with extract-vmlinux script

root@imagecreator:~# ./extract-vmlinux ubuntu-focal/boot/vmlinuz-5.4.0-26-generic > ubuntu-focal/boot/vmlinuz-5.4.0-26-generic.elf
extract-vmlinux: Cannot find vmlinux.

root@imagecreator:~# binwalk --extract --signature ubuntu-focal/boot/vmlinuz-5.4.0-26-generic --run-as=root

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Microsoft executable, portable (PE)
19377         0x4BB1          LZ4 compressed data, legacy
5375834       0x52075A        MySQL ISAM index file Version 1
6973538       0x6A6862        ELF, 64-bit LSB processor-specific, ("")
8810431       0x866FBF        xz compressed data
9322046       0x8E3E3E        AES Inverse S-Box
9322302       0x8E3F3E        AES S-Box
10001323      0x989BAB        LZ4 compressed data, legacy
10230704      0x9C1BB0        Certificate in DER format (x509 v3), header length: 4, sequence length: 1353
11410887      0xAE1DC7        LZ4 compressed data, legacy
11410923      0xAE1DEB        LZ4 compressed data, legacy
11656072      0xB1DB88        Object signature in DER format (PKCS header length: 4, sequence length: 1899
11656241      0xB1DC31        Certificate in DER format (x509 v3), header length: 4, sequence length: 1056

Then tried to extract output from binwalk file 866FBF.xz

root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# file 866FBF.xz
866FBF.xz: XZ compressed data, checksum

root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# xzcat 866FBF.xz > vmlinux
xzcat: 866FBF.xz: Compressed data is corrupt

root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# unxz 866FBF.xz
unxz: 866FBF.xz: Compressed data is corrupt

Uploaded the flist and tried testing without extracting the flist

Failed to send request to twinId 4927 with command: zos.deployment.get, payload: {"contract_id":442897} RMBError: 106 e2e error: invalid cipher data

Will try another kernel version instead of the default Linux kernel versions

PeterNashaat · 2024-05-08T07:05:37Z

Ubuntu 18 flist created but can's ssh to vms, investigating
ssh root@185.69.167.182 -A
ssh: connect to host 185.69.167.182 port 22: Connection refused

Not reachable over wireguard as well

root@grid-monitoring:/etc/wireguard# ping 10.20.2.2
PING 10.20.2.2 (10.20.2.2) 56(84) bytes of data.
^C
--- 10.20.2.2 ping statistics ---
55 packets transmitted, 0 received, 100% packet loss, time 55292ms

PeterNashaat · 2024-05-09T07:17:02Z

Deployed the ubuntu 20 flist on devnet as it has the latest zos version and no need to extract the kernel
- not reachable over wireguard

root@grid-monitoring:/etc/wireguard# ping 10.20.4.2
PING 10.20.4.2 (10.20.4.2) 56(84) bytes of data.
^C
--- 10.20.4.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4093ms

not reachable over mycelium

root@imagecreator:~# ping6 478:466d:a78f:64c4:ff0f:0:400:2
PING 478:466d:a78f:64c4:ff0f:0:400:2(478:466d:a78f:64c4:ff0f:0:400:2) 56 data bytes
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=1 Destination unreachable: No route
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=11 Destination unreachable: No route
^C
--- 478:466d:a78f:64c4:ff0f:0:400:2 ping statistics ---
11 packets transmitted, 0 received, +11 errors, 100% packet loss, time 10250ms

Node id 166 , Logs :
| | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9


2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109625 paused=false twin=1438
-- | -- | --
  |   | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109301 paused=true twin=89
  |   | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109300 paused=false twin=89
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.166 * Background saving terminated with success
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * DB saved on disk
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * Background saving started by pid 24546
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.139 * Background saving terminated with success
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.043 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.042 * DB saved on disk
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.038 * Background saving started by pid 15916
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.037 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.107 * Background saving terminated with success
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.011 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.010 * DB saved on disk
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * Background saving started by pid 9361
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.188 * Background saving terminated with success
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.091 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.090 * DB saved on disk
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.087 * Background saving started by pid 2981
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.086 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:false Limit:0 Storage: PersistedVolume:/mnt/dde7a534-22d5-4924-ac56-65d9a9aa2fab/rootfs:1438-109627-ubu20te9} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Creating new volume with size 26843545600
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Deleting volume 1438-109627-ubu20te9
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=cloud-container:c1f77d34c40c7879a220ba3d20b3535a storage= url=https://hub.grid.tf/tf-autobuilder/cloud-container-9dba60e.flist
  |   | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up mycelium tap interface tap-name=HVCbLtrUxUF5w
  |   | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up tap interface network-id=Gn8mNieVU6Dkq

Tried linux-generic-hwe-20.04 kernel version got the same error when extracting the kernel

PeterNashaat · 2024-05-09T11:47:02Z

Blocked on ubuntu 18 and 20.

PeterNashaat · 2024-05-09T11:47:42Z

Ubuntu 22 and 24 images are done and tested Ubuntu 22 & 24 image #237

PeterNashaat · 2024-05-14T13:34:15Z

Created a PR with updated ubuntu 22 and 24.04 full vm flists.
- Updated flists directories with 2 sub directories Fullvm and microvm
- Added automation script which automates the setup, configuration, archiving, and uploading of an Ubuntu system ready for use as an FLIST.
- Testing Full vms for now is done through Terrafrom
- Added readme.md file for Automated, Manual and Testing process

Flists links
https://hub.grid.tf/petep.3bot/ubuntu-jammy.flist.md
https://hub.grid.tf/petep.3bot/ubuntu-noble.flist.md

Tested both

xmonader assigned sabrinasadik Apr 24, 2024

xmonader added the type_story label Apr 24, 2024

PeterNashaat self-assigned this Apr 24, 2024

PeterNashaat added the state_inprogress label Apr 24, 2024

PeterNashaat added state_blocked and removed state_inprogress labels May 2, 2024

PeterNashaat added state_verification and removed state_blocked labels May 16, 2024

xmonader removed the state_verification label Jun 3, 2024

PeterNashaat mentioned this issue Jul 9, 2024

ubuntu 24.04 LTS images #228

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecate raw images #227

Deprecate raw images #227

AbdelrahmanElawady commented Apr 24, 2024 •

edited by PeterNashaat

Loading

xmonader commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

muhamadazmy commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

PeterNashaat commented Apr 28, 2024

AbdelrahmanElawady commented May 2, 2024

PeterNashaat commented May 7, 2024

PeterNashaat commented May 7, 2024

PeterNashaat commented May 8, 2024

PeterNashaat commented May 8, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 14, 2024 •

edited

Loading

Deprecate raw images #227

Deprecate raw images #227

Comments

AbdelrahmanElawady commented Apr 24, 2024 • edited by PeterNashaat Loading

Description

xmonader commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

muhamadazmy commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

PeterNashaat commented Apr 24, 2024

AbdelrahmanElawady commented Apr 24, 2024

PeterNashaat commented Apr 28, 2024

AbdelrahmanElawady commented May 2, 2024

PeterNashaat commented May 7, 2024

PeterNashaat commented May 7, 2024

PeterNashaat commented May 8, 2024

PeterNashaat commented May 8, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 9, 2024

PeterNashaat commented May 14, 2024 • edited Loading

AbdelrahmanElawady commented Apr 24, 2024 •

edited by PeterNashaat

Loading

PeterNashaat commented May 14, 2024 •

edited

Loading