Failed boot of Ignite VM

[nbrownuk]

I've successfully created and imported a kernel based on v5.3.18 source (using the Makefile)

# ignite kernel ls
KERNEL ID		NAME				CREATED	SIZE	VERSION
67d787f7f4401db3	nbrown/ignite-kernel:5.3.18	46m	62.3 MB	5.3.18

and a Ubuntu image based on 19.10 (using the Dockerfile)

# ignite image ls
IMAGE ID		NAME				CREATED	SIZE
eea6fe954600f2df	nbrown/ignite-ubuntu:19.10	49m	214.8 MB

The image contains the correct symlink to /lib/systemd/systemd

# docker run --rm nbrown/ignite-ubuntu:19.10 ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Feb  6 14:45 /sbin/init -> /lib/systemd/systemd

and the imported image also contains the symlink

# mount -t ext4 /var/lib/firecracker/image/eea6fe954600f2df/image.ext4 /home/nigel/tmp/mnt/
# ls -l /home/nigel/tmp/mnt/sbin/init /home/nigel/tmp/mnt//lib/systemd/systemd
-rwxr-xr-x 1 root root 1497296 Nov  1 20:33 /home/nigel/tmp/mnt//lib/systemd/systemd
lrwxrwxrwx 1 root root      20 Nov  1 20:33 /home/nigel/tmp/mnt/sbin/init -> /lib/systemd/systemd
# umount /home/nigel/tmp/mnt

If I attempt to run a VM based on the above, the VM fails to boot because it can't find an init.

# ignite run nbrown/ignite-ubuntu:19.10 --name my-vm -k nbrown/ignite-kernel:5.3.18 -i
INFO[0001] Created VM with ID "d387c17ae34273ba" and name "my-vm" 
INFO[0002] Networking is handled by "cni"               
INFO[0002] Started Firecracker VM "d387c17ae34273ba" in a container with ID "ignite-d387c17ae34273ba" 
d387c17ae34273ba
INFO[0001] Moving IP address 10.61.0.136 (255.255.0.0) from container to VM 
INFO[0001] Called startVMM(), setting up a VMM on /var/lib/firecracker/vm/d387c17ae34273ba/firecracker.sock 
INFO[0001] Starting DHCP server for interface "br_eth0" (10.61.0.136) 
INFO[0001] refreshMachineConfiguration: [GET /machine-config][200] getMachineConfigurationOK  &{CPUTemplate:Uninitialized HtEnabled:0xc00048a203 MemSizeMib:0xc00048a1a8 VcpuCount:0xc00048a1a0} 
INFO[0001] PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent  
INFO[0001] Attaching drive /dev/mapper/ignite-d387c17ae34273ba, slot 1, root true. 
INFO[0001] Attached drive /dev/mapper/ignite-d387c17ae34273ba: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent  
INFO[0001] Attaching NIC vm_eth0 (hwaddr 36:2a:88:bf:90:40) at index 1 
INFO[0001] PutGuestNetworkInterfaceByID: [PUT /network-interfaces/{iface_id}][204] putGuestNetworkInterfaceByIdNoContent  
INFO[0001] startInstance successful: [PUT /actions][204] createSyncActionNoContent  
[    0.000000] Linux version 5.3.18 (root@buildkitsandbox) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #1 SMP Tue Feb 25 12:28:35 UTC 2020
[    0.000000] Command line: console=ttyS0 reboot=k panic=1 pci=off ip=dhcp root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6

<snip>

[    1.055261] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    1.068617] devtmpfs: mounted
[    1.079631] Freeing unused decrypted memory: 2040K
[    1.088933] Freeing unused kernel image memory: 1392K
[    1.095830] Write protecting the kernel read-only data: 14336k
[    1.114506] Freeing unused kernel image memory: 2012K
[    1.126182] Freeing unused kernel image memory: 324K
[    1.132983] Run /sbin/init as init process
[    1.151276] Run /etc/init as init process
[    1.154247] Run /bin/init as init process
[    1.157346] Run /bin/sh as init process
[    1.159740] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    1.164735] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.3.18 #1
[    1.167421] Call Trace:
[    1.168591]  dump_stack+0x57/0x75
[    1.171579]  ? csum_partial_copy_generic+0x2790/0x27d0
[    1.173899]  panic+0xf3/0x2a9
[    1.174795]  ? rest_init+0xb0/0xb0
[    1.175937]  kernel_init+0xeb/0x100
[    1.177033]  ret_from_fork+0x35/0x40
[    1.178128] Kernel Offset: disabled
[    1.178904] Rebooting in 1 seconds..
INFO[0003] firecracker exited: status=0

Any suggestions as to how to troubleshoot this? Thanks.

[darkowlzz]

Looks to be related to #526
Have you tried running with Ubuntu 18.04 image (weaveworks/ignite-ubuntu)?

[nbrownuk]

Yes, weaveworks/ignite-ubuntu:18.04 runs OK, and I'm able to create my own version of an 18.04 image. Seems a very similar problem to #526. Is that PR to be merged?

[darkowlzz]

#526 was merged along with some more related fixes in #577 . This should work now.

[nbrownuk]

Thanks. After I worked out how to build a local copy of ignite to test, it works just fine.

[amd64] (master [!]) sudo ./ignite run nbrown/ignite-ubuntu:20.04 --name my-vm -k nbrown/ignite-kernel:5.4.13 --ssh
INFO[0008] Created VM with ID "100f65f2dd0f47b5" and name "my-vm" 
INFO[0008] Networking is handled by "cni"               
INFO[0008] Started Firecracker VM "100f65f2dd0f47b5" in a container with ID "ignite-100f65f2dd0f47b5" 
[amd64] (master [!]) sudo ./ignite exec my-vm cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu Focal Fossa (development branch)"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal