debian: Unable to boot Kairos installer

[6ixfalls]

Kairos version:

Fails to boot on kairos-debian-bookworm-standard-amd64-generic-v3.0.8-k3sv1.29.3+k3s1, success on kairos-debian-bookworm-standard-amd64-generic-v3.0.0-k3sv1.29.0+k3s1

CPU architecture, OS, and Version:

Linux localhost 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

(output from v3.0.0)

Describe the bug

The Kairos ISO is unable to boot and I'm unable to install Kairos (manually and automatically).

To Reproduce

Try to install Kairos on the latest version, not sure if this is reproducible. This is running in a KVM VM.

Expected behavior

I should be able to boot into the Kairos install ISO.

Logs

Additional context

This bug looks exactly like #2467, but trying the fix there and adding that to a Dockerfile doesn't resolve the issue.

[ci-robbot]

Hello, 6ixfalls! I'm an automated bot assisting with Github issue audits in the kairos project. I've added the 'question' label to your issue (#2522) because it appears we need more information to properly investigate your report.

To enhance our understanding and help us better address your problem, please provide:

• A clear description of the issue you're experiencing, including any error messages you receive.
• Steps to reproduce the problem, if possible.
• The versions of all relevant artifacts you're using, such as the Kairos version, CPU architecture, OS version, and any specific configurations or dependencies.

Please ensure that your description, steps to reproduce, and version details are explicitly mentioned in your issue. We appreciate your efforts to help us improve Kairos, and don't hesitate to reach out if you have any questions. Note that I am a bot, an experiment of @mudler and @jimmykarily.

Thanks!
kairos-io Githubbot

[6ixfalls]

This could be related, but I'm using a custom docker image with auroraboot to generate an ISO. The Dockerfile is here:
https://github.com/6ixfalls/taonet-cloud/blob/main/containers/kairos-debian/Dockerfile

It also appears this issue was introduced between 3.0.0 and 3.0.3 - this appears to be a fix to the issue:
tyzbit/kairos-distros@e11adda

[tyzbit]

A note: that was an attempted fix. It didn't fix it for me on 3.0.3 but I didn't try other versions.

[jimmykarily]

Maybe this is relevant? https://github.com/kairos-io/packages/blob/718aaa27e4688559433cd889513f1944a7679ef4/packages/static/kairos-overlay-files/files/system/oem/12_nvidia.yaml#L10

[jimmykarily]

Maybe this is relevant? https://github.com/kairos-io/packages/blob/718aaa27e4688559433cd889513f1944a7679ef4/packages/static/kairos-overlay-files/files/system/oem/12_nvidia.yaml#L10

oh wait, you are not on nvidia. On the other hand, maybe that module needs to be included somehow (?).

[jimmykarily]

Maybe this is relevant? https://github.com/kairos-io/packages/blob/718aaa27e4688559433cd889513f1944a7679ef4/packages/static/kairos-overlay-files/files/system/oem/12_nvidia.yaml#L10

oh wait, you are not on nvidia. On the other hand, maybe that module needs to be included somehow (?).

maybe not that irrelevant after all: https://forums.fedoraforum.org/showthread.php?325865-dracut-FATAL-iscsi-requested-but-kernel-initrd-does-not-support-iscsi

you could try to omit iscsi in dracut to see if this helps

[6ixfalls]

Maybe this is relevant? https://github.com/kairos-io/packages/blob/718aaa27e4688559433cd889513f1944a7679ef4/packages/static/kairos-overlay-files/files/system/oem/12_nvidia.yaml#L10

oh wait, you are not on nvidia. On the other hand, maybe that module needs to be included somehow (?).

maybe not that irrelevant after all: https://forums.fedoraforum.org/showthread.php?325865-dracut-FATAL-iscsi-requested-but-kernel-initrd-does-not-support-iscsi

you could try to omit iscsi in dracut to see if this helps

I'm not sure if this is how to correctly do it, but I tried this configuration and it did not fix the issue.

[tyzbit]

If this does what I suspect, this would break compatibility with at least Longhorn. Can we see what it takes for the kernel to support iscsi?

[mudler]

It looks like we need to disable iscsi as we do already for nvidia:

kairos/images/Dockerfile.nvidia

Line 101 in f5c1050

# Disable ISCSI

[Itxaka]

looks like iscsi modules are not properly set in the initramfs as dracut failure indicates that its checking for the iscsi_tcp mod to be available

You could try to install iscsiuio alongside and regenerate the initramfs as that seems to bring the proper iscsi_tcp module needed by dracut

Im gonna try a qucik test here, but I can see already that once installing that package the modules are available and iscsi is added to the dracut modules

what cmdline are you using?

[Itxaka]

with a quick patch to install the package alongside Kairos and letting dracut regenerate the initramfs the proper module is available and loaded:

[athnoc-dev]

I can confirm that customizing the Debian image (only tested this one) from v3.0.0 and up produces the "iscsi error" for dracut. I followed this doc https://kairos.io/docs/advanced/customizing/ at first. Then I used this docker file (https://github.com/kairos-io/kairos/blob/master/images/Dockerfile.kairos-debian) to rebuild the image from scratch while adding packages I needed. Still the iscsi error from dracut appeared. After that I added the "iscsiuio" package and net booting with Aurora worked... the first time.

The second time I launched Aurora at tried to net boot the server, it gave me the same error. I inspected the temp directory to which Auroraboot extracts the ISO and the /netboot directory contains all the net boot artifacts. I inspected the kernel file and compared it to the kernel files in the ISO (which are unpacked in the temp directory).

I found that the net boot kernel (kairos-kernel) was the oldest kernel file and not the most recent, which is why it did not contain the iscsi module of which dracut complains it is not present in the kernel during net boot. I copied the latest kernel and used the other artifacts in /tmp/netboot to start pixiecore and everything worked as expected.

It looks like Auroraboot is picking the wrong kernel (sometimes) for booting, can you confirm?

[jimmykarily]

Let's install iscsiuio by default (all flavors?) so that it makes it to the initramfs.

[tyzbit]

I tried that and it did not seem to help tyzbit/kairos-distros@e11adda
It does strongly seem to be an AuroraBoot issue

[mauromorales]

let's try to replicate in auroraboot and see if we can detect what the issue actually is

[athnoc-dev]

Check which kernel AuroraBoot is using in /tmp/netboot

In my case the errors persisted because an older kernel was used, instead of the latest that had the supporting iscsi modules.

I copied the latest kernel from the temp directory (the unpacked ISO) and replaced the kernel file and all worked fine.

[jimmykarily]

• I created a patch in kairos:

~/workspace/kairos/kairos (master)*$ git diff
diff --git a/images/Dockerfile.debian b/images/Dockerfile.debian
index 39d94482..07862509 100644
--- a/images/Dockerfile.debian
+++ b/images/Dockerfile.debian
@@ -64,6 +64,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     iputils-ping \
     isc-dhcp-common \
     isc-dhcp-client \
+    iscsiuio \
     jq \
     krb5-locales \
     less \
@@ -162,4 +163,4 @@ RUN systemctl enable systemd-networkd
 RUN systemctl enable ssh
 
 # Fixup sudo perms
-RUN chown root:root /usr/bin/sudo && chmod 4755 /usr/bin/sudo
\ No newline at end of file
+RUN chown root:root /usr/bin/sudo && chmod 4755 /usr/bin/sudo
diff --git a/images/Dockerfile.kairos-debian b/images/Dockerfile.kairos-debian
index 60c85c1d..3391363c 100644
--- a/images/Dockerfile.kairos-debian
+++ b/images/Dockerfile.kairos-debian
@@ -63,6 +63,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     iputils-ping \
     isc-dhcp-common \
     isc-dhcp-client \
+    iscsiuio \
     jq \
     krb5-locales \
     less \

• I built a container image:

earthly +base-image --VARIANT=core --FLAVOR=debian --FLAVOR_RELEASE=bookworm-slim  --BASE_IMAGE=debian:bookworm-slim  --MODEL=generic --FAMILY=debian

• I started Auroraboot with the result image:

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock --net host quay.io/kairos/auroraboot --set "container_image=docker://quay.io/kairos/debian:bookworm-slim-core-amd64-generic-v3.0.4-73-g8ddb9092-dirty"

• I started a VM in netboot mode (with virt-manager)

It successfully boots debian.

Since the docker command I used to run Auroraboot didn't mount any volumes, it's not possible to have cached any data between runs. @tyzbit how are you running Auroraboot? @athnoc-dev suggestion makes me think that some people might be using some command (from our docs?) that is mounting a volume and caches things. Is that the case?

[6ixfalls]

Since the docker command I used to run Auroraboot didn't mount any volumes, it's not possible to have cached any data between runs. tyzbit how are you running Auroraboot? athnoc-dev suggestion makes me think that some people might be using some command (from our docs?) that is mounting a volume and caches things. Is that the case?

This is true in my case - I use auroraboot to generate ISOs to upload to my Kairos nodes, and as a result I have a mount so that I can access the completed ISO. I don't think it should be expected behavior for auroraboot to not generate a new kernel if there's an existing one present - but I'm also not sure if reusing the same directory for building has any effect on the speed of the builds themselves either.

I'm actually not too sure if this is a kernel issue, because as far as I remember this issue occurs with a fresh auroraboot install. However, another thing that appears to be common among everyone who has the issue is that the Kairos Dockerfile is modified (is it possible that the Github Action caching the Docker buildsteps leads to this issue?)