QA rocky 9 from master (slack bug report)

[Itxaka]

Hey Kairos team. I think maybe the current version of master is broken.
I just rebuilt our image from
9-core-amd64-generic-master sha256:b2002f6bbb0aaf74ef38ca0040beaa37dfc33ab209023255f46122301e613b04
and it won't boot / install now.
Here's what we are seeing

Slack Message

Reported by @sarg3nt

[ci-robbot]

Hello Itxaka, thank you for bringing this to our attention. I have added the 'question' label to your issue to indicate that it needs more information. In order to properly triage and address your concern, please provide a detailed description of the problem you are experiencing, including steps to reproduce if it's a bug, and make sure to mention the versions of the relevant artifacts being used. This will help us better understand the issue and determine if it's related to the current master version. Thank you for your cooperation, and feel free to reply if you have any further questions. Remember, I am a bot, an experiment of @mudler and @jimmykarily.

[jimmykarily]

I built a rockylinux image from master (aad432b) with this command:

earthly +iso --VARIANT=core --FLAVOR=rockylinux --FLAVOR_RELEASE=9 --BASE_IMAGE=rockylinux:9  --MODEL=generic --FAMILY=rhel

In qemu it installs and boots fine:

...
Rocky Linux 9.4 (Blue Onyx)
Kernel 5.14.0-427.16.1.el9_4.x86_64 on an x86_64

localhost login: kairos
Password: 
[kairos@localhost ~]$ cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
KAIROS_PRETTY_NAME="kairos-core-rockylinux-9 v3.0.4-66-gaad432bb"
KAIROS_FAMILY="rhel"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ARTIFACT="kairos-rockylinux-9-core-amd64-generic-v3.0.4-66-gaad432bb"
KAIROS_MODEL="generic"
KAIROS_RELEASE="v3.0.4-66-gaad432bb"
KAIROS_VARIANT="core"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_VERSION="v3.0.4-66-gaad432bb"
KAIROS_ID_LIKE="kairos-core-rockylinux-9"
KAIROS_IMAGE_REPO="quay.io/kairos/rockylinux:9-core-amd64-generic-v3.0.4-66-gaad432bb"
KAIROS_IMAGE_LABEL="9-core-amd64-generic-v3.0.4-66-gaad432bb"
KAIROS_FLAVOR="rockylinux"
KAIROS_FLAVOR_RELEASE="9"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_ID="kairos"
KAIROS_NAME="kairos-core-rockylinux-9"
KAIROS_VERSION_ID="v3.0.4-66-gaad432bb"
KAIROS_TARGETARCH="amd64"

[jimmykarily]

I tried with auroraboot (using the same container image built above):

docker run --rm -ti -v $PWD/config.yaml:/config.yaml -v /var/run/docker.sock:/var/run/docker.sock --net host quay.io/kairos/auroraboot --set "container_image=docker://quay.io/kairos/rockylinux:9-core-amd64-generic-v3.0.4-66-gaad432bb" --cloud-config /config.yaml

config.yaml:

#cloud-config

strict: true
debug: true

install:
  auto: true
  reboot: true
  grub_options:
    extra_cmdline: "rd.immucore.debug"
users:
  - name: "kairos"
    passwd: "kairos"

fails like this:

[jimmykarily]

Simply switching to docker://quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.0.4-66-gaad432bb in the above example (auroraboot) makes it work just fine. It certainly has something to do with rockylinux.

[jimmykarily]

I also can't get a shell on rocky linux (to debug further). I tried with console rocky9 in virsh but it fails with:

Last login: Tue May 14 14:15:51 on ttyS0
 -- root: no shell: Permission denied

[jimmykarily]

I have to remove --progress from the rsync command the kairos-agent spins up to see where rsync fails:

[jimmykarily]

I disabled SELINUX (following this guide) and now it works just fine by netbooting with Auroraboot (installs, reboots to the installed system and I can login as kairos/kairos).

[jimmykarily]

Just for reference, the way I'm quickly tweaking the image to boot:

• I run a container with: docker run -it quay.io/kairos/rockylinux:9-core-amd64-generic-v3.0.4-66-gaad432bb and keep it running
• I make changes in the container
• I commit to an image: docker commit 67099a9cad00 rocky:test
• I restart the auraboot command: docker run --rm -ti -v $PWD/config.yaml:/config.yaml -v /var/run/docker.sock:/var/run/docker.sock --net host quay.io/kairos/auroraboot --set "container_image=docker://rocky:test" --cloud-config /config.yaml
• I reset the VM and make it netboot again