[RHEL-9] Remove deprecated option crashkernel=auto from all image definitions

[thozza]

The RHEL-only patch to support kernel option crashkernel=auto has been dropped in RHEL-9. For this reason none of the RHEL-9 images should use it for kernel command line.

The functionality has been reimplemented by introducing a new file /lib/modules/$kernel_ver/crashkernel.default installed by the kernel-core package, which content should be appended to the kernel command line instead.

More information in https://bugzilla.redhat.com/show_bug.cgi?id=2006692

[gicmo]

I had a quick look at this. So crashkernel=auto got removed and what happens instead is that there is a per-kernel file /usr/lib/modules/$kernel/crashkernel.default that contains the correct kernel arguments (memory regions) that should be passed via the kernel command line (a global option). The kexec-tools gain a rpm scriptlet (92-crashkernel.install) that will install and update the values depending on various factors (/etc/defaul/grub). What is problematic for us is that it uses information of the currently running system:

boot_kernel=$(uname -r)
boot_ck_cmdline=$(sed -n -e "s/^.*\(crashkernel=\S*\).*$/\1/p" /proc/cmdline)

And then acts upon it:

[[ -z $boot_ck_cmdline ]] && exit 0

This is a no-go for us. We should mask /proc/cmdline in osbuild to make sure that we are deterministic and that actually none of the code is run. The uname system call is more problematic right now, can't do anything about this.

The immediate idea would be to create a stage that would read /usr/lib/modules/$kernel/crashkernel.default and append it to the kernel command line. For us this leads to a problem: the kernel command line must be correct in /etc/kernel/cmdline before we execute the org.osbuild.rpm stage so that the grub rpm scriptlet 20-grub.install can generate the correct BLS snippets. But in order to know what to put into /etc/kernel/cmdline we need the kernel package installed because it contains /usr/lib/modules/$kernel/crashkernel.default. In short: if we modify /etc/kernel/cmdline after org.osbuild.rpm the bls snippets won't have the changes. We want to do it before the org.osbuild.rpm stage, but we don't have the information since it is inside a package.

Two solutions come to mind:

• install the kernel/modules in the build root and use the information from there (ugly)
• create our own org.osbuild.bls stage that creates the BLS snippets (and drop the fix-bls stage)
  Then we could modify the kernel command line after the rpms are installed and run the bls stage.

[gicmo]

This is also related to #1828

[thozza]

Related to osbuild/osbuild#881

[gicmo]

Filed a bug with kexec-tools to work from within our sandbox: https://bugzilla.redhat.com/show_bug.cgi?id=2024976

[atodorov]

@gicmo, @thozza do we have any workaround here or we need to wait for the kexec-tools bug to be resolved and for the osbuild issue to be fixed ? ATM this is blocking enabling all tests on aarch64 and I'd like to know if this is a real blocker or if QE can work around it.

[thozza]

Hi @atodorov . Can you please elaborate how is this blocking testing on aarch64?

[jrusz]

To be more exact, it's blocking testing on rhel-9 https://gitlab.com/osbuild/ci/osbuild/-/jobs/1796236328 ... At the time I wrote the comment we didn't yet comment out testing on rhel-9 and for my PR to get merged I had to switch from rhel-9-beta to rhel-9 runners but they were having issues with crashkernel during testing.

[atodorov]

Hi @atodorov . Can you please elaborate how is this blocking testing on aarch64?

Not blocking anymore.

[thozza]

To be more exact, it's blocking testing on rhel-9 https://gitlab.com/osbuild/ci/osbuild/-/jobs/1796236328 ... At the time I wrote the comment we didn't yet comment out testing on rhel-9 and for my PR to get merged I had to switch from rhel-9-beta to rhel-9 runners but they were having issues with crashkernel during testing.

Thanks for the additional info.

So the issue is that we tend to use Fedora systems reserved in Beaker for generating the image test cases and those don't have the crashkernel=auto in /proc/cmdline, but the images that we use for testing (https://github.com/osbuild/gitlab-ci-terraform/blob/428f543041dc410a592c7f3ead5f2dd4143722ab/aws/rhel-9.0-nightly-aarch64/main.tf#L5) have crashkernel=auto in /proc/cmdline. As a result, the script shipped with kexec-tools modifies the kernel cmdline arguments in the generated image if the host system on which the image is being built contains crashkernel=auto in its kernel cmdline.

The issue should go away as soon as osbuild is enhanced to mask /proc/cmdline from the host.

I think that a workaround could be to remove the crashkernel option from the kernel cmdline on the CI runners,