Strange failed (first) boot behavior of FCOS 36 on Hetzner Dedicated EX43 Alder Lake

[cryptoluks]

Describe the bug
I installed FCOS via coreos-installer through Hetzner PXE Rescue OS. Unfortunately, when installing stable (36.20220918.3.0) or testing (36.20221001.2.0) directly, the machine is unable to boot. However, when installing the next branch (37.20221003.1.0) the machine boots just fine. More strangely, doing a rebase with rpm-ostree rebase to the aforementioned stable or testing versions, the machine then ALSO boot just fine. Installing an older stable version directly like 36.20220806.3.0 was not successful either.

Reproduction steps
Steps to reproduce the behavior:

1. Install needed tools on Hetzner Rescue OS

apt install libzstd-dev libssl-dev pkg-config
curl https://sh.rustup.rs -sSf | sh
source "$HOME/.cargo/env"
cargo install coreos-installer
wget https://github.com/coreos/butane/releases/download/v0.15.0/butane-x86_64-unknown-linux-gnu
chmod +x butane-x86_64-unknown-linux-gnu
./butane-x86_64-unknown-linux-gnu --strict --pretty < config.btn > config.ign
coreos-installer install /dev/foo -i config.ign

2. Reboot machine, hangs at the following screen captured via remote console

Expected behavior
Machine should boot just fine with stable or testing versions installed directly.

Actual behavior
The machine does not boot when installing stable 36.20220918.3.0 or testing 36.20221001.2.0 directly. It is able to boot from this versions after installing next 37.20221003.1.0 first, and then rebasing to stable or testing.

System details

• Bare Metal
• EX43 Alder Lake i5-12500 with UEFI boot, see here
• Fedora CoreOS version 36.20220918.3.0 (stable)
• Fedora CoreOS version 36.20221001.2.0 (testing)

Ignition config

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: root
      ssh_authorized_keys:
        - ssh-ed25519 [..]
storage:
  files:
    - path: /root/.ssh/id_ed25519
      mode: 0600
      contents:
        inline: |
          [..]

Additional information
The most recent, most relevant issue on this tracker seems to be #1306, because EX43 is also using NVME (4.0) SSDs. But as the Hetzner EX43 machine boots just fine with the kernel after rebase, I suspect it is probably not a kernel-related issue?

Furthermore, I installed the stable version on a different dedicated Server from the Server Auction with somewhat older hardware (Intel i7 7700, older NVME SSDs), which also works just fine.

Thank you very much.

[jlebon]

This could be related to #567, which is fixed on next but not yet the other streams.

Can you try again with:

coreos-installer install /dev/foo -i config.ign --delete-karg console=ttyS0,115200n8

?

[cryptoluks]

This could be related to #567, which is fixed on next but not yet the other streams.

Can you try again with:

coreos-installer install /dev/foo -i config.ign --delete-karg console=ttyS0,115200n8

Yes! This was it!

Booting is now working as expected with the stable stream.

Thank you very much, have a nice day! ❤️