systemd-boot: Boot entries in reverse order

[hunger]

systemd version the issue has been seen with

254

Used distribution

Arch Linux

Linux kernel version used

6.4.11-arch2-1

CPU architectures issue was seen on

x86_64

Component

systemd-boot

Expected behaviour you didn't see

I have several UKI-kernels installed in /efi/EFI/Linux. There is no configuration file whatsoever in /efi.

This is the output of bootctl list:

         type: Boot Loader Specification Type #2 (.efi)
        title: Cleanroom (20230815.1356) (default)
           id: arch_20230815.1356.efi
       source: /efi//EFI/Linux/arch_20230815.1356.efi
     sort-key: 20230815.1356
      version: 20230815.1356
        linux: /efi//EFI/Linux/arch_20230815.1356.efi
      options: i915.fastboot=1 intel_iommu=igfx_off mem_sleep_default=deep rd.luks.uuid=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa rd.luks.opt>

         type: Boot Loader Specification Type #2 (.efi)
        title: Cleanroom (20230821.1859) (selected)
           id: arch_20230821.1859.efi
       source: /efi//EFI/Linux/arch_20230821.1859.efi
     sort-key: 20230821.1859
      version: 20230821.1859
        linux: /efi//EFI/Linux/arch_20230821.1859.efi
      options: i915.fastboot=1 intel_iommu=igfx_off mem_sleep_default=deep rd.luks.uuid=bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb rd.luks.opt>

         type: Automatic
        title: Reboot Into Firmware Interface
           id: auto-reboot-to-firmware-setup
       source: /sys/firmware/efi/efivars/LoaderEntries-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f

I expect the latest kernel (20230821.1859) to boot.

Unexpected behaviour you saw

The oldest kernel boots (20230815.1356).

Steps to reproduce the problem

Install several UKI-kernels, check boot order in bootctl.

Additional program output to the terminal or log subsystem illustrating the issue

No response

[medhefgo]

Try bootctl set-default ''.

[hunger]

Thank you for looking at my issue!

I just run bootctl set-default ''. bootctl list output is unchanged, I end up in the old kernel on reboot, too.

Shell history shows I never ran any bootctl command but bootctl list and bootctl install.

[medhefgo]

The issue is that the generated sort-key for the UKIs is too specific, forcing a specific sort order. The sort key is not supposed to be that unique as to allow ordering by (kernel) version.

Please provide the output of objcopy --dump-section .osrel=/dev/stdout /efi/EFI/Linux/arch_20230815.1356.efi

[hunger]

The output of objcopy --dump-section .osrel=/dev/stdout /efi/EFI/Linux/arch_20230815.1356.efi is:

NAME="cleanroom"
PRETTY_NAME="Cleanroom"
ID="clrm"
ID_LIKE="archlinux"
ANSI_COLOR="0;36"
HOME_URL="https://github.com/cleanroom-team/cleanroom"
SUPPORT_URL=""
BUG_REPORT_URL="https://github.com/cleanroom-team/cleanroom/issues"
VERSION="20230815.1356"
VERSION_ID="20230815.1356"
IMAGE_ID="20230815.1356"
IMAGE_VERSION="20230815.1356"

I did add the IMAGE_* keys recently, but the rest is getting generated like this for years now and systemd-boot used to boot that just fine:-)

I generate immutable Linux images several times a day with the timestamp as a version number. The images tend to use the same kernel version, but since the dm-verity hash of the root filesystem is baked into the UKI, each generated image comes with its own UKI kernel booting exactly that one image. So I do needed these overly-specific version numbers to get the sort order right for the last couple of years.

[medhefgo]

We use IMAGE_ID or ID when picking a sort-key. And reading man os-release I would argue that you're using the fields not as intended (but I could be wrong). As I see it, IMAGE_ID should not contain version identification (except for things like debian-11 vs debian-12).

To summarize: if the image updates are built and shipped as comprehensive units, IMAGE_ID+IMAGE_VERSION is the best fit. Otherwise, if updates eventually completely replace previously installed contents, as in a typical binary distribution, VERSION_ID should be used to identify major releases of the operating system. BUILD_ID may be used instead or in addition to VERSION_ID when the original system image version is important.

@poettering Your thoughts.

[hunger]

So I should use something like this:

NAME="cleanroom"
PRETTY_NAME="Cleanroom"
ID="clrm"
ID_LIKE="archlinux"
ANSI_COLOR="0;36"
HOME_URL="https://github.com/cleanroom-team/cleanroom"
SUPPORT_URL=""
BUG_REPORT_URL="https://github.com/cleanroom-team/cleanroom/issues"
VERSION="20230815.1356"
VERSION_ID="rolling"
IMAGE_ID="arch-rolling"
IMAGE_VERSION="20230815.1356"

But considering that all the kernels have the same uname output: What order will those show up in then?

Do I need to append the version/timestamp to the kernel uname when creating the UKI? That seems wrong somehow: The kernel is unchanged and identical to whatever arch ships. The UKI is specific to the image due to dm-verity root hashes in the kernel cmdline only.

---

I saw that you are using IMAGE_ID and ID to pick the sort key: That is exatly why I put my version/timestamp in there :-)

In my case all the sort-keys can be comparable with each other in a meaningful way and put in an order. All the groups with the same sort-key contain one element only, so you can sort those any way you want without affecting the order. So I think there should not be a technical reason for this to not work.

Indeed sorting does seem to work: Systemd-boot consistently puts the oldest kernel to the top of the list of kernels to boot and puts the newer kernels in order below that all the way to the newest. I am just surprised by the sort order: Why is the oldest version bubbling to the top?

I think you will probably see the same issue when installing several "debian" and several "fedora" kernels. I bet they also get sorted in reverse alphabetical order, which again is unexpected to me.

But yes, you will not really notice this issue if you only have kernels with the same ID/IMAGE_ID as long as sorting by kernel version works after sorting by sort key.

[medhefgo]

Well, the sorting has become complicated because a certain someone prefers silly sorting based on file names, so that people can do silly things like 00-Zubuntu first.conf. And that basically forces them to be sorted in reverse order so that if they have (kernel) versions in them the expected order happens (zubuntu.conf<debian_4.6.conf<debian_4.5.conf). In fact, this is the only way to sort these, as other things are not looked at.

Now to make this really funny, setting a sort-key will change the algorithm to this: sort-key<machine-id<version<filename (in ascending order, the version descending). UKIs have a sort-key picked based on IMAGE_ID<ID and version based on IMAGE_VERSION<VERSION<VERSION_ID<BUILD_ID.

(If it were up to me, we'd use the sort-key algorithm for .conf entries too to keep things consistent).

Note that the kernel uname has no bearing in all of this if it's not put in any of the picked version fields of the UKI. So you have to do that, if that's what you require.

So your suggested new osrel should do the trick, though I think IMAGE_ID=clrm-arch or so would be more appropriate, no?

[hunger]

@medhefgo: Thanks for explaining all this!

It always looks sooo easy and obvious when you look at it from the outside (like I did), missing 90% of the use cases and just focusing on the one thing you care for at this one moment in time! Sorry to be such an ignorant user myself in this instance.

I'll try updating the os-release file tonight and get back to you tomorrow. I'll probably need to create a few images to verify the sorting, so I might not get back to you for a few days.

[hunger]

Changing the os-release file as suggested resolved the issue!

Thank you for your time and patience, I would not have spotted this on my own!