Systemd-boot fails to recognize keyboard input on some systems #143847

anund · 2021-10-30T14:14:31Z

Describe the bug

systemd-boot fails to correctly recognize keyboard input after version 248 on some systems. This is a known issue systemd/systemd#19191 with a known patch https://github.com/archlinux/svntogit-packages/blob/packages/systemd/trunk/0003-PARTIAL-REVERT-commit-tree-wide-replace-strverscmp-and-str_verscmp-with-strverscmp_improved.patch

Steps To Reproduce

Steps to reproduce the behavior:

Fix nixos/systemd-boot: Fix installed version regexp #140046
Update to unstable sometime post 6455697 (systemd v249 #131618)
Rebuild
Boot
Only the default option will successfully boot as keyboard input does not work. (on some systems)

Expected behavior

systemd-boot accepts keyboard input

Additional context

Reading through the systemd issue it appears this is related to compiling with GCC 10. There is a suggestion it goes away with GCC 11 and there is a change on master at the end of the issue (different from the known patch) that also suggests it fixes the issue.

The associated arch systemd-stable patch will not apply as is.

#140046 Currently prevents systemd-boot updates from applying as bootctl --path=... --update is not being called. I think this prevents most users on unstable from seeing this issue.

Notify maintainers

@andlr @eelco @flokli @kloenk

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.66, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(<user>): `"home-manager"`
 - channels(root): `"nixos-21.11pre325810.2deb07f3ac4"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: systemd
# a list of nixos modules affected by the problem
module:

The text was updated successfully, but these errors were encountered:

anund · 2021-10-31T11:35:00Z

Fixed the systemd reference issue link.

flokli · 2021-11-01T07:48:46Z

@anund with systemd/systemd#19191 merged, did this end up getting fixed upstream (with systemd/systemd@e98d271)?

If so, has it been cherry-picked to systemd-stable?

xaverdh · 2021-11-01T10:50:27Z

Judging from https://github.com/systemd/systemd-stable/blob/v249-stable/src/boot/efi/boot.c it has not been picked up there.

flokli · 2021-11-01T13:40:57Z

I asked in systemd/systemd#19191 (comment).

anund · 2021-11-01T15:40:54Z

I'm working through verifying the suggested fixes. I've settled on the following patching method to rebuild less than the entire world.

diff --git a/nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix b/nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix
index 0f76d7d6b24..2d78ad0935f 100644
--- a/nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix
+++ b/nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix
@@ -14,7 +14,13 @@ let
 
     inherit (pkgs) python3;
 
-    systemd = config.systemd.package;
+#    systemd = config.systemd.package;
+#    systemd = (config.systemd.package.override {
+#      stdenv = pkgs.gcc11Stdenv;
+#    });
+    systemd = config.systemd.package.overrideAttrs (oldAttrs: {
+      patches = oldAttrs.patches ++ [ ./0024-PATCH-PARTIAL-REVERT-commit-tree-wide-replace-strver.patch ];
+    });
 
     nix = config.nix.package.out;

The arch partial revert patch may not actually work. They may have fixed it by building with gcc11. ~few days and I should be able to say which of the three possible options fix the bootloader for my system. Likely the slowest thing at this point would be figuring out what the back port patch for 19191 will be for the nix systemd-stable + patch set.

anund · 2021-11-02T05:28:52Z

All three fixes work. Arch patch manually applied to fix line number differences, compiling with gcc11 the way indicated above, and manually back porting systemd/systemd@e98d271.

anund · 2021-11-02T07:16:25Z

systemd/systemd#19191 (comment) next 249 stable release will have the fix.

IvarWithoutBones · 2021-11-02T17:41:25Z

Should we add a patch with these fixes while we wait for the next upstream release? I'm experiencing this issue as well, it's quite annoying.

flokli · 2021-11-03T08:13:49Z

I think we could. systemd updates need to go to staging and the staging-next cycle, so take some time.

Could you file a PR that fetchpatch'es 218c0839b24853899d2ea15cb2973ab0d56a7f31 and /9e47ea7c98d1f4e0a75edb5d1590e5742f253317 from systemd-stable until 249.6 is tagged?

anund · 2021-11-03T12:26:30Z

Created #144422 with the two cherry-picked commits as two patches.

doronbehar · 2021-11-26T11:47:38Z

I still can't press any keys while in boot menu - I just updated to 73369f8 which includes the systemd patch added in #144422 . I also ran sudo bootctl install before I ran nixos-rebuild to this nixpkgs revision in order to solve the could not find any previously installed systemd-boot issue. How can I further debug this?

IvarWithoutBones · 2021-11-26T19:33:54Z

It seems the issue still occurs for me as well.

anund · 2021-11-27T09:40:17Z

Try nixos-rebuild --install-bootloader... to overwrite the existing broken bootloader. Normal rebuilds use bootctl update which should noop if the version is the same. You need #144007 for the --install-bootloader to work.

doronbehar · 2021-11-27T10:37:10Z

Normal rebuilds use bootctl update which should noop if the version is the same.

Can you point me / us to where that version is compared? You mean the version of systemd-boot?

anund · 2021-11-27T10:37:45Z

Why wouldn't sudo bootctl install the right thing? What could have happened for @doronbehar

Existing nixpkgs version the system is running from includes broken systemd version (v249.4/5)
sudo bootctl install force installs that broken version (any nixpkgs version after 6455697)
nixos-rebuild attempts to update systemd to v249.5+patch set resulting in a noop because the versions are the same

For debugging you need to make https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py#L231-L267 happy. The first branch of the if is applicable with --install-bootloader is set for nixos-rebuild. If using --install-bootloader doesn't work (check you've got a nixpkgs version including #144007) you can try messing around with a custom copy of systemd just for the bootloader following this comment #143847 (comment).

anund · 2021-11-27T11:34:04Z

The version of the systemd-boot on disk is the important version. There are three version comparison that come up during rebuild:

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py#L247-L252 manual version comparison to determine if we should call bootclt ... update.
bootctl ... update doing it's own version comparison to avoid force overwriting existing, probably working, boot partition contents.
systemd-boot reporting it's version, via bootclt ... status as 249.4 or 247 and the version of nixpkgs using a regex that can't find that value.

The first will skip updating the bootloader if the currently running copy of systemd and systemd-boot agree they're the "same" version. Systemd version comes from bootctl --version. Post #140046 the value compared changes from the second word to the third in systemd 249 (249.5) when splitting on space. Systemd-boot version comes from bootctl ... status and an additional regex step. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py#L261 means the versions are compared as exact match strings ie (249.4).

The second will avoid downgrading to an old version. This will happen if systemd-boot is version 249.4 and nixpkgs is pointed at a systemd version <247.6.

The third comparison is where many of the could not find any previously installed systemd-boot messages come from. Nixpkgs has a regex that can't match bootctl ... status. Either because systemd-boot is version <247.6 resulting in a version of 247 and nixpkgs is looking for ([\d.]+[^)]*) or systemd-boot version >249 resulting in a version of 249.4 and nixpkgs is looking for (\d+).

anund · 2021-11-27T12:39:24Z

Nixos unstable has three changes https://nixpk.gs/pr-tracker.html?pr=140046, https://nixpk.gs/pr-tracker.html?pr=144422, and https://nixpk.gs/pr-tracker.html?pr=144429.

If your bootloader is currently broken with the problems listed in this bug nixos-rebuild --install-bootloader boot/switch will overwrite the broken bootloader. If that does not fix your machine you will need to start from zero using the above comments as guides.

doronbehar · 2021-11-27T16:01:34Z

nixos-rebuild --install-bootloader Worked. Thanks for investigating such a lot of thought into this, I kind of lost you there with all the details TBH :).

Also thanks for showing a good usage for https://nixpk.gs/pr-tracker.html hadn't known this website before!

anund · 2021-12-02T13:33:51Z

@IvarWithoutBones does the suggested fix work for you like doronbehar?

IvarWithoutBones · 2021-12-02T16:07:48Z

That does seem to work, thank you for the great explanation! I think this can be closed again now :)

vcunat · 2021-12-02T18:00:01Z

It sounds a bit unpleasant that regular nixos-rebuild usage won't (ever?) fix this issue, though that would perhaps be a separate issue.

anund · 2021-12-03T02:06:47Z

The current reliance on bootctl to install and update the boot loader is the disconnect. From the store we can know it's a different bootloader than was built (different patch set/version) but bootctl just sees v249.5 on disk in the boot partition and v249.5 initiating an "update". bootctl chosing to noop in that case makes sense outside nixos. (*for normal users that don't normally update to the same version)

In retrospect waiting for v249.6 to be tagged and released upstream (it is now, v249.7 is tagged) should have avoided the problems Ivar and doronbehar encountered since bootctl would have seen a version increase and installed a new bios during the update call. ('course we didn't know long the .6 release would take and I wouldn't have found the bug in --install-bootloader without this issue)

vcunat · 2021-12-03T08:00:58Z

Oh OK, so it should at least get updated automatically on the next version bump.

flokli · 2021-12-03T22:46:09Z

There hopefully should be a 249.8 soon: systemd/systemd#21498 (comment)

FlorianFranzen · 2021-12-04T21:44:37Z

I am still having this issue on the latest nixos-unstable branch, even though I ran nixos-rebuild with --install-bootloader, bootctl install and bootctl update.

anund · 2021-12-05T13:23:57Z

This bug is machine dependent so there's an outside possibility your hardware isn't fixed by the change upstream. Please provide some more information @FlorianFranzen. Things that might help:

what sort of machine is it (the original upstream bug only reported a few)
nixos version (or various methods here https://discourse.nixos.org/t/how-to-see-what-commit-is-my-channel-on/4818/2 to double check what commit is actually being used for your local nix-channel)
bootctl --version what version of systemd is currently running
bootctl --path=<boot path, probably /boot> --status what version of systemd-boot is currently on disk

If nothing obvious pops out as a mistake you may need to follow the above comments to debug your machine more directly.

FlorianFranzen · 2022-01-03T16:04:27Z

I have been unable to resolve this issue so far. I tried various alternative patches without success. I will wait for the next systemd release before I invest any more time in this.

vcunat · 2022-01-03T16:53:03Z

Updated systemd is in the staging-next branch, and it has all Linux binaries already on cache.nixos.org.

vcunat · 2022-01-13T10:50:00Z

We now updated systemd to 249.7 on master since commit 76e5d23.

anund added the 0.kind: bug Something is broken label Oct 30, 2021

anund mentioned this issue Oct 31, 2021

nixos/systemd-boot: Fix installed version regexp #140046

Merged

5 tasks

veprbl added the 6.topic: systemd label Oct 31, 2021

anund mentioned this issue Nov 3, 2021

systemd: fix systemd-boot keyboard handling lockup #144422

Merged

12 tasks

vcunat closed this as completed in d216b21 Nov 22, 2021

vcunat reopened this Nov 26, 2021

vcunat closed this as completed Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Systemd-boot fails to recognize keyboard input on some systems #143847

Systemd-boot fails to recognize keyboard input on some systems #143847

anund commented Oct 30, 2021 •

edited

Loading

anund commented Oct 31, 2021

flokli commented Nov 1, 2021 •

edited

Loading

xaverdh commented Nov 1, 2021

flokli commented Nov 1, 2021

anund commented Nov 1, 2021 •

edited

Loading

anund commented Nov 2, 2021

anund commented Nov 2, 2021

IvarWithoutBones commented Nov 2, 2021

flokli commented Nov 3, 2021

anund commented Nov 3, 2021 •

edited

Loading

doronbehar commented Nov 26, 2021

IvarWithoutBones commented Nov 26, 2021

anund commented Nov 27, 2021 •

edited

Loading

doronbehar commented Nov 27, 2021

anund commented Nov 27, 2021 •

edited

Loading

anund commented Nov 27, 2021 •

edited

Loading

anund commented Nov 27, 2021

doronbehar commented Nov 27, 2021

anund commented Dec 2, 2021

IvarWithoutBones commented Dec 2, 2021

vcunat commented Dec 2, 2021

anund commented Dec 3, 2021 •

edited

Loading

vcunat commented Dec 3, 2021

flokli commented Dec 3, 2021

FlorianFranzen commented Dec 4, 2021

anund commented Dec 5, 2021 •

edited

Loading

FlorianFranzen commented Jan 3, 2022

vcunat commented Jan 3, 2022 •

edited

Loading

vcunat commented Jan 13, 2022

Systemd-boot fails to recognize keyboard input on some systems #143847

Systemd-boot fails to recognize keyboard input on some systems #143847

Comments

anund commented Oct 30, 2021 • edited Loading

Describe the bug

Steps To Reproduce

Expected behavior

Additional context

Notify maintainers

Metadata

anund commented Oct 31, 2021

flokli commented Nov 1, 2021 • edited Loading

xaverdh commented Nov 1, 2021

flokli commented Nov 1, 2021

anund commented Nov 1, 2021 • edited Loading

anund commented Nov 2, 2021

anund commented Nov 2, 2021

IvarWithoutBones commented Nov 2, 2021

flokli commented Nov 3, 2021

anund commented Nov 3, 2021 • edited Loading

doronbehar commented Nov 26, 2021

IvarWithoutBones commented Nov 26, 2021

anund commented Nov 27, 2021 • edited Loading

doronbehar commented Nov 27, 2021

anund commented Nov 27, 2021 • edited Loading

anund commented Nov 27, 2021 • edited Loading

anund commented Nov 27, 2021

doronbehar commented Nov 27, 2021

anund commented Dec 2, 2021

IvarWithoutBones commented Dec 2, 2021

vcunat commented Dec 2, 2021

anund commented Dec 3, 2021 • edited Loading

vcunat commented Dec 3, 2021

flokli commented Dec 3, 2021

FlorianFranzen commented Dec 4, 2021

anund commented Dec 5, 2021 • edited Loading

FlorianFranzen commented Jan 3, 2022

vcunat commented Jan 3, 2022 • edited Loading

vcunat commented Jan 13, 2022

anund commented Oct 30, 2021 •

edited

Loading

flokli commented Nov 1, 2021 •

edited

Loading

anund commented Nov 1, 2021 •

edited

Loading

anund commented Nov 3, 2021 •

edited

Loading

anund commented Nov 27, 2021 •

edited

Loading

anund commented Nov 27, 2021 •

edited

Loading

anund commented Nov 27, 2021 •

edited

Loading

anund commented Dec 3, 2021 •

edited

Loading

anund commented Dec 5, 2021 •

edited

Loading

vcunat commented Jan 3, 2022 •

edited

Loading