Key sharing for bcachefs and ecryptfs

[hyperfekt]

Issue description

Since systemd version 233 each system service gets their own kernel keyring. This prevents disk decryption in cryptsetup, ecryptfs and bcachefs from sharing the keys with other services/instances that need them, making the volumes unmountable.
(A bcachefs ([GUID]): error requesting encryption key message is added to the kernel message buffer when attempting to do so.)
In version 235 (which we would need to update to) this is fixed by making available a new setting 'KeyringMode' for unit files, which can be set to 'shared' to achieve the previous behavior. A fix for cryptsetup is already included in systemd's cryptsetup-generator. Now it needs to be added to the other filesystems in order to restore proper decryption functionality.
Unfortunately my expertise with systemd is practically nil, so I could use some help - first of all I was not able to determine where or if the unit files for these two are actually generated.

Relevant:
Arch Linux discussion
the KeyringMode change
the cryptsetup-generator change

Steps to reproduce

1. add boot.supportedFilesystems = [ "bcachefs" ]; to configuration.nix and nixos-rebuild
2. create new encrypted bcachefs volume with bcachefs format --encrypted /dev/sdXX
3. enter a passphrase twice
4. unlock the volume with bcachefs unlock /dev/sdXX
5. enter the passphrase given earlier
6. attempt to mount with mount -t bcachefs /dev/sdXX /mnt

Technical details

• system: "x86_64-linux"
• host os: Linux 4.11.0, NixOS, 18.03pre121094.9c048f4fb6 (Impala)
• multi-user?: no
• sandbox: yes
• version: nix-env (Nix) 1.11.15
• channels(root): "nixos-18.03pre121094.9c048f4fb6"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[hyperfekt]

I stumbled upon this discussion of the same issue by @intelfx .
Apparently there is not much we can do here, since mount can be launched in any context and will not have the KeyringMode=shared property?
Meaning the fix here for bcachefs is to wait for an update to bcachefs-tools and then switch to that, if I'm not mistaken. /cc @koverstreet
Not sure what the ideal solution for ecryptfs is, however, although I understand Arch Linux implemented something to fix it.

[hyperfekt]

Turns out the analogue issue for ecryptfs was #27574, and the fix for that (#30333) also fixes the issue for bcachefs (tested with 17.09).
If anyone who understands more about the issue confirms that the fix is the correct solution for this problem, even with systemd v235, this can probably be closed as a duplicate.

[eternaleye]

I suspect it's not the correct solution, as systemd/systemd#6342 reverted linking the user keyring to the session keyring for all services (as making pam_keyinit.so default does) on security grounds.

I've filed upstream issue systemd/systemd#8159 for a more targeted solution, matching the approach taken in systemd/systemd#6832 for services.

[Chiiruno]

This should be fixed now by #37647 and 361bd59...2d2ab94 which upgraded systemd to 238.

[Chiiruno]

Or so I thought, but upon further attempts at a new install, it appears the same old problem is appearing for bcachefs.
dmesg | grep bcachefs
[ 1411.490679] bcachefs: bch2_fs_journal_init()
[ 1411.490684] bcachefs: bch2_fs_journal_init() ret 0
[ 1411.490685] bcachefs: bch2_fs_btree_cache_init()
[ 1411.490785] bcachefs: bch2_fs_btree_cache_init() ret 0
[ 1411.490786] bcachefs: bch2_fs_encryption_init()
[ 1411.542552] bcachefs (1808534f-da16-4882-ac6e-8d4925673f0e): error requesting encryption key: -126
[ 1411.542555] bcachefs: bch2_fs_encryption_init() ret -126

All the needed keys ARE in the keyring
keyctl list @u
8 keys in keyring:
565092892: --alsw-v 0 0 logon: bcachefs:1808534f-da16-4882-ac6e-8d4925673f0e
1045826354: --alswrv 0 0 user: bcachefs:adfedea3-0f2a-41be-a692-44fdb47db68a
146276675: --alsw-v 0 0 logon: bcachefs:adfedea3-0f2a-41be-a692-44fdb47db68a
1072420447: --alswrv 0 0 user: bcachefs:d8ded406-2657-41fe-b07b-f4317a3b442c
194219927: --alsw-v 0 0 logon: bcachefs:d8ded406-2657-41fe-b07b-f4317a3b442c
351605712: --alswrv 0 0 user: bcachefs:eacfd316-7c70-4cf6-9352-1a7ee8a7e27e
369762748: --alsw-v 0 0 logon: bcachefs:eacfd316-7c70-4cf6-9352-1a7ee8a7e27e
835897310: --alswrv 0 0 user: bcachefs:1808534f-da16-4882-ac6e-8d4925673f0e

I double checked, and I am in fact using systemd version 238.

I created the bcachefs partition with mkfs.bcachefs --compression_type lz4 --encrypted -L root --discard -t 0 /dev/nvme0n1p4 < /root/crypto/root.key
I unlocked it with bcachefs unlock /dev/nvme0n1p4 < /root/crypto/root.key
I attempted to mount it with mount -t bcachefs /dev/nvme0n1p4 /mnt
The error message I got was mount: /mnt: mount(2) system call failed: Cannot allocate memory.
I do have a separate dmcrypt partition open, but I tested this without it open and the same problem exists.

[vcunat]

Eh, after skimming this and linked tickets, I don't know if the problem got solved (by systemd-238 bump) or not :-/ EDIT: from comments I'd think it is not really solved, but fpletz closed this.

[Chiiruno]

...Why is this closed?
I'll test out bcachefs on a USB I guess, but I didn't see anything there that would have fixed this.

[Chiiruno]

Yeah, that didn't work. Still the same problem, can't request the encryption key and it runs out of memory.
That was on a staging USB with systemd 238.
New issue here: #38366

[Chiiruno]

Closed the new issue since this is back up.

[Chiiruno]

Still getting the same error, even though we're on 239. (or at least should be, for tests)
mount: /tmp/mnt: mount(2) system call failed: Cannot allocate memory.
Related to #45774

[Chiiruno]

cc @vcunat @fpletz
Systemd should be updated for this, but it's still not working.
Is there something in the repo blocking this?

[vcunat]

I'm not aware of anything, but I don't think I know the related stuff well.

[hyperfekt]

After talking to Kent Overstreet, the author of bcachefs, I know that there will not be a fix in the forseeable future.
I suggest using #49506 to replicate the change from Arch Linux for systems with bcachefs enabled only, possibly constrained to single user installations only to limit the security implications (whose accuracy I frankly don't entirely understand).

[ZoomRmc]

Bumped into this issue while installing a new system from custom-built image. Systemd v239. Tried with lz4 compression and reformatted without, both times getting the same error.

[hyperfekt]

Here is a workaround, with unevaluated security implications:

options.security.pam.services = with lib; mkOption {
    type = types.loaOf (types.submodule {
      config.text = mkDefault (mkAfter "session required pam_keyinit.so force revoke");
    });
  };

[Chiiruno]

How does this have security implications?
I heard something about limiting it to single user installs, but I'm not entirely sure why this would be an issue.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[yorickvP]

Still a thing

[stale]

I marked this as stale due to inactivity. → More info

[Chiiruno]

Still important to me.

[Madouura]

Why is this a problem in a nixos vm but not on real hardware?

[Madouura]

I think we can now safely close this as encryption for bcachefs works on bare metal without any workarounds, and our tests can now call keyctl to work around a particular peculiarity.

[Madouura]

Never mind, this has managed to rear it's ugly head again.
Related: koverstreet/bcachefs#374
It would appear that on bare-metal (non-usb) everything works fine, however on a nixos installation usb, I get this issue.

[Madouura]

The issue is a bad interaction between nixos stable (21.11) and the parts of nixos unstable (master) from using an override for the unstable version of bcachefs pulls.
I do need to do keyctl link @u @s in the usb installation environment to get it to work, however.

[obadz]

Here is a workaround, with unevaluated security implications:

options.security.pam.services = with lib; mkOption {
    type = types.loaOf (types.submodule {
      config.text = mkDefault (mkAfter "session required pam_keyinit.so force revoke");
    });
  };

This issue and proposed solution seems related to #27574 (comment)

[clonejo]

Still an issue for me, i am running into this while trying to do a manual install with a freshly built unstable boot image. @Madouura's workaround worked out, luckily:

env -f '<nixpkgs>' -iA keyutils
keyctl link @u @s
bcachefs unlock …
mount …
nixos-install
reboot
# profit :)

[onny]

Bug still present in NixOS 23.05. Added a note about this to the NixOS wiki https://nixos.wiki/wiki/Bcachefs#NixOS_installation_on_bcachefs

[kennystrawnmusic]

Workaround works for me, however I need to hit "Esc" a bunch of times when the Plymouth boot screen appears; is there a Nix-specific way to have plymouth ask-for-password run at startup?

[onny]

Bug still present in NixOS 24.05. Added a note about this to the NixOS wiki https://nixos.wiki/wiki/Bcachefs#NixOS_installation_on_bcachefs