fscryptctl insert_key does not persist during switch_root

[embetrix]

I would like to encrypt my root partition using fscrypt.
I insert the key in initramfs:

fscryptctl insert_key < /tmp/key.data

then mount my rootfs:

mount -t ubifs ubi0:rootfs /mnt

Finally excuting switch_root:

exec switch_root /mnt /sbin/init

The last command fails since the key does not survive the switch_root probably because /proc & /sys are remounted.

Is there anyway to persist the key during this stage ?

[embetrix]

To mention that the root filesystem is using systemd as init, here is a log:

+ fscryptctl insert_key < /tmp/key.data
791af0e84aba1525
+ rm /tmp/key.data
+ keyctl show
Session Keyring
 648590972 --alswrv      0 65534  keyring: _uid_ses.0
 561932472 --alswrv      0 65534   \_ keyring: _uid.0
  68236785 --alsw-v      0     0       \_ logon: fscrypt:791af0e84aba1525
+ mount -t ubifs ubi0:rootfs /mnt
+ keyctl show
Session Keyring
 648590972 --alswrv      0 65534  keyring: _uid_ses.0
 561932472 --alswrv      0 65534   \_ keyring: _uid.0
  68236785 --alsw-v      0     0       \_ logon: fscrypt:791af0e84aba1525
+ exec switch_root /mnt /sbin/init 'console=ttymxc0,115200n8,115200' 'mtdparts=gpmi-nand:2m(spl),2m(uboot),1m(env),-(sys)' rw quiet 'ubi.mtd=3' 'root=ubi0:rootfs'


imx6ul login: root       

  
                                                         
root@imx6ul:~# keyctl show
Session Keyring
 753570088 --alswrv      0     0  keyring: _ses
 394877377 ----s-rv      0     0   \_ user: invocation_id

[josephlr]

So if you run any fscrypt (not fscryptctl) command you should get an error about how your user keyring is not in your session keyring. Without this you will not have access to any data (this is a kernel limitation).

To solve this issue, you can either:

• Setup pam_keyinit.so correctly in your PAM stack
• Manually run keyctl link @u @s
• Add KeyringMode=shared in the [Service] section of your Systemd unit (will only work if a process run by systemd needs access to an encrypted directory).

[embetrix]

@josephlr : Thanks for the hints:

when using keyctl link @U @s in my initramfs I can see:

During Initramfs initialization

$ keyctl show
Session Keyring
 367846887 --alswrv      0 65534  keyring: _uid_ses.0
 863703966 --alswrv      0 65534   \_ keyring: _uid.0
 663201869 --alsw-v      0     0   \_ logon: fscrypt:77e0e16fcb6fd0c6

When my rootfs is started

$ keyctl show @us
Keyring
 367846887 --alswrv      0 65534  keyring: _uid_ses.0
 863703966 --alswrv      0 65534   \_ keyring: _uid.0
 663201869 --alsw-v      0     0   \_ logon: fscrypt:77e0e16fcb6fd0c6

My partition is also decrypted and all files visible.

However this is not what I want, my final goal is to have an encryted rootfs decrypted in initramfs and boot from it using switch_root.

In this configuration systemd does not even start! (located in encrypted rootfs)

[embetrix]

I enabled systemd debugging in Kernel using

systemd.log_level=debug systemd.log_target=console

Before I do a switch_root, I can see that my rootfs is correclty decrypted

Kernel cmdline: console=ttymxc0,115200n8,115200 mtdparts=gpmi-nand:2m(spl),2m(uboot),1m(env),-(sys) systemd.log_level=debug systemd.log_target=console ubi.mtd=3 root=/dev/mmcblk0p2
Root device: /dev/mmcblk0p2
77e0e16fcb6fd0c6
Session Keyring
 680677289 --alswrv      0 65534  keyring: _uid_ses.0
  98031435 --alswrv      0 65534   \_ keyring: _uid.0
 338204520 --alsw-v      0   [    3.773680] UBIFS (ubi0:4): background thread "ubifs_bgt0_4" started, PID 133
  0   \_ logon: fscrypt:77e0e16fcb6fd0c6
[    3.797199] UBIFS (ubi0:4): recovery needed
[    3.868020] UBIFS (ubi0:4): recovery completed
[    3.872563] UBIFS (ubi0:4): UBIFS: mounted UBI device 0, volume 4, name "rootfs_update"
[    3.880660] UBIFS (ubi0:4): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    3.890660] UBIFS (ubi0:4): FS size: 187289600 bytes (178 MiB, 1475 LEBs), journal size 9396224 bytes (8 MiB, 74 LEBs)
[    3.901425] UBIFS (ubi0:4): reserved for root: 4952683 bytes (4836 KiB)
[    3.908111] UBIFS (ubi0:4): media format: w5/r0 (latest is w5/r0), UUID 822708DA-4C2F-4518-BFFA-386C46289E0A, small LPT model
drwxr-xr-x    2 root     root          8032 Jul 30  2019 bin
drwxr-xr-x    2 root     root           160 Jul 30  2019 boot
drwxr-xr-x    2 root     root           160 Jun 28  2019 dev
drwxr-xr-x   21 root     root          5536 Jul 30  2019 etc
drwxr-xr-x    3 root     root           256 Jul 30  2019 home
drwxr-xr-x    7 root     root          5536 Jul 30  2019 lib
drwx------    2 root     root           160 Jul 30  2019 lost+found
drwxr-xr-x    2 root     root           160 Jul 30  2019 media
drwxr-xr-x    2 root     root           160 Jun 28  2019 mnt
drwxr-xr-x    3 root     root           256 Jul 30  2019 opt
dr-xr-xr-x    2 root     root           160 Jan  1 00:00 proc
-rw-r--r--    1 root     root         80177 Jul 30  2019 rootfs.sha256
drwxr-xr-x    2 root     root           160 Jun 28  2019 run
drwxr-xr-x    2 root     root          5920 Jul 30  2019 sbin
dr-xr-xr-x    2 root     root           160 Jan  1 00:00 sys
drwxrwxrwt    2 root     root           160 Jun 28  2019 tmp
drwxr-xr-x   10 root     root           928 Jul 30  2019 usr
drwxr-xr-x   10 root     root          1312 Jul 30  2019 var
drwxr-xr-x    6 root     root           640 Jul 30  2019 www

It looks like systemd is started but the encyption is somehow lost, I see many errors with

systemd-journald.service: Failed to execute command: Required key not available
systemd-journald.service: Failed at step EXEC spawning /lib/systemd/systemd-journald: Required key not available
systemd-udev-trigger.service: Failed to execute command: Required key not available
systemd-udev-trigger.service: Failed at step EXEC spawning /bin/udevadm: Required key not available
systemd-tmpfiles-setup-dev.service: Failed to execute command: Required key not available
systemd-tmpfiles-setup-dev.service: Failed at step EXEC spawning /bin/systemd-tmpfiles: Required key not available

[josephlr]

@embexus your input has helped me root-cause this. Thank you.

Encryption keys in fscrypt and fscryptctl used the session keyring to hold encryption keys. However, this presents a big problem if sessions are reset (like when running switch_root). When the session is reset, the encryption key is garbage-collected by the kernel, and access is lost.

fscrypt solved this problem by switching to using the user keyring with v0.2.0

However, this change was not added to fscryptctl which is a bug.

[josephlr]

As a workaround, fscrypt does the right thing, so you can just use that.

[embetrix]

@josephlr : thanks for the hint, but for my small embedded system GO application is NoGo, way too big.

do you think this change in fscyptctl will do it ?

diff --git a/fscryptctl.c b/fscryptctl.c
index 9c6575f..b089ea5 100644
--- a/fscryptctl.c
+++ b/fscryptctl.c
@@ -91,6 +91,7 @@ struct fscrypt_policy {
 typedef int32_t key_serial_t;
 #define KEYCTL_GET_KEYRING_ID 0     /* ask for a keyring's ID */
 #define KEY_SPEC_SESSION_KEYRING -3 /* current session keyring */
+#define KEY_SPEC_USER_KEYRING   -4 /* - key ID for UID-specific keyring */
 
 key_serial_t add_key(const char *type, const char *description,
                      const void *payload, size_t plen, key_serial_t ringid) {
@@ -276,7 +277,7 @@ static int insert_logon_key(const uint8_t key_data[FS_MAX_KEY_SIZE],
   // We cannot add directly to KEY_SPEC_SESSION_KEYRING, as that will make a new
   // session keyring if one does not exist, rather than adding it to the user
   // session keyring.
-  int keyring_id = keyctl_get_keyring_ID(KEY_SPEC_SESSION_KEYRING, 0);
+  int keyring_id = keyctl_get_keyring_ID(KEY_SPEC_USER_KEYRING, 0);
   if (keyring_id < 0) {
     return -1;
   }

[josephlr]

@embexus that will fix the issue. I would accept such a patch. I would probably want to also add some checking that the user keyring is accessible from the current session keyring (to prevent issues like this in the future). It would be a change to fscryptctl's behavior, but as there isn't an official release yet, I'm not super concerned.

@josephlr : thanks for the hint, but for my small embedded system GO application is NoGo, way too big.

I've recently considered adding additional functionality to fscryptctl to support fscrypt (i.e. emulating the upcoming fscrypt kernel patches on older kernels). As this would be more complex behavior, I was considering adding functionality with either C++ or Rust.

However, I'd like to know more about your constrains for using fscryptctl vs fscrypt. fscryptctl is 27 kB and fscrypt is 4.7 MB (when built on my computer). Are your constraints only for binary size?

[embetrix]

@josephlr I testet the changes using fscryptctl with USER Keyring instead of Session Keryring (cf. Patch above) however still getting the same error!

I did also the setup with my rootfs using eCryptfs and it did work !

I suspect there is an issue with systemd and keys of type logon.

I have size constraints for my embedded system, initramfs should be a small as possible.
fscryptctl is a native C application I can easily used or linked against my application (MIT License is great).

fscryptctl compiled for ARM is around 17kB (stripped)
fscrypt GO Application has many dependencies is a bit tidy to cross compile and very big > 4MB.

fscryptctl is definitely much better suited for embedded systems and should be further maintained ;-)

[josephlr]

I would like to encrypt my root partition using fscrypt.

I don't know how I missed this part. Are you trying to encrypt the whole partition, or just some of the directories on the partition. Encrypting the whole partition is better suited to tools like dmcrypt and cryptsetup. Encrypting some of the files on a partition (or having multiple encryption keys for one partition) is a good job for fscrypt/fscryptctl.

I suspect there is an issue with systemd and keys of type logon.

This is possible, but it's most likely not the type of the keys, but rather which keyring they are in. With that patch, the key will be now always be put in the user keyring (in this case, the root user's keyring). However, you still have to have the root user's keyring linked into your session keyring for everything to work.

This means that any software either needs to either:
a) be invoked from a session that links in the user keyring (say using pam_keyinit.so)
b) if the software is creating new sessions (like systemd does), you will have to set KeyringMode=shared in any systemd units that access files encrypted with filesystem encryption.

This systemd limitation makes setup harder when using filesystem encryption with system files. This is why filesystem encryption is usually just used with user data (like home directories) or data directories (say a media or documents directory). We've been trying to fix it, but the real solution is probably this kernel patchset by @ebiggers.

fscryptctl is a native C application I can easily used or linked against my application (MIT License is great).

Note that fscrypt and fscryptctl are actually under Apache 2.0 (not that it's really that much different from MIT, still a permissive license).

fscryptctl is definitely much better suited for embedded systems and should be further maintained ;-)

No worries on that front, I was just considering adding functionality to fscryptctl, and wanted to make sure that it wouldn't be too big (or whatever) for your use case. I'll keep both the ARM cross-compiling concerns and initramfs size constraints in mind. I'll make sure that whatever changes I make don't break that sort of thing.

[embetrix]

I would like to encrypt my root partition using fscrypt.

I don't know how I missed this part. Are you trying to encrypt the whole partition, or just some of the directories on the partition. Encrypting the whole partition is better suited to tools like dmcrypt and cryptsetup. Encrypting some of the files on a partition (or having multiple encryption keys for one partition) is a good job for fscrypt/fscryptctl.

@josephlr, I'm encrypting my whole rootfs. As I wrote before I use a Nand Flash with UBIFS so dmcrypt is not suited. eCryptfs did the Job, but I prefere using fscrypt/fscryptctl as it's lightweight, has better performance and much easier to setup.

I suspect there is an issue with systemd and keys of type logon.

This is possible, but it's most likely not the type of the keys, but rather which keyring they are in. With that patch, the key will be now always be put in the user keyring (in this case, the root user's keyring). However, you still have to have the root user's keyring linked into your session keyring for everything to work.

I added

keyctl link @u @s 

But same results.

This means that any software either needs to either:
a) be invoked from a session that links in the user keyring (say using pam_keyinit.so)

I don't use PAM at all (disabled in Systemd)

b) if the software is creating new sessions (like systemd does), you will have to set KeyringMode=shared in any systemd units that access files encrypted with filesystem encryption.

This systemd limitation makes setup harder when using filesystem encryption with system files. This is why filesystem encryption is usually just used with user data (like home directories) or data directories (say a media or documents directory). We've been trying to fix it, but the real solution is probably this kernel patchset by @ebiggers.

But it does work with eCryptfs, what is the difference ?

fscryptctl is a native C application I can easily used or linked against my application (MIT License is great).

Note that fscrypt and fscryptctl are actually under Apache 2.0 (not that it's really that much different from MIT, still a permissive license).

fscryptctl is definitely much better suited for embedded systems and should be further maintained ;-)

No worries on that front, I was just considering adding functionality to fscryptctl, and wanted to make sure that it wouldn't be too big (or whatever) for your use case. I'll keep both the ARM cross-compiling concerns and initramfs size constraints in mind. I'll make sure that whatever changes I make don't break that sort of thing.

will be great, thanks for maintaining this Software ;-)

[josephlr]

@josephlr, I'm encrypting my whole rootfs. As I wrote before I use a Nand Flash with UBIFS so dmcrypt is not suited.

Ohhhh, thats interesting. I've actually not heard of this particular drawback of dmcrypt. Do you have any info about the specifics? Is it because of not wanting to encrypt the FS metadata?

keyctl link @u @s 

Doing this should fix issues in the session when you run this command, but will not fix the issue in other sessions that don't have the user keyring linked.

But it does work with eCryptfs, what is the difference?

I think it's a difference in the in-kernel implementation. As ecryptfs is a stacked filesystem, the key (i think it's called the FEFEK) is read at mount time (from the kernel keyring), but then is persisted for the duration mount by the kernel. But fscrypt is a not a stacked filesystem. It's just part of whatever filesystem you are using, where encrypted directories behave like eCryptfs mount points. As mount/unmount cannot be used to manage lifetimes anymore, the key lifetime (in the keyring) is used instead.

The choice to use the kernel keyring at all (which is buggy, complex, and has DOS vectors) was a mistake by both eCryptfs and fscrypt, which the linked kernel patches tries to fix. I would try seeing if there's a way to just have systemd use KeyringMode=shared globally, as that should fix the problem (you might also still need to run keyctl link @u @s on login).

[ebiggers]

Note that Linux 5.4 added support for a new encryption policy version and a new way of managing filesystem encryption keys. When used, the encryption keys are added to a filesystem keyring rather than a session keyring; this avoids problems where processes can't access encrypted files that are supposed to be unlocked. #16 will add support for this to fscryptctl. As the new way is much better, I don't plan to implement workarounds in fscryptctl to make the old way work better than it did.

[josephlr]

@embexus now that V2 policies have been added in #16 (and support for V1 policies have been removed) I'm going to close this. Using V2 policies should resolve all of the issues you mentioned above (as well as other security/usability improvements).

Please reopen if the recent changes don't resolve your issues.