Kernel panic when trying to boot BTRFS+LUKS system (Arch Linux)

[0xSigi]

Hi, I was preparing a new machine and decided to give booster a try, since I wanted the system to be as minimal and as snappy as possible.. I've set it up as usual with the btrfs layout + luks and systemd as my bootloader choice.

First issue when trying to boot after pacstrapping and preparing the system - kernel panic attempting to kill init - Initially I thought that maybe this is an issue with the setup or some weird hardware issue so decided to play around with the config, to maybe add some modules manually (btrfs for example) and try building host specific and universal images.. In all cases no matter the config I ended up with kernel panic..

Running short on time I decided to try mkinitcpio and of course it booted as it always did no problems here.. This got me thinking that maybe this was due to some modules not being added and I needed a fully booted kernel to build a proper init image, this was unfortunately not the case.. Even when building images from a fully working workstation with every possible kernel and/or booster option did not yield a single proper boot.. So far for the last two or maybe even three days I did the following:

• Tried creating image from arch-chroot when staging the machine - kernel panic
• Tried creating image from fully working system booted with mkinitcpio (same configs, same or slightly modified boot profile for booster) - kernel panic
• Tried playing around with all possible booster options, building host specific and universal image - kernel panic every time
• Tried building the init image for different kernels available for Arch - Linux, Linux-lts, Linux-zen, Linux-TKG (custom build for my needs) - kernel panic every time

So no matter what I try it always ends up the same way - kernel panic right after it tries to unlock my luks partition. I wonder if this might be related to the setup of luks, since I don't run with the "defaults" and have custom options for most of the parameters and unlocking the drive takes approx 30-35 seconds - maybe there's some race condition going on here (I will try to do some test by redoing the parameters, seeing if this solves the issue)? I am attaching a video I took when attempting to boot, apologies for crap quality..

If there's anything else I can provide or test, please let me know as I'm very interested in not only trying to fix this, but run booster since I want (need) yubikey unlock for my luks drives.

Cheers!

https://imgur.com/a/2tTgjqo

[anatol]

So no matter what I try it always ends up the same way - kernel panic right after it tries to unlock my luks partition. I wonder if this might be related to the setup of luks, since I don't run with the "defaults" and have custom options for most of the parameters and unlocking the drive takes approx 30-35 seconds - maybe there's some race condition going on here (I will try to do some test by redoing the parameters, seeing if this solves the issue)? I am attaching a video I took when attempting to boot, apologies for crap quality..

The init process got panicked in luks-related function (at return v.SetupMapper(mapping.name) call in particular) so my guess is indeed somehow related to the LUKS parameters.

Is there any chance you can get previous log lines, those are the most informative ones. Maybe you can modify screen resolution with vga= boot parameter?

Also if you provide the way you setup your LUKS partition I'll be glad to add it as a test case to luks.go library to make sure it properly handled.

[0xSigi]

Also if you provide the way you setup your LUKS partition I'll be glad to add it as a test case to luks.go library to make sure it properly handled.

Sure, here it is, nothing "military" grade, but I simply don't like to run with defaults..

cryptsetup luksFormat -c camellia-xts-plain64 -s 512 -h sha512 -i 8000 --pbkdf argon2id --pbkdf-memory 4100000 /dev/disk/by-partlabel/cryptsystem

As for changing the screen resolution while booting, I'll see what I can do, and how readable it gets.

[0xSigi]

So, there's no way (at least I have not found one) to change the boot resolution.. vga option does nothing here, either via ask, or trying to set a specific value, vconsole to make the font smaller also does not work since it triggers only after sucesfully decrypting the drive.. Since this is a new device, and to make sure this is indeed related to the actual setup of luks, I'll reencrypt the drive lowering the parameters and maybe trying to change the cipher (don't remember if reencrypt allows for it) and will report back, later today. Cheers

[anatol]

cryptsetup luksFormat -c camellia-xts-plain64 -s 512 -h sha512 -i 8000 --pbkdf argon2id --pbkdf-memory 4100000 /dev/disk/by-partlabel/cryptsystem

luks.go library does not support camelia block cipher. It needs to be added there first. I will look at it.

[anatol]

BTW why camelia, what is its advantage over aes?

[0xSigi]

luks.go library does not support camelia block cipher. It needs to be added there first. I will look at it.

Thanks, hopefully this won't be too much work to add it to the code..

BTW why camelia, what is its advantage over aes?

On paper, there's none.. But it's a non standard cipher that is not AES, for some this is a good enough argument. Thou I won't be bringing up my tinfoil hat, and will just say that I don't like to use the "good enough" standards for my devices especially when it comes to encryption. And yes, I know that this is a very niche configuration, probably overkill for 95+ % of people using LUKS, but it works (worked?) fine for me over the years so I sticked with it :)

[anatol]

I just pushed a fix for panic you see to wip branch. If you build wip branch then all you see at the console would be a warning something like camillia cipher is not supported which is a much more friendly way to tell the error than the panic.

camillia cipher functionality support still needs to be added to luks.go project.

[anatol]

I added camellia support to luks.go library. It is able to unlock LUKS partition configured with your parameters.

I pushed an updated booster to wip branch. Is there any chance you can rebuild booster from wip and see if it fixes your issue?

[0xSigi]

Thanks for the quick response to this, yeah I'll try to build it and see how it works. Cheers

[0xSigi]

I think I spoke too quick ;) While I don't have problems building packages manually, adding branches to the mix is apparently too much for me :D Spend last hour trying to figure out how to build it via makepkg against specific branch but can't wrap my head around.. Some directions (or a manual I can reference since pkgbuild wiki does not mention this) would be helpful to get this going..

[anatol]

The easiest way for you would be to install booster-wip-git branch from AUR https://aur.archlinux.org/packages/booster-wip-git

Once you are done testing it - reinstall the official package back.

[0xSigi]

That did not work, because of:

install: cannot stat 'generator/booster': No such file or directory

However pointing booster-git to the wip branch via source in PKGBUILD (git+https://github.com/anatol/booster#branch=wip) allowed me to build the package just fine. Unfortunately while I do not get kernel panics anymore, I still can't boot, attaching latest screenshot showing bit more debug info, however it still does not produce anything to either journal or dmesg..

[anatol]

It probably comes from the fact that camellia is distributed as a separate kernel module at Arch. booster does not know what cipher will be used and does not include all possible options in the image. You need to tell it that camelia is needed.

Either use universal mode that adds a bunch of kernel modules to the image. For that add universal: true to your /etc/booster.yaml

Or specify this module explicitly with modules: camellia in your /etc/booster.yaml

Then regenerate the image and reboot.

[0xSigi]

Turns out, I can't just specify "camellia" as a module since module with such name does not exist. Instead it seems to be broken into several modules as per listing:

/lib/modules/5.19.11-zen1-1-zen/build/include/config/CRYPTO_CAMELLIA
/lib/modules/5.19.11-zen1-1-zen/build/include/config/CRYPTO_CAMELLIA_AESNI_AVX2_X86_64
/lib/modules/5.19.11-zen1-1-zen/build/include/config/CRYPTO_CAMELLIA_AESNI_AVX_X86_64
/lib/modules/5.19.11-zen1-1-zen/build/include/config/CRYPTO_CAMELLIA_X86_64
/lib/modules/5.19.11-zen1-1-zen/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko.zst
/lib/modules/5.19.11-zen1-1-zen/kernel/arch/x86/crypto/camellia-aesni-avx2.ko.zst
/lib/modules/5.19.11-zen1-1-zen/kernel/arch/x86/crypto/camellia-x86_64.ko.zst
/lib/modules/5.19.11-zen1-1-zen/kernel/crypto/camellia_generic.ko.zst

Will test with universal image if it works, and then with seperate modules untill I find the ones needed.. Was wondering if I can expand them by say typing camellia*, CRYPTO_CAMELLIA*? Trying to do it like I wrote, does not work and breaks image generation with error that module camellia* does not exist..

[anatol]

The kernel module name resolution in Linux a bit tricky. It has so called module aliases when the same module has multiple names/aliases.

For example "camelia" aliased to its implementations like this one https://github.com/torvalds/linux/blob/5bfc75d92efd494db37f5c4c173d3639d4772966/crypto/camellia_generic.c#L1072 so I believe "camellia" will resolve to "camelia_generic.ko" and included to the image.

Universal image will include all crypto libraries.

[0xSigi]

Name resolution is indeed weird, specying camellia or camellia-generic, always tripped the image generation with same error that the module XXX does not exist.. I tried building universal image, and image containing all camellia modules added manually, unfortunately I still end up with same error message I send via screenshot yesterday..

Anything else I could try here? not stripping the modules?

[anatol]

crypt: Error allocating crypto tfm (-ENOENT) most likely means that some kernel module is missing in the image. If you have a chance could you please post the list of your universal booster and initramfs image that works for you? Use booster ls image.img. Comparing the content of these images will give more hints on what is going on there.

[anatol]

not stripping the modules?

It is worth giving a try. Use strip: false in your config.

[deathtrip]

I have the same problem, with the twofish cipher, GRUB as the bootloader, and LUKS on lvm.
You asked for the previous log lines, so here are mine.

[anatol]

@deathtrip yeah, it is the same issue. If you get the latest patches from wip then you'll not see the panic message anymore - the error is handled properly now.

It looks like the problem is with loadable crypto modules. cryptsetup/dmsetup or kernel has implemented their unobvious modules loading internally. And I cannot find where all those custom modules (like camellia) are loaded.

Error allocating crypto tfm is a kernel error message that says "something is missing". But it is pretty useless to understand what exactly is missing. Unless someone has better ideas I am considering patching the kernel to provide a better error message for this error case.

`

[0xSigi]

OK, so I had a moment to list the modules loaded in both initram images, turns out booster loads just one, whereas mkinitcpio loads 4.. On top of that, booster seems to have issues finding the additional modules in the first place, please refer to the left hand side of the screen where you can find the error message and me listing the available modules, showing it is where it is supposed to be.. I am not sure why this happens, unfortunately my programming knowledge is close to none, so while I can help do all the testing you need, at this point I don't have any ideas how to implement it inside the code..
booster_zen_modules.txt
mkinitcpio_zen_modules.txt

[anatol]

@IronsideSec as a workaround try to add modules_force_load: xts,camellia_generic,camellia_aesni_avx2,camellia_aesni_avx_x86_64,camellia_x86_64 to the config file and then rebuld the image. Does it help?

[0xSigi]

While it does not help me boot with booster, I think it helped potentially find the culprit of the issue, I am attaching a screenshot taken just now. I did try to forceload camellia kernel modules, and two things happened, one it complained that the module camellia does not exist, at the same it it loaded it just fine to the image, however with changed name.. I have listed how mkinitcpio adds the modules and how booster does it.. It seems like the modules are not found and kernel panicks simply because booster changes the names of these modules?

[0xSigi]

I've tried every possible combination, what I don't understand why it complains that the module does not exist yet when you list modules via booster ls, same module is listed.. Anyway I've decided to test if aes-xts-plain64 will work fine. Will report soon.. Thou this does not fix the initial issue since I'd like to stick with camilla as my cipher.

[anatol]

@IronsideSec wait a bit, I have patches on the way that might fix your original issue with camellia block cipher.

[anatol]

@IronsideSec I just pushed changes to wip branch that allows you to use xts-camellia mode either

1. by using universal mode
2. or by specifying modules: crypto-xts,crypto-camellia in your config file

module aliases are supported by modules and modules_force_load parameters.

[0xSigi]

Thanks a lot for your help here, will test it and report back later tonight. Cheers

[0xSigi]

Hi @anatol I can confirm this works now! booting is blazing fast and I can finally easily unlock the drive with a yubikey instead of a very long passphrase :) Much appreciated, thank you.

[deathtrip]

I tried building the AUR package and also got
install: cannot stat 'generator/booster': No such file or directory
The pkgbuild already points to git+https://github.com/anatol/booster#branch=wip as source, which was the solution suggested above.
I had to go to /tmp/makepkg/booster-wip-git/src/booster/generator and rename the generator executable to booster for it to build. So that pkgbuild needs to be updated.
I will test if it works and report tomorrow.

[deathtrip]

I built a universal image and added the modules to the config file.

Here are my findings:

1. Boosters images are incompatible with kernel command-line parameters: module.sig_enforce=1 and lockdown=confidentiality. Each one of them causes a bunch of finit(modulename): key was rejected by service errors when loading modules. And the loading seems to stop after that, no password prompt.

2. When i remove said parameters and get to the unlocking stage i get a Unknown cipher: twofish error, after entering password.
   However the modules are present in the image when i do a booster ls.
   usr/lib/modules/twofish_x86_64.ko usr/lib/modules/twofish_avx_x86_64.ko usr/lib/modules/twofish_x86_64_3way.ko usr/lib/modules/twofish_common.ko usr/lib/modules/twofish_generic.ko

3. At least the kernel panic is gone.

[0xSigi]

@deathtrip while waiting for Anatol to look into the issue, try this to see if this fixes issue nr.1 as per documentation

strip is a boolean flag that enables ELF files stripping before adding it to the image. Binaries, shared libraries and kernel modules are examples of ELF files that get processed with strip UNIX tool.

This options is not compatible with signed modules. If you see booster: finit(crc32,generic): key was rejected by service boot error please set the strip config option to false.

Issue nr 2 most probably require same treatment as with the camellia-xts cipher, I don't think these are being loaded/used properly by booster similar to camellia.

[anatol]

@deathtrip, answering your comments

1. Please make sure you do not use stripping with booster. module signing works with unstripped modules only. If it does not help, then please provide more information on how to reproduce the issue. What OS do you use? What kernel package? Any additional config steps...
2. Unknown cipher: twofish that was a problem with luks.go that is fixed now. Please pull the latest changes from wip and try it again.

[anatol]

A note about twofish. The golang crypto package says

Deprecated: Twofish is a legacy cipher and should not be used for new applications.
Also, this package does not and will not provide an optimized implementation. 
Instead, use AES (from crypto/aes, if necessary in an AEAD mode like crypto/cipher.NewGCM) or XChaCha20-Poly1305 (from golang.org/x/crypto/chacha20poly1305).

[anatol]

tried building the AUR package and also got
install: cannot stat 'generator/booster': No such file or directory

The booster-wip-git AUR package is fixed now

[deathtrip]

After updating the AUR package and disabling module stripping, i'm finally able to use host specific images to unlock my system.
Thank you guys for the help.

Also as booster gets more users, it's likely they too will encounter the problem of unsupported ciphers/hashes.
I think you should consider adding support for the popular ones, that have been out for a long time. The whirlpool hash function and serpent cipher come to mind as the most likely candidates. Maybe also a list of supported ciphers and hashes in the manpage.

[anatol]

@deathtrip it is great to hear this news.

I filed a ticket to review/add support for more algorithms anatol/luks.go#8