Update intramfs after kernel update

[NicoHood]

Hello,
I encrypted my raspberry root filesystem with this steps:
https://github.com/NicoHood/NicoHood.github.io/wiki/Raspberry-Pi-Encrypt-Root-Partition-Tutorial

Now I did a kernel update and after disk decryption the initramfs cannot find the old kernel anymore. This means the initramfs was not regenerated on a kernel update.

shiftplusone from the irc chat told me that this line is probably causing the issue:
https://github.com/RPi-Distro/firmware/blob/debian/debian/raspberrypi-kernel.postinst#L257

I do not really know the whole technique behind kernel updates and initramfs, but I suggest to add those regeneration scripts if initramfs is used. Because personally I have locked out of my raspberry completely. I also have trouble to fix this issue now, hopefully a chroot will do it.

[popcornmix]

ping @XECDesign
Also might be worth enabling issues on https://github.com/RPi-Distro/firmware

[XECDesign]

Enabled issues there

In the past, we didn't call the kernel hooks at all. When I split the kernel out to a separate packed, I added hooks to support DKMS, but left initrd generation disabled. It was untested and I haven't seen anybody request it. Will take a look at it some time this week.

[NicoHood]

I try to fix my current raspbian installation from another raspbian. I plugged in the broken one in the sd card slot and used the following commands:

Edit: A kernel update on the host raspberry to the same kernel solved the issue luckly.

# On the host
    1  mkdir -p /mnt/chroot/boot
    3  cryptsetup luksOpen /dev/sda2 crypt
    5  mount /dev/mapper/crypt /mnt/chroot/
    6  mount /dev/sda1 /mnt/chroot/boot/
    7  mount -t proc none /mnt/chroot/proc
    8  mount -t sysfs none /mnt/chroot/sys
    9  mount -o bind /dev /mnt/chroot/dev
   10  mount -o bind /dev/pts /mnt/chroot/dev/pts
   12  LANG=C chroot /mnt/chroot/

# On the chroot (borken) raspbian
   12  apt-get update
   13  ls /boot
   14  apt-get dist-upgrade
   16  update-initramfs -u
   17  mkinitramfs -o /boot/initramfs.gz

But I get the following error:

WARNING: missing /lib/modules/4.4.9-v7+
Ensure all necessary drivers are built into the linux image!
depmod: ERROR: could not open directory /lib/modules/4.4.9-v7+: No such file or directory
depmod: FATAL: could not search modules: No such file or directory
depmod: WARNING: could not open /var/tmp/mkinitramfs_UfnGVI/lib/modules/4.4.9-v7+/modules.order: No such file or directory
depmod: WARNING: could not open /var/tmp/mkinitramfs_UfnGVI/lib/modules/4.4.9-v7+/modules.builtin: No such file or directory

The host is running the old kernel which was used to generate the initramfs (which is now broken). The broken raspbian has got the newer kernel. I did not upgrade my host raspbian, because this may be related. Maybe I need to upgrade it as well? Edit: Yes!

Linux raspberrypi 4.4.9-v7+ #884 SMP Fri May 6 17:28:59 BST 2016 armv7l GNU/Linux

# The broken os shows the same kernel (possibly because of chroot).
# The modules directory has the newer kernel. so an upgrade on the host may help?
root@raspberrypi:/# ls /lib/modules
4.4.11+  4.4.11-v7+

[XECDesign]

Regarding the error you get. You need to give mkinitramfs the -k parameter with the new kernel version.

Here are the main incompatibilities with the default debian initramfs hooks:
Debian builds the initrd for 1 kernel. Our kernel package contains two, so it won't work properly on all pi models. We can't guess which one you want.
Debian adds the kernel version to the end of the filename, so /boot/config.txt needs to be modified each time.

I would recommend you take a look at /etc/kernel/*/initramfs-tools and add your own hooks that do exactly what you want. Perhaps modify the existing ones to run regardless of what the INITRD variable is and then add another hook to sed your config.txt to point to the right filename.

[NicoHood]

Why 2 kernels? One for arm6 and one for arm7? Can the pi2 boot both kernels?

As i said, I am not an expert in this, but doesnt initramfs load another kernel anyways? So one could build it with the arm6 kernel, but dynamically load the pi1/2/3 kernel. I mean somehow the pi must select one of those 2 kernels anyways. How is this done without initramfs?

Can't you also detect if an initramfs is used, and only then regenerate one with the kernel type that is currently running? So for normal use it would not change at all, only if you activated initramfs.

[TheSin-]

just like kernel.img and kernel7.img gets auto selected, couldn't we detect the presence of initrd.img and initrd7.img and auto select? Add a config.txt option for initrd=no or something if you want to force disable it. Maybe it already does this I'm not sure, just pinging @popcornmix and @pelwell as I'm sure they would know.

Then you wouldn't need to do anything really cept fix the deb postinst script to auto gen both on update, which really isn't that much work at all.

[XECDesign]

@NicoHood Yes, one v6 and one v7 kernel. No, the pi 2 can only boot one of them.

Initramfs does not load a kernel. It continues to use the kernel it was started with and then switches the root filesystem and runs the new init. The pi's bootloader is aware of what it's running on, so it knows which kernel to use.

There are all sorts of ways to work around it, but it will depend on what you want to achieve, so I think your own hooks are the way to go for now.

@TheSin- you could use the [] conditional tags in config.txt to specify the different initramfs files.But still, the default debian initramfs-tools hooks generate initramfs files with the kernel version tacked on, so either that package would need to be modified, or an extra hook to update config.txt would need to be added.

[TheSin-]

Oh I agree we need our own hook, I'm just saying that if the firmware is aware then we don't need to change config.txt. Easier and cleaner for packaging is all. I have my own kernel packages I maintain, and I have already written my own initramfs hooks which are working perfectly.

[NicoHood]

May you share your initramfs hooks? I have honestly no idea how to write them for now.

Wouldn't it be possible to change the bootloader which then can run the (correct) initramfs instead of the kernel?

[XECDesign]

initramfs doesn't run instead of a kernel. It's loaded into memory along with the kernel, then the kernel uses it as the root filesystem.

I am testing a hook right now. Will let you know how it goes.

[XECDesign]

At the end of config.txt, I've added:

initramfs initrd.img-4.4.9+ followkernel
[pi2]
initramfs initrd.img-4.4.9-v7+ followkernel
[pi3]
initramfs initrd.img-4.4.9-v7+ followkernel
[all]

In /etc/kernel/postinst.d/initramfs-tools, I've commended out this bit:

# exit if kernel does not need an initramfs
#if [ "$INITRD" = 'No' ]; then
#       exit 0
#fi

I recommend doing the same in /etc/kernel/postrm.d/initramfs-tools

Then back in /etc/kernel/postinst.d/initramfs-tools, I've added this:

if echo ${version} | grep -qE "\-v7+"; then
        sed -r "s|(initramfs initrd.img-).*v7\+( followkernel$)|\1${version}\2|" -i /boot/config.txt
else
        sed -r "s|(initramfs initrd.img-).*\.[[:digit:]]+\+( followkernel$)|\1${version}\2|" -i /boot/config.txt
fi

Can't say that this is a particularly robust solution, but it seems to work.
Edit: I'd probably add a copy of the script with the modifications, rather than modify the original.

[Ferroin]

I may be missing something here, but why not just add something to copy the generated initramfs image to the expected location? Then you wouldn't have to manipulate config.txt, could still use Debian's mkinitrd, and things should Just Work.

[XECDesign]

Yeah, but you'd be using up an extra ~10MB on having the extra copies.

[NicoHood]

Can't you symlink them?

[Ferroin]

So delete the version tagged copy after copying it to the un-tagged one. We have no current obligation to have it at all for compatibility reasons, so why should we keep it around?

[Ferroin]

@NicoHood Symlinks don't exist on FAT filesystems. Most of the UNIX filesystem stuff taken for granted by Linux applications doesn't work on FAT filesystems.

[XECDesign]

AFAIK, initramfs-tools keeps track of the initramfs files it creates, so deleting files using something other than update-initramfs -d may not be the right way to do it. Anyway, that was just one approach to show that it's possible to make the hooks work. How you go about it on your own system is up to you.

[NicoHood]

I am getting back here as I now has a little bit more knowledge after installing arch with full disk encryption, even /boot.

But I still do not understand the problem. Can't we check if the user created an initramfs already and if so, just regenerate it? We will do this for both arm6 and arm7 kernel and create different names. In config.txt the user has to add an if/else for 2 different initramfs (arm6/7) and everything should be fine. Or if it works, maybe the arm6 initramfs would also work with arm7?

If we make the config.txt setting required by the user, but always name the initramfs the same, there should not be much problems?

[XECDesign]

Yes, you could check and regenerate initramfs, but that would need to be a custom hook.

[NicoHood]

It would be cool if this works by default. Arch Linux for the raspi already supports initramfs regeneration on kernel updates. Nobody on the internet know this, but they added this feature some time ago and i could verify that it works. It would be nice if raspbian can get this hook by default too.

[XECDesign]

I agree and may revisit it for the next kernel package build. There are a few remaining concerns, but since it is a relatively commonly requested feature, it may as well be added.

[Robpol86]

I came across this issue while writing my own LUKS guide for the Raspberry Pi. The two issues I came across was: my SD card no longer working on different Raspberry Pi hardware (e.g. mkinitramfs on a Pi Zero and then putting it in a Pi 3) and updating the kernel via apt-get wouldn't update the initramfs.gz so rebooting the Pi would lead to failure.

I want to post my two hook scripts I wrote to overcome these two problems in case someone else runs into this issue.

/etc/kernel/postinst.d/initramfs-rebuild:

#!/bin/sh -e

# Rebuild initramfs.gz after kernel upgrade to include new kernel's modules.

# Remove splash from cmdline.
if grep -q '\bsplash\b' /boot/cmdline.txt; then
  sed -i 's/ \?splash \?/ /' /boot/cmdline.txt
fi

# Exit if not building kernel for this Raspberry Pi's hardware version.
version="$1"
current_version="$(uname -r)"
case "${current_version}" in
  *-v7+)
    case "${version}" in
      *-v7+) ;;
      *) exit 0
    esac
  ;;
  *+)
    case "${version}" in
      *-v7+) exit 0 ;;
    esac
  ;;
esac

# Exit if rebuild cannot be performed or not needed.
[ -x /usr/sbin/mkinitramfs ] || exit 0
[ -f /boot/initramfs.gz ] || exit 0
lsinitramfs /boot/initramfs.gz |grep -q "/$version$" && exit 0  # Already in initramfs.

# Rebuild.
mkinitramfs -o /boot/initramfs.gz "$version"

/etc/initramfs-tools/hooks/other_kernel:

#!/bin/sh -e
# Copy other kernel's modules into initramfs image.
PREREQ=""
prereqs () {
  echo "${PREREQ}"
}
case "${1}" in
  prereqs)
    prereqs
    exit 0
  ;;
esac
. /usr/share/initramfs-tools/hook-functions
case "${version}" in
  *-v7+) other_version="$(echo ${version} |sed 's/-v7+$/+/')" ;;
  *+) other_version="$(echo ${version} |sed 's/+$/-v7+/')" ;;
  *)
    echo "Warning: kernel version doesn't end with +, ignoring."
    exit 0
esac
cp -r /lib/modules/${other_version} ${DESTDIR}/lib/modules/

Regarding remove splash from cmdline: The Raspbian PIXEL image has the boot splash screen enabled which interfered with the LUKS prompt for me. I had to remove "splash" from cmdline.txt to be able to see the prompt and type the password. Updating the kernel seems to restore "splash" in the cmdline hence I had to add logic to initramfs-rebuild to remove it again.

When updating the kernel through apt-get both kernels are built, and so initramfs-rebuild is called twice (once for each kernel as parameter $1). To avoid rebuilding initramfs.gz twice I had to put in logic so it only builds once and skips the other time. other_kernel includes the other kernel's modules so you always end up with both kernel's modules in the image.

This will lead to initramfs.gz images being twice as big, but they still fit in the /boot partition with a few megabytes to spare.

I tested these two scripts on 2017-01-11-raspbian-jessie.img on a Pi Zero v1.3 and a Pi 3 Model B v1.2.

[XECDesign]

RPi-Distro#1 (comment)

Initramfs generation should be easily enable-able now. It's still using initramfs-tools' scripts, so the filename will change and config.txt needs to be updated manually or through your own hook.

In the future, the firmware may load initrd(7).img, which would be generated by a forked initramfs-tools and renamed package.

[JamesH65]

@NicoHood @XECDesign Looks like this issue can be closed?

[XECDesign]

I'd close it as a wontfix. It's possible to generate initrd automatically, but the user would need to set it up themselves. I can look at making it easier and documenting it, but not in the near future.

[JamesH65]

Closing as a Won't Fix. This means that although the issue is acknowledged, it is felt that no further action is likely. This may be down to a bad benefit vs cost analysis, or it may actually not be possible to fix.

[nimbl]

As reference to other people encountering this issue, a possible workaround that does not affect the existing initrd codebase, uses the fact that the update-initramfs command has a post update hook option builtin to it, that will run any scripts in the /etc/initramfs/post-update.d/ directory (you may have to create them.) So by adding a script like the following, you will ensure that the /config/boot.txt file gets updated whenever update-initramfs is triggered (from kernel upgrades or module changes.)

#!/bin/sh
## File: /etc/initramfs/post-update.d/update-rpi-boot-config-file (called from post update hook by update-initramfs)
version="$1"
initramfs="$2"

echo "update-rpi-boot-config-file: Ensuring that /boot/config.txt references initrd version ${version}"

update_boot_config(){
        if echo ${version} | grep -qE "\-v7+"; then
                sed -r "s|(initramfs initrd.img-).*v7\+( followkernel$)|\1${version}\2|" -i /boot/config.txt
        else
                sed -r "s|(initramfs initrd.img-).*\.[[:digit:]]+\+( followkernel$)|\1${version}\2|" -i /boot/config.txt
        fi
}

if grep -qE "^initramfs initrd.img-${version}" /boot/config.txt; then
        echo "update-rpi-boot-config-file: /boot/config.txt already references this version, no changes made."
else
        echo "update-rpi-boot-config-file: Updating /boot/config.txt to reference new version."
        update_boot_config
fi

[beren12]

@JamesH65 could you please at least add this post-update hook? it'd be really nice for people who do setup the initrd (which is pretty easy). I'm using it to boot off of zfs so I can tell when my SD card starts to die and I get almost free transparent compression/snapshots with zfs

[JamesH65]

Nothing to do with me. @XECDesign ?

[XECDesign]

Something equivalent to this will be a part of the kernel package rework, which has recently become a higher priority. So yes, it's coming.