Hate to bother you with this...not sure who else would know!!

[silverspr]

I've been on ZFS-root for 3-4 months, struggling with the error below for at least a month now. I've tried uninstalling and reinstalling dracut, rolling back to an older snapshot, fixing install using apt/dpkg, update-initramfs is NOT installed.
Reviewed zfs-root install script: dracut section and all the configs look to be in place i.e. zfs module ... but can't get rid of the error. Hoping you, an expert have an idea of where to go with this!! thanks....
uname -r
6.17.0-20-generic (not sure why below, says generating /boot/initrd.img-6.14.0-37-generic)

When running apt get upgrade
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up dracut (060+5-1ubuntu3.3) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for dracut (060+5-1ubuntu3.3) ...
dracut: Generating /boot/initrd.img-6.14.0-37-generic
dracut[E]: Module 'zfs' cannot be found.
dpkg: error processing package dracut (--configure):
installed dracut package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
dracut
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Halfwalker]

How weird - haven't seen that happen. Seems like something has gotten out of sync somewhere ... especially around the kernel/initrd versions

Couple of typical things to check :

• Verify you don't have initramfs-tools installed - that should have been marked as hold dpkg -l \*initramfs\*
• Check which packages are on hold with apt-mark showhold
• update-initramfs is part of the initramfs-tools pkg - something else may have pulled that in

  ❯ apt-file search update-initramfs
  backport-iwlwifi-dkms: /usr/src/backport-iwlwifi-11510/scripts/update-initramfs.sh
  casper: /usr/share/casper/casper-update-initramfs
  initramfs-tools: /etc/initramfs-tools/update-initramfs.conf
  initramfs-tools: /usr/sbin/update-initramfs
    :
  

• If it IS installed, can you remove it with dpkg --purge --force-all initramfs-tools ? Might show what's depending on it
• Look in /boot - what kernels and initrd files are there ? Do they match up with each other ?
• Verify the zfs module exists for each kernel - for example

  ❯ find /lib/modules -iname zfs\*
  /lib/modules/6.14.0-29-generic/kernel/zfs/
  /lib/modules/6.14.0-29-generic/kernel/zfs/zfs.ko.zst
  /lib/modules/6.8.0-100-generic/kernel/zfs/
  /lib/modules/6.8.0-100-generic/kernel/zfs/zfs.ko.zst
  

• Check that the headers and modules for the kernel(s) are installed - you should have image, headers, modules and modules-extra

  ❯ dpkg --no-pager -l \*6.14.0-29\*
  Desired=Unknown/Install/Remove/Purge/Hold
  | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
  |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
  ||/ Name                                   Version              Architecture Description
  +++-======================================-====================-============-===================================================
  ii  linux-headers-6.14.0-29-generic        6.14.0-29.29~24.04.1 amd64        Linux kernel headers for version 6.14.0
  ii  linux-hwe-6.14-headers-6.14.0-29       6.14.0-29.29~24.04.1 all          Header files related to Linux kernel version 6.14.0
  ii  linux-image-6.14.0-29-generic          6.14.0-29.29~24.04.1 amd64        Signed kernel image generic
  un  linux-image-unsigned-6.14.0-29-generic <none>               <none>       (no description available)
  ii  linux-modules-6.14.0-29-generic        6.14.0-29.29~24.04.1 amd64        Linux kernel extra modules for version 6.14.0
  ii  linux-modules-extra-6.14.0-29-generic  6.14.0-29.29~24.04.1 amd64        Linux kernel extra modules for version 6.14.0
  un  linux-tools-6.14.0-29-generic          <none>               <none>       (no description available)
  

• Try regenerating the initramfs for a specific kernel with dracut -v -f 6.14.0-37-generic
• If it all looks OK, you can try a fix-broken with apt --fix-broken install

[silverspr]

Yah...this is the second time this has happened. The first time I rolled back to "almost" a fresh install and reinstalled updates etc. Was good for a bit then started again, not sure what update is triggering this!! OK onto troubleshooting:

Neither initramfs-tools or initramfs-update is installed. Items on hold i.e. apt-mark showhold is identical to the hold items in ZFS-root script.

/boot
initrd.img-6.14.0-37-generic vmlinuz-6.14.0-37-generic
initrd.img-6.17.0-19-generic vmlinuz-6.17.0-19-generic
initrd.img-6.17.0-20-generic vmlinuz-6.17.0-20-generic

ZFS modules: kernel 6.14.0-37 for brevity
/lib/modules/6.14.0-37-generic/kernel/zfs
/lib/modules/6.14.0-37-generic/kernel/zfs/zfs.ko.zst

Headers/modules (had to add bullets otherwise strikeout was turned on and couldn't turn off!!!)

• ii linux-headers-6.14.0-37-generic 6.14.0-37.37~24.04.1 amd64 Linux kernel headers for version 6.14.0
• ii linux-hwe-6.14-headers-6.14.0-37 6.14.0-37.37~24.04.1 all Header files related to Linux kernel version 6.14.0
• ii linux-image-6.14.0-37-generic 6.14.0-37.37~24.04.1 amd64 Signed kernel image generic
• un linux-image-unsigned-6.14.0-37-generic (no description available)
• ii linux-modules-6.14.0-37-generic 6.14.0-37.37~24.04.1 amd64 Linux kernel extra modules for version 6.14.0
• ii linux-modules-extra-6.14.0-37-generic 6.14.0-37.37~24.04.1 amd64 Linux kernel extra modules for version 6.14.0

Regenerate initramfs sudo dracut -v -f 6.14.0-37-generic
dracut[I]: Executing: /usr/bin/dracut -v -f 6.14.0-37-generic
dracut[I]: Module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut[I]: Module 'warpclock' will not be installed, because command 'hwclock' could not be found!
dracut[I]: Module 'systemd-coredump' will not be installed, because command 'coredumpctl' could not be found!
dracut[I]: Module 'systemd-coredump' will not be installed, because command '/usr/lib/systemd/systemd-coredump' could not be found!
dracut[I]: Module 'systemd-pcrphase' will not be installed, because command '/usr/lib/systemd/systemd-pcrphase' could not be found!
dracut[I]: Module 'systemd-portabled' will not be installed, because command 'portablectl' could not be found!
dracut[I]: Module 'systemd-portabled' will not be installed, because command '/usr/lib/systemd/systemd-portabled' could not be found!
dracut[I]: Module 'modsign' will not be installed, because command 'keyctl' could not be found!
dracut[I]: Module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut[I]: Module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut[I]: Module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut[I]: 62bluetooth: Could not find any command of '/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'!
dracut[D]: Module 'btrfs' will not be installed, because it's in the list to be omitted!
dracut[I]: Module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut[I]: Module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut[I]: Module 'pcsc' will not be installed, because command 'pcscd' could not be found!
dracut[I]: Module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut[I]: Module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut[I]: Module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut[I]: Module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut[I]: memstrack is not available
dracut[I]: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut[E]: Module 'zfs' cannot be found.

sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up dracut (060+5-1ubuntu3.3) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for dracut (060+5-1ubuntu3.3) ...
dracut: Generating /boot/initrd.img-6.14.0-37-generic
dracut[E]: Module 'zfs' cannot be found.
dpkg: error processing package dracut (--configure):
installed dracut package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
dracut
E: Sub-process /usr/bin/dpkg returned an error code (1)

sooo....dunno everything looks to be right?? and thanks again for looking at this, I'm sure this isn't part of what you signed on for!!

[Halfwalker]

Where did kernel 6.17.0-20 come from ? Do you see it in the installed packages ? I don't think that has anything to do with it, but weird nonetheless ...

Is zfs-dracut still installed ? I'll be honest, I'm not all that up on how dracut does its thing ... The kernel module is there, so it should be able to find it. Something is causing dracut to not know about zfs ...

How about the dracut configs ? This is an example of a bare 24.04 NOENC setup

packer@noble-NOENC:/etc/dracut.conf.d$ l
total 12
drwxr-xr-x  2 root root   4 Mar  4 18:08 .
drwxr-xr-x 85 root root 168 Apr 11 19:30 ..
-rw-r--r--  1 root root  70 Mar  4 18:07 100-zol.conf
-rw-r--r--  1 root root  87 Mar  4 18:08 no-crypt-ssh.conf

packer@noble-NOENC:/etc/dracut.conf.d$ cat 100-zol.conf
nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs "

packer@noble-NOENC:/etc/dracut.conf.d$ cat no-crypt-ssh.conf
# We do NOT need crypt-ssh in the main system initrd
omit_dracutmodules+=" crypt-ssh "

I think that update-initramfs there is part of the trigger, and this is where dracut is defined as the tool to use

packer@noble-NOENC:~$ cat /var/lib/dpkg/triggers/update-initramfs
dracut

For debugging I can think of a couple of approaches ...

You can snapshot and clone the system, then roll back the clone to where it works. Then (painful I know) try to replicate the various installs you did, watching carefully for anything weird. Do a dracut -v -f .... each time to ensure it's still working. At one point you'll hit whatever it was that messed things up.

Or snapshot/clone, then roll back one snapshot at a time, testing with dracut -v -f ... each time to see which snapshot is the one where the issue was introduced. Then start poking around in shell history etc.

Ugh.

[silverspr]

darn...
I didn't have no-crypt-ssh.conf...I've added it with the output you provided.
When I try install zfs-dracut ..
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
zfs-dracut : Depends: systemd-cryptsetup but it is not installable
E: Unable to correct problems, you have held broken packages.

Ok starting the rollback....

[silverspr]

On a lark I tried
sudo dpkg --force-all -i Downloads/zfs-dracut_2.4.1-2arter97~ubuntu24.04.1_all.deb (looked like it worked)
dpkg: zfs-dracut: dependency problems, but configuring anyway as you requested:
zfs-dracut depends on dracut; however:
Package dracut is not configured yet.
zfs-dracut depends on systemd-cryptsetup; however:
Package systemd-cryptsetup is not installed.

sudo dracut -f -v
I'm able to compile initramfs images

However when getting updates, zfs-dracut shows as upgradable, if I try to upgrade
sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
zfs-dracut : Depends: systemd-cryptsetup but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
apt --fix-broken install doesn't resolve anything....I' guessing due to forced install. Ignoring upgrade zfs-dracut message .. I'm sorta fixed!!

[Halfwalker]

Hoy, sorry about the delay, I didn't see this update.

Where did you get zfs-dracut_2.4.1 from ? Ubuntu 24.04 uses 2.2.2, and doesn't have a dependency on systemd-cryptsetup. I think 2.4.1 is a Debian Trixie version ? Ubuntu 26.04 uses 2.3.5 I believe.