Pool Cannot Be Imported After Restart

[minorsatellite]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	22.04.3 LTS
Kernel Version	5.15.0-91
Architecture	arm64
OpenZFS Version	zfs utils 2.2.99-275, kmod 2.1.5

Describe the problem you're observing

I have an instance of Ubuntu 20.04.3 LTS Server running on Apple Silicon (as a VM) where I have performed an In-Tree Build and Install of ZFS as per the instructions provided here. None of the native distro ZFS packages were previously installed.

After installation, I successfully created my test pool as a two-way mirror. Its all good until I restart and discover that the ppol has not imported. When I manually attempt to import the pool, I receive the error that the pool is not importable due to an unsupported version or feature, despite the fact that the pool was created on the very same running instance of ZFS.

This seems like a bug to me.

Describe how to reproduce the problem

Every time I try to import the pool unsuccessfully.

Include any warning/errors/backtraces from the system logs

sudo pool import tank
This pool uses the following feature(s) not supported by this system:
        com.klarasystems:vdev_zaps_v2
cannot import 'tank': unsupported version or feature

[rincebrain]

You're running Ubuntu's builtin version of ZFS after reboot. I would assume that however you set it up, the kernel module version didn't get updated when you rebooted into a new kernel.

Ubuntu ships their ZFS kernel modules in the base kernel package, so you get those without installing anything, only the userland tools are in debs.

Not a bug.

[dmdx86]

Your zfs-utils version is mismatching the kmod version after a reboot. This is likely because the older zfs module that comes with Ubuntu is in your initrd but the new zfs module is not, so the old kmod is being loaded at boot time.

Do a update-initramfs -c -k all and try rebooting again.

[minorsatellite]

As noted before I never installed the Ubuntu ZFS packages, so how can that be?

Unfortunately updating the initramfs did not improve matters.

[minorsatellite]

apt list --installed | grep zfs returns zero results

[rincebrain]

Ubuntu ships their ZFS kernel modules in the base kernel package, so you get those without installing anything, only the userland tools are in debs.

[minorsatellite]

OK, so how do I back out of that?

And moreover, would the kmod and utility mismatch produce such an error?

[dmdx86]

Ubuntu includes a zfs module in the standard linux-modules package. Here’s an Ubuntu 22.04 VM that I use that has no ZFS package installed (don’t use ZFS at all on this VM) but the zfs module is nonetheless there.

root@candy:~# find /lib/modules -name zfs\*
/lib/modules/6.2.0-1019-gcp/kernel/zfs
/lib/modules/6.2.0-1019-gcp/kernel/zfs/zfs.ko

root@candy:~# dpkg -S /lib/modules/6.2.0-1019-gcp/kernel/zfs/zfs.ko
linux-modules-6.2.0-1019-gcp: /lib/modules/6.2.0-1019-gcp/kernel/zfs/zfs.ko

OK I re-read your original post and followed your link. I am not familiar with those instructions. They have you do a “make install” and I am not entirely sure what that does to the system.

My recommendation would be to follow the “Custom Packages” instructions and build the .deb packages. I always use dkms.

https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html#dkms-1

In any case, can you do a “find /lib/modules -name zfs*” and see if a zfs module actually got installed into the kernel modules tree?

[rincebrain]

I'm moving this to a discussion because it's not a bug.

[minorsatellite]

@rincebrain thank you; @dmdx86 yes your hunch is correct.

I will look at the Custom Package approach as well; @rincebrain had suggested that earlier and I made one attempt at it but ran into issues which more likely the result of user error.

[rincebrain]

You should probably not follow the instructions for "how to develop ZFS" for "I want to run ZFS".

[minorsatellite]

It is under In-Tree Build and Install, which is what I was referencing earlier, but not under Custom Packages, and only its only mentioned there (and once), so for literalists like myself, the elliptical nature of the documentation can be quite confusing.

[minorsatellite]

Here is how I finally got it to work, the last line being key since it is Ubuntu

sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev linux-headers-generic python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx

wget https://github.com/openzfs/zfs/releases/download/zfs-2.2.2/zfs-2.2.2.tar.gz
tar -xzf zfs-2.2.2.tar.gz

cd zfs
./configure --enable-systemd
make -j1 deb-utils deb-kmod
sudo apt-get install --fix-missing ./*.deb

sudo sed -i.bak 's/updates/extra updates/' /etc/depmod.d/ubuntu.conf

Thanks to @gmelikov for that tip from an earlier post concerning the 0.8.3 release. Thanks also to @rincebrain and others for holding my hand.

Now to do some proper testing.

[rincebrain]

That's actually in the documentation.