zfs-initramfs package is broken in PPA

[wrouesnel]

In the current stable and daily PPA's, then zfs-initramfs package has completely broken dependencies.

This could be easily fixed by removing the dependencies, so the package would just install the ZFS hooks for booting with ZFS on Root.

The use case is that the mounting of ZFS in the initramfs is not in anyway dependent on whether GRUB can bootstrap itself from the ZFS volume, and in the case where you might use an ext2 /boot partition, you presently have to manually fiddle around with getting the initramfs installed.

If ZFS and grub need to be linked, it should probably be through a separate meta-package that does so.

[dajhorn]

This could be easily fixed by removing the dependencies, so the package would just install the ZFS hooks for booting with ZFS on Root.

Use the equivs utility to install a dummy package that satisfies the dependency.

The use case is that the mounting of ZFS in the initramfs is not in anyway dependent on whether GRUB can bootstrap itself from the ZFS volume

The technical support case is, however, about many people wanting the pure ZFS system and missing the non-obvious grub dependency.

This currently overrides any argument for a slightly tighter dependency list on zfs-initramfs, so I will not make the requested change. The downstream ticket is dajhorn/pkg-zfs#38.

If ZFS and grub need to be linked, it should probably be through a separate meta-package that does so.

Doing this at the packaging level is impossible because system configurations can change between invocations of the package manager. Regular things like system imaging and backup recovery would break any non-clairvoyant package.

[pdf]

The technical support case is, however, about many people wanting the pure ZFS system and missing the non-obvious grub dependency.

I think you underestimate the support overhead generated in the opposite direction by this stance. Anyone who's not running root on ZFS (which, at this stage I would have thought would be a significant percentage?) will find all sorts of functionality broken without this package. Like being able to mount zpools using zdevs - or any of the other functionality provided by the udev hooks - during boot. See: many bugs in the tracker pertaining to failure to mount on boot.

I was definitely banging my head against a wall, spent a good couple of hours debugging mountall and other stuff, thinking things were just broken, and eventually coming to the conclusion that adding the udev hooks to initramfs might solve the problems. Then I remembered that this package exists in the other repo (of course, enabling that repo opens me to accidentally pulling in the ZFS-patched grub, which I don't want).

Doing this at the packaging level is impossible because system configurations can change between invocations of the package manager. Regular things like system imaging and backup recovery would break any non-clairvoyant package.

Can you expand on this a little further?

[dajhorn]

The only reason to install the zfs-initramfs package is for a native ZFS root. There is no other reason to load the ZFS driver early on a Debian or Ubuntu computer. Any such configuration is unsupported and known to be unreliable.

There are tens of thousands of working ZoL installations that do not use the zfs-initramfs package. Operator error has been the cause of all mountall bugs where I have engaged the user for troubleshooting.

It is entirely possible that you have a legitimate bug, but you (or somebody else) needs to submit a patch for review.

[pdf]

Loading the driver early is not the goal - it's populating the additional device nodes like zdevs via udev, which doesn't happen by the time mountall is called unless the udev rules are in the initrd. The only reliable method for mountall working on boot without the udev helpers is for pools using by-path, which is unmanageable with large numbers of disks, and dependant on the backing hardware presenting sane paths.

[dajhorn]

The only reliable method for mountall working on boot without the udev helpers is for pools using by-path,

Right. Using a /dev/disk/by-* path is best practice for ZoL.

Furthermore, udev is indeed a dependency.

which is unmanageable with large numbers of disks,

Unsupported. Using bare device nodes is wrong for ZoL.

Frankly, this is not a problem for operators that know how to run big iron. Try generating a zdev.conf file according to your site requirements, or writing custom udev rules that override the bundled udev rules.

and dependant on the backing hardware presenting sane paths.

Post manufacturer and model information to the ZoL discussion list and describe your problem there. If somebody can suggest a kludge that does not compromise systems with good equipment, then I will integrate it.

[pdf]

Unsupported. Using bare device nodes is wrong for ZoL.

Frankly, this is not a problem for operators that know how to run big iron. Try generating a zdev.conf file according to your site requirements, or writing custom udev rules that override the bundled udev rules.

That's what I've been saying - zdev.conf is useless unless the zdev device nodes are populated before mountall, which doesn't seem to happen unless the udev rules are in the initrd.

[dajhorn]

Note that:

• Ubuntu usually requires that system mounts be on cold plugged devices.
• ZoL itself does not automatically import pools on hot plugged devices.

Rather, all pool members must be ready when upstart calls mountall, which happens early in the init stack. These two things preclude getting the desired result from removable media, RAID implementations with sloppy JBOD modes, and some SAN configurations.

Such things on Ubuntu require a finesse for proper integration that is inappropriate for general release in a package. If you have dud equipment, a goofy SAN fabric, or use a proprietary HBA driver, then you probably need to do a custom configuration.

Ubuntu has some implementation oddities and design mistakes in this area, but correcting or subverting The Ubuntu Way is not something I'm currently interested in doing. Invasive changes to udev or upstart will not happen.

That said, if I still have a misunderstanding, then you should explain it to me with a patch that produces the desired behavior. Describe your system, including manufacturer and model information, to get a better assessment.

This issue further motivates changing the default search path per #274.

[pdf]

Where is the mountall source stored, and where should I file a bug against it?

[dajhorn]

Run apt-get source mountall and diff against that. The bzr branch for it at Launchpad is stale.

File a bug at https://github.com/dajhorn/pkg-zfs/issues or continue the discussion here.

[pdf]

Okay, I've taken your package and moved the call to parse_zfs_list after udev_catchup, before the main loop so that /dev/disk/zpool etc is populated before the mount list is updated, otherwise zdev pools can't be imported. I must admit I haven't looked hard at this to make sure that's not evil, but it looks safe enough.

PS - You should look into git-buildpackage for working with upstream packages, the source package for your mountall was a bit of a mess (pre-patched files and .pc included in the package, missing quilt format). I've pushed my working dir to pdf/mountall-zfs temporarily, but I'd suggest starting with git-buildpackage from a clean upstream version for easier maintenance.

EDIT: I've got a step-by-step from scratch of what happens with pools of zdevs without this change, if you want me to post it, including mountall debug logs. But, it's very long due to the logs, and basically just says, "zfs filesystems on pools of zdevs aren't mounted at boot using mountall". I'll post it if you're not clear on what I'm trying to solve here though?

[dajhorn]

Okay, I've taken your package and moved the call to parse_zfs_list after udev_catchup,

Thanks. I will try it soon.

You should look into git-buildpackage for working with upstream packages

The primary pkg-spl and pkg-zfs repositories already use git-buildpackage.

I tried doing something similar for mountall a year ago, but its upstream repo is bzr, and it wasn't worthwhile because the lp+bzr facility is awkward.

[pdf]

I tried doing something similar for mountall a year ago, but its upstream repo is bzr, and it wasn't worthwhile because the lp+bzr facility is awkward.

If you just grab the source package (.dsc, diff, orig.tar.gz, etc - what you get from apt-get source, or off LP), then use git-import-dsc you'll get a working git-buildpackage repo, making imports of new upstream versions easy (just use git-import-dsc again). Then you can manage the patch-queue via gbp-pq.

Anyway, getting off-topic. Let me know how you go with mountall. It's consistent on the few systems I've tested with so far.

EDIT: Scratch that, results are not consistent.

[pdf]

I'm still really thinking here that getting the udev stuff into the initrd is going to be the only sane way of handling this - I see this as analogous to LVM, and that requires udev helpers in the initrd. Yes, it will mean running update-initramfs if zdev.conf changes, but the same goes for modifying lvm.conf for anything required at boot, and I'd assume zdev.conf will be modified with about the same frequency. So, I'm considering splitting the initramfs stuff out, moving the udev parts into zfsutils, with the rest staying in zfs-initramfs. At that point, mountall can stay as-is in 2.36-zfs1. Thoughts?

EDIT: It's after 2am here, going to come back to this with fresh eyes in the morning.

[pdf]

I've been testing dajhorn/pkg-zfs#39 all day, zdev-based pools appear to be working here both with and without root on zfs (and hence, with/without zfs-initramfs).