Removes zfs from only the "new_kernel" installer variants #59863

samueldr · 2019-04-19T02:11:00Z

This was tested using this patch to neuter the zfs kernel module.

nix-build nixos/release-combined.nix -A nixos.iso_minimal_new_kernel.x86_64-linux

and on aarch64

nix-build nixos/release-combined.nix -A nixos.iso_minimal_new_kernel.aarch64-linux

I wasn't able to think of an elegant way to reduce the duplication of the list of supported filesystems. Referencing to a configured value within the module system is an infinite recursion. The only way I thought of is a bit ugly, is through having the list in a .nix file which would be imported (and filtered as needed). This would have made one bizarro file which would live within the other module files, but couldn't be added to imports.

Any inputs on ways to remove the mostly duplicated list?

This is because of the phases where zfs support is not yet available for the new kernel, forcing "new_kernel" variants not to be built. This might be affecting aarch64-linux more often than x86_64-linux. The x86_64-linux image is seldom used, most likely uses are for extremely new hardware exhibiting issues on the current LTS. The aarch64-linux image is more likely to be used considering how board support is always improving. It was the previous default on aarch64-linux.

infinisil · 2019-04-19T14:08:03Z

nixos/modules/profiles/base.nix

@@ -49,7 +49,7 @@
  ];

  # Include support for various filesystems.
-  boot.supportedFilesystems = [ "btrfs" "reiserfs" "vfat" "f2fs" "xfs" "zfs" "ntfs" "cifs" ];
+  boot.supportedFilesystems = lib.mkDefault [ "btrfs" "reiserfs" "vfat" "f2fs" "xfs" "zfs" "ntfs" "cifs" ];


I think you could condition including zfs based on the kernel version from config.boot.kernelPackages

^ seems interesting for sure. I'm a bit concerned about using mkDefault. Would users who have already added an additional FS find their system unbootable?

Hmm. Looks like it can't be done through comparing config.boot.kernelPackages.kernel.version; infinite recursion, which I think is coming from the zfs task, which also looks at supportedFilesystems, and config.boot.kernelPackages. Though I'm unsure about the diagnosis.

What's the trace with --show-trace?

Here it is:

Trace

error: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release-combined.nix:20:34, called from undefined position: while evaluating 'removeMaintainers' at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release-combined.nix:17:23, called from /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release-combined.nix:20:37: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release.nix:146:32, called from /Users/samuel/tmp/nixpkgs/nixpkgs/lib/attrsets.nix:292:43: while evaluating 'makeIso' at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release.nix:42:5, called from /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release.nix:146:40: while evaluating 'hydraJob' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/customisation.nix:157:14, called from /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/release.nix:46:5: while evaluating the attribute 'drvPath' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/customisation.nix:174:13: while evaluating the attribute 'drvPath' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/customisation.nix:141:13: while evaluating the attribute 'sources' of the derivation 'nixos-minimal-19.09pre56789.gfedcba-x86_64-linux.iso' at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/lib/make-iso9660-image.nix:49:3: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/lib/make-iso9660-image.nix:56:18, called from undefined position: while evaluating the attribute 'source' at /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/modules/installer/cd-dvd/iso-image.nix:585:11: while evaluating the attribute 'boot.kernelPackages.kernel' at undefined position: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:75:45, called from undefined position: while evaluating the attribute 'value' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:338:9: while evaluating the option `boot.kernelPackages': while evaluating the attribute 'isDefined' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:375:5: while evaluating the attribute 'values' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:364:9: while evaluating the attribute 'values' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:458:7: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:350:28, called from /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:350:17: while evaluating 'dischargeProperties' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:417:25, called from /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:351:62: while evaluating the attribute 'condition' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:502:14: while evaluating the attribute 'condition' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:502:14: while evaluating the attribute 'condition' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:502:14: while evaluating the attribute 'boot.supportedFilesystems' at undefined position: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:75:45, called from undefined position: while evaluating the attribute 'value' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:338:9: while evaluating the option `boot.supportedFilesystems': while evaluating the attribute 'isDefined' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:375:5: while evaluating the attribute 'values' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:364:9: while evaluating the attribute 'values' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:458:7: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:350:28, called from /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:350:17: while evaluating 'dischargeProperties' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:417:25, called from /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:351:62: while evaluating the attribute 'value' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:234:44: while evaluating 'optional' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/lists.nix:241:20, called from /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/modules/profiles/base.nix:64:6: while evaluating 'versionOlder' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/strings.nix:461:22, called from /Users/samuel/tmp/nixpkgs/nixpkgs/nixos/modules/profiles/base.nix:66:5: while evaluating the attribute 'boot.kernelPackages' at undefined position: while evaluating anonymous function at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:75:45, called from undefined position: while evaluating the attribute 'value' at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:338:9: infinite recursion encountered, at /Users/samuel/tmp/nixpkgs/nixpkgs/lib/modules.nix:338:9

(feel free to ping me on IRC if you have some time for me to pick your brain :))

fpletz

If we remove zfs on these images we should either remove it from all install mediums or not at all since this is an invisible difference only waiting for users to trip on it and start getting confused.

Another option would be to always use the latest Linux kernel compatible to zfs.

samueldr · 2019-09-28T17:24:54Z

If we remove zfs on these images we should either remove it from all install mediums or not at all since this is an invisible difference only waiting for users to trip on it and start getting confused.

I agree, this is a reason why this hasn't progressed further yet.

Another option would be to always use the latest Linux kernel compatible to zfs.

Sometimes they get EOL'd before the next is compatible. This means that we may have "new kernel" pointing to the LTS (e.g. 4.19 last not EOL'd version) sporadically.

This not something that is exactly under our control.

We really are between a rock and a hard place here, the kernel advancing, and zfs not being ready for stable yet for weeks at a time.

We may want to review the policy on including zfs on all installer media. We may want to make it only on the media we can support adding zfs to, meaning LTS.

While this may be confusing at first, we could be doing this by having those media:

installer_lts-kernel[_zfs]
installer_latest-kernel[_zfs]

Always specifying the kernel type in the media name, and always appending _zfs as a "feature" flag.

The only published variants would be:

installer_lts-kernel_zfs
installer_latest-kernel

This would at least reduce confusion about the features of the available media, by making it something that is described in their names. Thus, removing the "invisible" difference in features.

This example skipped over the _graphical feature flag that should also be considered the same.

installer_lts-kernel_zfs_graphical

stale · 2020-06-01T08:23:03Z

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.

aristidb · 2020-10-29T20:56:49Z

How frequent is it actually that a kernel is EOL'd before ZFSonLinux adds support for it? And is it really completely unacceptable to use an EOL'd kernel?

stale · 2021-06-04T08:56:10Z

I marked this as stale due to inactivity. → More info

domenkozar · 2021-08-02T08:48:16Z

As this is open for 2+ years, it seems like we won't make this perfect but it's better to have a live image with the latest kernel without ZFS support rather than none at all?

samueldr · 2021-08-04T01:04:44Z

I guess we could really go with the plan I previously proposed:

Removes zfs from only the "new_kernel" installer variants #59863 (comment)

And deprecate the "unqualified" iso names. We could go with that scheme without producing all variants. E.g. produce only "lts+zfs", but (try to) produce "latest+zfs", and produce "latest+none".

domenkozar · 2021-08-04T11:45:10Z

I guess we could really go with the plan I previously proposed:

Removes zfs from only the "new_kernel" installer variants #59863 (comment)

And deprecate the "unqualified" iso names. We could go with that scheme without producing all variants. E.g. produce only "lts+zfs", but (try to) produce "latest+zfs", and produce "latest+none".

Isn't that identical to having only latest that includes zfs if zfs.meta.isBroken == false?

If we wanted to improve UX it would print that ZFS is disabled, but that seems like a separate step.

samueldr · 2021-08-04T18:17:26Z

Do we know that zfs is broken at eval time?

I haven't checked, maybe we have a meta.broken set to the last known good major kernel version to build zfs.

Though there is a fundamental UX difference in always producing the unstable image without zfs, and when possible producing the one with zfs: consistency. If the "same" iso image is produced sometimes with, sometimes without, it will be harder for end-users to really know whether the current unstable iso has zfs support. If instead there is always the "unstable without zfs support" variant, and sometimes the additional "unstable with zfs" variant isn't built, the messaging stays more consistent.

So even if, at eval time, we can know zfs support is present or not, we shouldn't arbitrarily change the supported features of the published unstable iso.

Does that make sense?

domenkozar · 2021-08-04T21:22:06Z

Do we know that zfs is broken at eval time?

I haven't checked, maybe we have a meta.broken set to the last known good major kernel version to build zfs.

No way to ensure that, but ZHF does end up marking it as broken.

Though there is a fundamental UX difference in always producing the unstable image without zfs, and when possible producing the one with zfs: consistency. If the "same" iso image is produced sometimes with, sometimes without, it will be harder for end-users to really know whether the current unstable iso has zfs support. If instead there is always the "unstable without zfs support" variant, and sometimes the additional "unstable with zfs" variant isn't built, the messaging stays more consistent.

So even if, at eval time, we can know zfs support is present or not, we shouldn't arbitrarily change the supported features of the published unstable iso.

Does that make sense?

Makes sense, although the expectation that you need to use another image for zfs is an UX issue itself (and the fact that currently users are used to zfs being part of latest kernel) so there's no ideal solution.

Either way, whatever gets us this fixed sounds good to me.

domenkozar · 2021-09-20T09:03:23Z

We're losing users due to this bug: https://twitter.com/KirinDave/status/1439053837914968068

vcunat · 2021-09-20T09:27:02Z

Can you point out to me what's the breakage in his case? I see lots of green for new_kernel ISOs: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.iso_minimal_new_kernel.x86_64-linux

but they also wrote

I'm not confident in setting up NixOS without a gui

which would mean graphical ISO (right?) but we don't even have those with new_kernel and they're channel-blocking 🤷🏽

vcunat · 2021-09-20T09:31:49Z

Ah, perhaps you meant the fact that we don't have new_kernel ISOs on 21.05 anymore due to ZFS? (combination mentioned in Graham's blog post; it's meta.broken)

vcunat · 2021-09-20T09:51:56Z

The current code knows compatible version combinations and sets meta.broken based on that. I believe that should work automatically on kernel updates, etc... so perhaps include it in default based on that? (as suggested by SamuelDR)

vcunat · 2021-09-20T09:55:20Z

The only other way I can see (that haven't been mentioned) is to use kernel that's not latest upstream but a bit older which hopefully has such issues resolved (also nvidia is often reported). But that seems too complicated to me; arguably I don't personally care for those cases prone to breaking.

domenkozar · 2021-09-20T10:04:14Z

I don't see why ZFS should get special treatment, if it doesn't work on the latest kernel it should just be disabled. There are other uses of an OS than using only ZFS and latest kernel is important for newer hardware - laptops.

We could have another image with zfs and latest compatible kernel, but that's really another issue beyond fixing new_kernel.

vcunat · 2021-09-20T10:15:26Z

Well, I personally don't mind removing ZFS from new_kernel ISOs (unconditionally). Using meta.broken just seemed like a cheap compromise.

nixos-discourse · 2021-09-23T20:12:23Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/package-zfs-kernel-2-0-5-5-14-5-is-marked-as-broken-refusing-to-evaluate/15139/11

vcunat · 2021-10-02T07:59:21Z

Another option not mentioned above would be to use zfs.latestCompatibleLinuxPackages (which exists already), i.e. for new_kernel images use possibly a slightly older kernel.

Yes, it would be a special treatment for ZFS. I don't use ZFS myself, but I can understand it's hard to install with ZFS on newer HW without some image like this – such argument does not hold in other cases like nvidia drivers AFAIK.

samueldr · 2021-10-02T17:33:33Z

I will leave my current thoughts about a solution:

Re-think image generation with "features"; we currently have two features:

new_kernel
zfs

We do not need to build all features combinations. Adding features should be done only when absolutely needed.

A generation needs the following two "feature sets" to pass to advance:

[ zfs ]
[ new_kernel ]

And these installer image feature sets will be tried to be built (exhaustively):

[ zfs ]
[ new_kernel ]
[ new_kernel zfs ]

The website can only directly link to the iso images with feature sets required for advancing. (But we don't necessarily need to link to the new kernel image directly.)

The website could link to a generated page (generated by the channels script, which already copies the iso) with a list all of the built images. Images not built could (should?) provide a short blurb explaining why they aren't built (do we have that info from the Hydra build attempt, or eval?). (This generated list of images could also include other more "exotic" images like aarch64 iso and sd cards in the future.)

As for naming, nixos-XX.YY-${concatStringsSep "-" features}.iso (and similar for SD card images).

Anything missing? Anything unclear?

Oops, should have read the previous comment. newest_zfs_kernel could be a feature too, if deemed desirable. And rather than [ new_kernel zfs ], we would build [ newest_zfs_kernel zfs ] images.

stale · 2022-04-16T03:16:08Z

I marked this as stale due to inactivity. → More info

samueldr · 2022-12-23T04:23:11Z

I give up, no one else is getting sniped into fixing this. I took the time to fix this.

nixos: Add new_kernel_no_zfs image variants #207371

samueldr requested a review from infinisil as a code owner April 19, 2019 02:11

samueldr requested a review from grahamc April 19, 2019 02:12

samueldr added 2 commits April 18, 2019 22:13

nixos: build the armv7l image without zfs support

ca35830

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Apr 19, 2019

samueldr force-pushed the feature/new-kernel-no-zfs branch from 95119c6 to ca35830 Compare April 19, 2019 02:20

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Apr 19, 2019

infinisil reviewed Apr 19, 2019

View reviewed changes

samueldr added the 2.status: work-in-progress label Apr 19, 2019

samueldr mentioned this pull request Sep 28, 2019

Expose installer ISO's with a newer kernel #69687

Open

fpletz requested changes Sep 28, 2019

View reviewed changes

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 3, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 4, 2021

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 2, 2021

vcunat mentioned this pull request Aug 6, 2021

nvidia driver build failure on 21.05 #132840

Closed

vcunat mentioned this pull request Oct 2, 2021

21.11 Feature Freeze #140168

Closed

vcunat added the 9.needs: community feedback label Oct 2, 2021

jonringer added this to In progress in 21.11 Blockers via automation Oct 21, 2021

samueldr mentioned this pull request Oct 24, 2021

Advertise aarch64 installer image NixOS/nixos-homepage#747

Closed

1 task

samueldr mentioned this pull request Feb 22, 2022

nixos.sd_image_new_kernel.aarch64-linux is not building on Hydra #161296

Closed

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 16, 2022

samueldr mentioned this pull request Aug 31, 2022

ZFS enabled in nixos/modules/profiles/base.nix breaks linuxPackages_latest #189184

Open

samueldr mentioned this pull request Dec 7, 2022

nixos/release: build iso_minimal_new_kernel for stable #204773

Closed

13 tasks

samueldr mentioned this pull request Dec 23, 2022

nixos: Add new_kernel_no_zfs image variants #207371

Merged

13 tasks

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Dec 23, 2022

samueldr closed this Dec 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removes zfs from only the "new_kernel" installer variants #59863

Removes zfs from only the "new_kernel" installer variants #59863

samueldr commented Apr 19, 2019

infinisil Apr 19, 2019

grahamc Apr 19, 2019

samueldr Apr 19, 2019

infinisil Apr 19, 2019

samueldr Apr 19, 2019 •

edited

fpletz left a comment •

edited

samueldr commented Sep 28, 2019 •

edited

stale bot commented Jun 1, 2020

aristidb commented Oct 29, 2020

stale bot commented Jun 4, 2021

domenkozar commented Aug 2, 2021 •

edited

samueldr commented Aug 4, 2021

domenkozar commented Aug 4, 2021

samueldr commented Aug 4, 2021

domenkozar commented Aug 4, 2021

domenkozar commented Sep 20, 2021

vcunat commented Sep 20, 2021 •

edited

vcunat commented Sep 20, 2021 •

edited

vcunat commented Sep 20, 2021

vcunat commented Sep 20, 2021 •

edited

domenkozar commented Sep 20, 2021

vcunat commented Sep 20, 2021

nixos-discourse commented Sep 23, 2021

vcunat commented Oct 2, 2021

samueldr commented Oct 2, 2021 •

edited

stale bot commented Apr 16, 2022

samueldr commented Dec 23, 2022

Removes zfs from only the "new_kernel" installer variants #59863

Removes zfs from only the "new_kernel" installer variants #59863

Conversation

samueldr commented Apr 19, 2019

infinisil Apr 19, 2019

Choose a reason for hiding this comment

grahamc Apr 19, 2019

Choose a reason for hiding this comment

samueldr Apr 19, 2019

Choose a reason for hiding this comment

infinisil Apr 19, 2019

Choose a reason for hiding this comment

samueldr Apr 19, 2019 • edited

Choose a reason for hiding this comment

fpletz left a comment • edited

Choose a reason for hiding this comment

samueldr commented Sep 28, 2019 • edited

stale bot commented Jun 1, 2020

aristidb commented Oct 29, 2020

stale bot commented Jun 4, 2021

domenkozar commented Aug 2, 2021 • edited

samueldr commented Aug 4, 2021

domenkozar commented Aug 4, 2021

samueldr commented Aug 4, 2021

domenkozar commented Aug 4, 2021

domenkozar commented Sep 20, 2021

vcunat commented Sep 20, 2021 • edited

vcunat commented Sep 20, 2021 • edited

vcunat commented Sep 20, 2021

vcunat commented Sep 20, 2021 • edited

domenkozar commented Sep 20, 2021

vcunat commented Sep 20, 2021

nixos-discourse commented Sep 23, 2021

vcunat commented Oct 2, 2021

samueldr commented Oct 2, 2021 • edited

stale bot commented Apr 16, 2022

samueldr commented Dec 23, 2022

samueldr Apr 19, 2019 •

edited

fpletz left a comment •

edited

samueldr commented Sep 28, 2019 •

edited

domenkozar commented Aug 2, 2021 •

edited

vcunat commented Sep 20, 2021 •

edited

vcunat commented Sep 20, 2021 •

edited

vcunat commented Sep 20, 2021 •

edited

samueldr commented Oct 2, 2021 •

edited