Policy for allowing flake outputs in nix flake check

[thufschmitt]

nix flake check checks for “known” flake outputs (packages, devShells, etc..).

Most of these are things that Nix cares about, some are NixOS-specific things (nixosConfigurations, nixosModules, etc..), but there’s also a number of projects out there that want their own output fields (home-manager or nixops for example).

It would be nice to have a policy to decide which outputs to bless in nix flake check

[NickCao]

I think we might only check the a few blessed outputs against their designated schema, and simply ignore the rest. if someone have an additional output in their flake, they do mean that, no need for a warning to annoy them.

[fogti]

It might be a good idea to bless an outputs which starts to become common in flake projects; like lib, nixosModule outputs.
I think the reasonable thing to do about non-blessed outputs is: check if it might be a spelling mistake compared to one of the blessed outputs, and warn about that (maybe give the user a way to silence such warnings), otherwise just list them, but don't warn about them, or make it less invasive, just a small note like " (not checked for conformancy)".

[tomberek]

In addition to showing, there are differences in how they are evaluated. For example, hydraJobs accepts a nested attrset, legacyPackages will ignore some errors.

There are times when people want nested apps, or nested lib, or to be more lenient with errors and will re-use this special behavior. We should consider how to either allow flake writers to specify the behavior or to remove how they are special. (Looking for consistency)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/custom-flake-outputs-for-checks/18877/5

[edolstra]

A few ideas:

• Let's lower the severity of the warning for unrecognized flake outputs. Currently you can have arbitrary flake outputs, but nix flake check warns about them, which is annoying.
• We can add an evalChecks flake output attribute to allow a flake to provide its own self-checks.
• In the long term, we can have a type system for flake outputs. (I think this was @shlevy's original vision for flakes.)

[roberth]

• We can add an evalChecks flake output attribute to allow a flake to provide its own self-checks.

This might be a good means for flake-parts to implement global assertions

• Feature request: NixOS-style assertions support hercules-ci/flake-parts#77

tl;dr undue strictness in "prod" attrs is bad for perf and errors can block users

For some checks we don't want to perform them unless absolutely necessary, because otherwise assertions may cause all of a flake to be evaluated even when only a single attribute is needed by the user. Worse, the user may have a flake that's in a partially broken state, and the strictness induced by the assertions could block them from recovering, even if the attributes needed for recovery (e.g. the dev shell) would evaluate if it wasn't for the assertions.

• In the long term, we can have a type system for flake outputs. (I think this was @shlevy's original vision for flakes.)

flake-parts achieves this with the module system. It's the most widely used type system for Nix.
I would love to see a static type system as much as anyone, but pragmatically, the module system can already serve this need.

That said, I wouldn't go as far as to make Nix Flakes require the module system to function. Perhaps it'd make sense for such a framework to set a flake output attribute that signifies that the framework handles the checks; specifically disabling the unknown output attribute warnings.