Purely functional metadata-rich derivation poisoning

[shlevy]

Problem: We want broken/unfree/etc. packages (and their dependents) to refuse to evaluate and build without specific user request and to have an unpolluted hydra eval errors tab. Exception throwing does anything more structured than "yes this evaluates"/"no this doesn't" violates the purely functional nature of nix: The order of evaluate affects the result that a function catching an exception will see.

Proposed solution: Add a "poisoned derivation" type, which infects any derivations that depend on it in a non-eval-order dependent way and which can be introspected for arbitrary metadata but which by default simply causes an exception to be thrown.

Example: Package foo is marked broken, package bar depends on foo. nix-build -A bar throws "foo is marked broken, add allowBroken to try anyway" as soon as foo is evaluated, whereas:

let
  maybe-poisoned = builtins.diagnosePoison pkgs.bar;
in if maybe-poisoned.poisoned && lib.only-broken-or-unfree maybe-poisoned.poisons
  then { /* some set which doesn't reference bar */}
  else { /* some set which does reference bar (via maybe-posoned.value), which either evaluates fine or throws if bar is poisoned for some reason besides being broken or unfree */ }`

will evaluate to the first set, unless bar is also poisoned for some other reason we don't want to ignore. The specifics of the poisons attribute I haven't yet worked out, but can be made evaluation-order independent.

Alternative: Instead of the diagnosePoison builtin, we can just have different eval modes, where nix-build says "throw on poison" and hydra says "don't throw on poison" and poisoned drvs can be queried directly for poison status.

@edolstra I will implement this if it is an acceptable solution.

See also #1000 and NixOS/nixpkgs#7830

[shlevy]

@copumpkin @domenkozar

[Ericson2314]

This is exactly what I've wanted!!!!! Honestly I'd even go with no exception at all, as relying on the metadata of broken packages doesn't seem that bad. (Maybe I'm naive here.)

[shlevy]

@Ericson2314 So what would nix-build -A bar do? inspect the top-level drv for poison in C++? I guess that works but it does end up doing more work than needed.

[Ericson2314]

Just work, I was thinking.

[shlevy]

The whole point is we don't want to build broken packages by default

[Ericson2314]

Err I misread the example and thought bar always only depended on the metadata of foo.

[copumpkin]

First off, general 👍 on this direction.

As I was trying to think this through, I thought about how I might model dependencies in Haskell: I'd probably make a simple Applicative that let me depend on things, along the lines of this post. I could then reconstruct your notion of "poison" by enumerating the dependencies and accumulating poison up the dependency tree.

That then got me wondering if a simpler solution would be to just expose derivation dependencies through some sort of magic meta attribute or builtin. If I can enumerate which derivations someone depends on (not just buildInputs, but all the random crap people use in string contexts and the like) then I can reconstruct your notion of poison in Nix itself.

Does that sound more difficult to implement than poison? Not sure if I'm missing something that'll kill my proposal. It seems like ultimately your poison builtin would need to decide where to propagate its poison to, and that's just what I'd call a "dependency" above. So in some sense it's just a generalization, but maybe it generalizes too far?

[shlevy]

Yeah, I was actually going to first try to implement this by replacing the normal string context, where strings can refer to paths, with arbitrary serializable values, then add nix primops to manipulate the context. This would require either special-casing paths in context as magic in nix (because derivationStrict needs to know which values in context to care about and which not to) or implementing most of derivationStrict in nix itself and just having a primop that takes the full struct derivation (i.e. you have to pass in inputDrvs and inputSrcs in nix, rather than them being inferred from context).

This would be useful for a lot more than just poison and would move much of the infrastructure to nixpkgs, but it may make things significantly slower. Waiting on @edolstra's approval for the overall project before I dig in.

[Ericson2314]

I'm reminded of NixOS/nixpkgs#11094. If we do go the poisoned derivation route, reflecting on poisoning would just be a matter of doing `"${poison}" == "${myDerivation}"``.

implementing most of derivationStrict in nix itself and just having a primop that takes the full struct derivation (i.e. you have to pass in inputDrvs and inputSrcs in nix, rather than them being inferred from context).

That is very interesting.

[shlevy]

@edolstra Still would be interested in doing this

[copumpkin]

This would also be great for tracking licenses in a fully reliable manner. I want to be able to enumerate licenses used by a package I care about.

[roberth]

That then got me wondering if a simpler solution would be to just expose derivation dependencies through some sort of magic meta attribute or builtin.

We could add this to the .drv without causing unnecessary rebuilds. This might not be optimal performance-wise for also catching this in the expression language, but it would allow external tools to perform analysis.

• Have our cake and eat it too derivation metadata #10780