Impure derivations #6227

edolstra · 2022-03-10T21:37:26Z

This is a revived implementation of 18c512d.

Impure derivations are derivations that can produce a different result every time they're built. Example:

stdenv.mkDerivation {
  name = "impure";
  __impure = true; # marks this derivation as impure
  #outputHashAlgo = "sha256"; # optional, default is sha256
  #outputHashMode = "recursive"; # optional, default is recursive
  buildCommand = "date > $out";
};

Some important characteristics:

This requires the 'impure-derivations' experimental feature.
Impure derivations are not "cached". Thus, running "nix-build" on the example above multiple times will cause a rebuild every time.
They are implemented similar to CA derivations, i.e. the output is moved to a content-addressed path in the store. The difference is that we don't register a realisation in the Nix database.
Only impure or fixed-output derivations are allowed to depend on impure derivations directly. "Pure" (i.e. input-addressed or floating CA derivations) can depend on impure derivations indirectly, if there is a fixed-output derivation in between. Thus the fixed-output derivation forms an "impurity barrier" in the dependency graph.
When sandboxing is enabled, impure derivations can access the network in the same way as fixed-output derivations. In relaxed sandboxing mode, they can access the local filesystem.

A motivating example for impure derivations is the problem of using fetchurl on a dynamically generated tarball whose contents are deterministic, but where the tarball does not have a canonical form. Previously, this required fetchurl to do the unpacking in the same derivation. (That's what fetchzip does.) But now we can separate these two steps:

tarball = stdenv.mkDerivation {
  __impure = true;
  name = "tarball";
  buildInputs = [ curl ];
  buildCommand =
    "curl --fail -Lk https://github.com/NixOS/patchelf/tarball/c1f89c077e44a495c62ed0dcfaeca21510df93ef > $out";
};

unpacked = stdenv.mkDerivation {
  name = "unpacked";
  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "1jl8n1n36w63wffkm56slcfa7vj9fxkv4ax0fr0mcfah55qj5l8s";
  buildCommand =
    "mkdir $out; tar xvf ${tarball} -C $out";
};

Another application could be impure (final) steps in CA pipelines like Hydra jobsets, like a job that uploads a build result somewhere.

Ericson2314 · 2022-03-11T01:26:30Z

src/libstore/derivations.hh

+/* Impure output which is moved to a content-addressed location (like
+   CAFloating) but isn't registered as a realization.
+ */
+struct DerivationOutputImpure
+{
+    /* information used for expected hash computation */
+    FileIngestionMethod method;
+    HashType hashType;
+};
+


I don't think we need this? I didn't think we did, and I went through and indeed it seems this always handled the same way as a CA output.

src/libstore/derivations.hh

Ericson2314 · 2022-03-11T01:43:16Z

Pure derivations are not allowed to depend on impure derivations. In the future fixed-output derivations will be allowed to depend on impure derivations, thus forming an "impurity barrier" in the dependency graph.

Actually, even that not the only way to get back to purity. Once a pure derivation is resolved, we don't where any of it's CA imputs came from --- since they are "just" data! --- and that means we don't care whether those imputs came from CA derivations. What this means is that pure unresolved derivations perhaps can only be impurely resolved, so we can use our in-memory resolution cache to finish the job once pure resolution gets stuck, at the cost of "tainting" any downstream resolutions.

If we squint and imagine a pure derivation is a function a -> b, what the above means is that impurity is a functor, and we are in effect mapping our derivation function to impure a -> impure b.

All that said, for impure a -> b, yes a fixed output derivation is needed. No way to "map" a derivation to that type.

src/libstore/build/derivation-goal.cc

thufschmitt

That’s great 😎

I managed to break a number of things (which is expected given that this adds a whole new derivation type, but better be aware of them). I don’t think these should be blockers for this PR (afaik, nothing breaks as long as there’s no impure derivation involved), just something to keep in mind for later:

nix-build

$ nix-build impure-derivations.nix -A impure
warning: unknown experimental feature 'ca-references'
this derivation will be built:
  /tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv
warning: unknown experimental feature 'ca-references'
building '/tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv'...
impure
nix-build: src/nix-build/nix-build.cc:585: void main_nix_build(int, char**): Assertion `maybeOutputPath' failed.
[1]    4184961 IOT instruction (core dumped)  nix-build impure-derivations.nix -A impure

nix-shell

$ nix-shell impure-derivations.nix -A impureOnImpure
warning: unknown experimental feature 'ca-references'
this derivation will be built:
  /tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv
warning: unknown experimental feature 'ca-references'
building '/tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv'...
impure
warning: output 'out' of input '/tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv' missing, aborting the resolving
nix-shell: src/nix-build/nix-build.cc:410: void main_nix_build(int, char**): Assertion `resolvedDrv && "Successfully resolved the derivation"' failed.
[1]    7110 IOT instruction (core dumped)  nix-shell impure-derivations.nix -A impureOnImpure

nix develop

$ nix develop -f impure-derivations.nix impureOnImpure
warning: unknown experimental feature 'ca-references'
error: pure derivation '/tmp/nix-test/default/store/gw8z2q3qnc6bx1ap3s8v81fbvd5qxlrr-impure-on-impure-env.drv' depends on impure derivation '/tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv'

Trying to copy an impure derivation will cause Nix to segfault:

$ nix copy --to /tmp/nix --file impure-derivations.nix impure
[1/0/1 built] building impure: impurenix: src/libstore/derived-path.cc:106: nix::BuiltPath::toRealisedPaths(nix::Store&) const::<lambda(const Built&)>: Assertion `thisRealisation' failed.
[1]    4183600 IOT instruction (core dumped)  nix copy --to /tmp/nix --file impure-derivations.nix impure

nix why-depends is broken too:

$ nix why-depends --file impure-derivations.nix inputAddressedAfterCA impure
warning: unknown experimental feature 'ca-references'
this derivation will be built:
  /tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv
error: cannot operate on an output of unbuilt content-addressed derivation 'sha256:ddffc26775f61118da8597d1f2f62c544ec92461b31b5b1075ed8689a7ee8292!out'

import-from-derivation (though I’m not sure that’s something that we want to support):

$ nix eval --impure --expr 'import (import ./impure-derivations.nix).impure'
error: output 'out' of derivation '/tmp/nix-test/default/store/hhrsdj65kvf1n0d5y1yj6qc21dqr0gki-impure.drv' has no store path mapped to it

      … while realising the context of path '/135sba7ahqvizkd4xq2cbpwm2l4dy8dqli0ym1jmrsxjn975pz5w'

      at «string»:1:1:

            1| import (import ./impure-derivations.nix).impure
            | ^

(but as I said, these are just stuff to keep in mind for later, definitely not a criticism of the PR)

src/libstore/build/derivation-goal.cc

tests/impure-derivations.sh

nixos-discourse · 2022-03-18T14:35:19Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-26/18252/1

Ericson2314 · 2022-03-18T18:28:35Z

I am fixing the conflicts in this.

Ericson2314 · 2022-03-18T18:54:19Z

I must say, while the BuildResult returning is convenient, the fact that we using it to drift away from the good "Command Query Responsibility Segregation" that we had before makes me nervous.

I think I would rather create a notion of transactions.

struct LayeredTrustMapStore: Store {
// some trust map functions,
// letting us know whether the results came from this store
// of the underlying store.
}

ref<LayeredTrustMapStore> Store::createTransacation();

The idea is that

All the commands @thufschmitt mentioned should work again.
Thanks to virtual we have the option of e.g. persisting impure trust maps in DB but segregated.
Interesting overlap with https://gist.github.com/edolstra/afa5a41d4acbc0d6c8cccfede7fd4792
Impure and pure flows re trust maps should be more similar: same how just different where the mapping goes.

Ericson2314 · 2022-03-18T19:18:16Z

master...Ericson2314:impure-derivations-ng Here is the rebase. Note I didn't (intentionally) change any design --- my code review comments are separate from this just, which is just fixing the conflicts I caused recent on master.

Impure derivations are derivations that can produce a different result every time they're built. Example: stdenv.mkDerivation { name = "impure"; __impure = true; # marks this derivation as impure outputHashAlgo = "sha256"; outputHashMode = "recursive"; buildCommand = "date > $out"; }; Some important characteristics: * This requires the 'impure-derivations' experimental feature. * Impure derivations are not "cached". Thus, running "nix-build" on the example above multiple times will cause a rebuild every time. * They are implemented similar to CA derivations, i.e. the output is moved to a content-addressed path in the store. The difference is that we don't register a realisation in the Nix database. * Pure derivations are not allowed to depend on impure derivations. In the future fixed-output derivations will be allowed to depend on impure derivations, thus forming an "impurity barrier" in the dependency graph. * When sandboxing is enabled, impure derivations can access the network in the same way as fixed-output derivations. In relaxed sandboxing mode, they can access the local filesystem.

Ericson2314 · 2022-03-31T18:25:55Z

src/libstore/derivations.hh

 > _DerivationTypeRaw;

 struct DerivationType : _DerivationTypeRaw {
    using Raw = _DerivationTypeRaw;
    using Raw::Raw;
    using InputAddressed = DerivationType_InputAddressed;
    using ContentAddressed = DerivationType_ContentAddressed;
-
+    using Impure = DerivationType_Impure;


@edolstra I am a bit annoyed you added this and didn't wait for code review. A huge point of my refactor of the derivation type was that we can represent all pure/sandboxing combinations in the content addressed case, rather than having another tacked on variant ad-hoc-ly. In addition to commenting to this effect prior, the merge conflict resolution I linked in the PR thread implemented all this in the code so you could see exactly what I am talking about.

I know you are BDFL and don't have to care about what others write on your PRs, but it would be nice even if you disagree to at least acknowledge you read and understood the comments others leave.

(I was about to compare this to the BuildResult issue, but I see in #6311 (comment) you did in fact reply again, and I missed that and didn't link the PR I had opened (until now), so that is my fault for not following up again there more promptly.)

Ericson2314 · 2022-03-31T18:33:41Z

src/libstore/derivations.cc

        [](const ContentAddressed & ca) {
-            return !ca.pure;
+            return true;


A ContentAddressed that is neither fixed-output or sandboxed is impure.

nixos-discourse · 2022-04-01T11:29:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-27/18433/1

ghost · 2022-09-16T08:18:58Z

Is there a timeline for de-experimentalizing this feature?

Along with NixOS/nixpkgs#188587 and #7052 it will allow us to eliminate all those "remember not to use fetchpatch here" comments all over the place in nixpkgs.

# Note: this package is used for bootstrapping fetchurl, and thus
# cannot use fetchpatch! All mutable patches (generated by GitHub or
# cgit) that are needed here should be included directly in Nixpkgs as
# files.

Some of these banners are extraneous, others are missing, and all of them are unchecked. It would be great if we could just remove the restriction instead.

thufschmitt · 2022-09-16T10:12:40Z

Is there a timeline for de-experimentalizing this feature?

Afaik no, but one blocker (at least) for Nixpkgs is that hydra currently doesn't support it, and it's probably not a trivial change

ghost · 2022-09-16T17:38:56Z

hydra currently doesn't support it, and it's probably not a trivial change

Hrm, what version of Nix does Hydra run (and is that documented anywhere)?

I assume from your comment that it must be running something older than 2.8, because since Nix 2.8 enabling impure derivations has been a trivial change (--extra-experimental-features impure-derivations).

thufschmitt · 2022-09-16T18:32:20Z

Hydra is generally pretty close from Nix master, but it's not that simple unfortunately, because Hydra actually re-implements part of the Nix build loop, which means that features like that requires changing hydra.

edolstra requested review from thufschmitt and Ericson2314 March 10, 2022 21:37

edolstra force-pushed the impure-derivations-ng branch from c9a58c3 to 871069d Compare March 10, 2022 21:45

Ericson2314 reviewed Mar 11, 2022

View reviewed changes

src/libstore/derivations.hh Outdated Show resolved Hide resolved

edolstra commented Mar 11, 2022

View reviewed changes

src/libstore/build/derivation-goal.cc Outdated Show resolved Hide resolved

edolstra marked this pull request as ready for review March 11, 2022 12:32

thufschmitt reviewed Mar 14, 2022

View reviewed changes

src/libstore/build/derivation-goal.cc Outdated Show resolved Hide resolved

tests/impure-derivations.sh Outdated Show resolved Hide resolved

Ericson2314 mentioned this pull request Mar 17, 2022

Clean up DerivationOutput, and headers #6283

Merged

Ericson2314 mentioned this pull request Mar 18, 2022

Simplify the handling of the hash modulo #6268

Merged

edolstra added 4 commits March 31, 2022 13:43

Support fixed-output derivations depending on impure derivations

18935e8

tests/impure-derivations.sh: Restart daemon

b2ae922

Fix test

162beb2

edolstra force-pushed the impure-derivations-ng branch from abb0ba2 to 162beb2 Compare March 31, 2022 11:43

edolstra added 8 commits March 31, 2022 15:59

Fix macOS build

d7fc33c

Document isPure()

4e043c2

needsNetworkAccess() -> isSandboxed()

e279fbb

Rename 'pure' -> 'sandboxed' for consistency

6051cc9

Fix macOS build

a99af85

Provide default values for outputHashAlgo and outputHashMode

7537097

Update release notes

d63a5f5

tests/impure-derivations.sh: Ensure that inputAddressed build fails

6377442

edolstra changed the title ~~Add support for impure derivations~~ Impure derivations Mar 31, 2022

edolstra merged commit c9a29d0 into master Mar 31, 2022

edolstra deleted the impure-derivations-ng branch March 31, 2022 17:58

Ericson2314 reviewed Mar 31, 2022

View reviewed changes

Ericson2314 mentioned this pull request Mar 31, 2022

Get rid of impureOutputHash; fix possible bug #6346

Open

7 tasks

tomberek mentioned this pull request May 12, 2022

Impure derivations #520

Open

Kha mentioned this pull request Jul 2, 2022

nix-build: requires non-existent output 'out' from input derivation #6572

Closed

ghost mentioned this pull request Sep 16, 2022

allow fetchpatch everywhere, without restrictions NixOS/nixpkgs#188587

Closed

7 tasks

Artturin mentioned this pull request Nov 3, 2022

tests/impure-derivations.sh: remove unknown experimental feature 'ca-… #7257

Merged

bryanhonof mentioned this pull request Apr 13, 2023

Move documentation for impure derivation #8212

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Impure derivations #6227

Impure derivations #6227

edolstra commented Mar 10, 2022 •

edited

Loading

Ericson2314 Mar 11, 2022

Ericson2314 commented Mar 11, 2022 •

edited

Loading

thufschmitt left a comment

nixos-discourse commented Mar 18, 2022

Ericson2314 commented Mar 18, 2022

Ericson2314 commented Mar 18, 2022 •

edited

Loading

Ericson2314 commented Mar 18, 2022 •

edited

Loading

Ericson2314 Mar 31, 2022 •

edited

Loading

Ericson2314 Mar 31, 2022

Ericson2314 Mar 31, 2022

nixos-discourse commented Apr 1, 2022

ghost commented Sep 16, 2022

thufschmitt commented Sep 16, 2022

ghost commented Sep 16, 2022

thufschmitt commented Sep 16, 2022

Impure derivations #6227

Impure derivations #6227

Conversation

edolstra commented Mar 10, 2022 • edited Loading

Ericson2314 Mar 11, 2022

Choose a reason for hiding this comment

Ericson2314 commented Mar 11, 2022 • edited Loading

thufschmitt left a comment

Choose a reason for hiding this comment

nixos-discourse commented Mar 18, 2022

Ericson2314 commented Mar 18, 2022

Ericson2314 commented Mar 18, 2022 • edited Loading

Ericson2314 commented Mar 18, 2022 • edited Loading

Ericson2314 Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

Ericson2314 Mar 31, 2022

Choose a reason for hiding this comment

Ericson2314 Mar 31, 2022

Choose a reason for hiding this comment

nixos-discourse commented Apr 1, 2022

ghost commented Sep 16, 2022

thufschmitt commented Sep 16, 2022

ghost commented Sep 16, 2022

thufschmitt commented Sep 16, 2022

edolstra commented Mar 10, 2022 •

edited

Loading

Ericson2314 commented Mar 11, 2022 •

edited

Loading

Ericson2314 commented Mar 18, 2022 •

edited

Loading

Ericson2314 commented Mar 18, 2022 •

edited

Loading

Ericson2314 Mar 31, 2022 •

edited

Loading