Make the Derived Path family of types inductive for dynamic derivations #8369

Ericson2314 · 2023-05-18T17:54:57Z

Motivation

We want to be able to write down /nix/store/asdfasdfasdfasf-foo.drv^bar.drv^baz: /nix/store/asdfasdfasdfasf-foo.drv^bar.drv is the dynamic derivation (since it is itself a derivation output, bar.drv from foo.drv).

To that end, we create Single{Derivation,BuiltPath} types, that are very similar except instead of having multiple outputs (in a set or map), they have a single one. This is for everything to the left of the rightmost ^.

NixStringContextElem has an analogous change, and now can reuse SingleDerivedPath at the top level. In fact, if we ever get rid of DrvDeep, NixStringContextElem could be replaced with SingleDerivedPath entirely!

Context

Important note: some JSON formats have changed.

We already can produce dynamic derivations, but we can't refer to them directly. Today, we can merely express building or example at the top imperatively over time by building foo.drv^bar.drv, and then with a second nix invocation doing <result-from-first>^baz, but this is not declarative. The ethos of Nix of being able to write down the full plan everything you want to do, and then execute than plan with a single command, and for that we need the new inductive form of these types.

~~Depends on #8691~~
~~Depends on #8724~~

Not in scope

builtins.outputsOf
scheduling nested ^

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
documentation in the internal API docs
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

Ericson2314 · 2023-05-18T18:00:34Z

I am kinda of nauseated by how big this commit is, but at least it is the last semi-preparatory one changing data structures without adding much by way of new features. There are just a whole bunch of data structures that need to be changed at once.

Ericson2314 · 2023-05-18T18:06:40Z

src/libexpr/primops.cc

+// TODO dedup with copy-paste in installables.cc
+template<typename DP>
+static StorePath getStorePathFrom(const DP & derivedPath)
+{
+    return std::visit(overloaded {
+        [&](const typename DP::Built & bfd) {
+            return getStorePathFrom<SingleDerivedPath>(*bfd.drvPath);
+        },
+        [&](const typename DP::Opaque & bo) {
+            return bo.path;
+        },
+    }, derivedPath.raw());
+}
+


I was skeptical of making this a method when it just occurred in installables.cc, because it is rarely the right thing and yet awfully tempting to use as the one DerivedPath -> StorePath (no Store, no extra IO) function. But now that we are using it twice, I guess it should be...

src/libstore/store-api.hh

src/libcmd/installables.cc

src/libstore/derived-path.hh

fricklerhandwerk

Reviewed collaboratively in Nix maintainer meeting:

Add a release note that the JSON output format has changed, and possibly that it was never documented to begin with. We could update the release note if we add docs before the next release.

src/libstore/derived-path.hh

src/libstore/downstream-placeholder.cc

src/libstore/downstream-placeholder.hh

src/libstore/misc.cc

src/libstore/remote-store.cc

src/libstore/tests/derived-path.cc

fricklerhandwerk · 2023-07-10T12:31:18Z

src/libstore/misc.cc

+            return req;
+        },
+        [&](const SingleDerivedPath::Built & bfd0) -> SingleDerivedPath {
+            SingleDerivedPath::Built bfd {


This name is confusing, what does it mean?

fricklerhandwerk · 2023-07-10T12:33:53Z

src/libstore/misc.cc

+    auto & evalStore = evalStore_ ? *evalStore_ : store;
+
+    return std::visit(overloaded {
+        [&](const SingleDerivedPath::Opaque &) -> SingleDerivedPath {


@roberth suggests: We may want to rename this to DerivablePath as it's not derived yet, and also the constructors to Constant and Output.

@fricklerhandwerk

This is a part of CA derivations that we forgot to put behind the experimental feature. This was caught by @fricklerhandwerk in NixOS#8369 (comment)

@fricklerhandwerk

This is a part of CA derivations that we forgot to put behind the experimental feature. This was caught by @fricklerhandwerk in NixOS#8369 (comment)

dpulls · 2023-07-14T11:12:20Z

🎉 All dependencies have been resolved !

roberth

First part of first round.

Note to self: resume review at src/libexpr/eval.cc

src/libstore/derived-path.hh

tests/build.sh

src/libcmd/installables.cc

src/libexpr/eval-cache.cc

This makes it more useful. In general, the derivation will be in one store, and the realisation info is in another. This also helps us avoid duplication. See how `resolveDerivedPath` is now simpler because it uses `queryPartialDerivationOutputMap`. In NixOS#8369 we get more flavors of derived path, and need more code to resolve them all, and this problem only gets worse. The fact that we need a new method to deal with the multiple dispatch is unfortunate, but this generally relates to the fact that `Store` is a sub-par interface, too bulky/unwieldy and conflating separate concerns. Solving that is out of scope of this PR. This is part of the RFC 92 work. See tracking issue NixOS#6316

dpulls · 2023-07-21T12:53:23Z

🎉 All dependencies have been resolved !

tomberek · 2023-07-24T13:31:31Z

src/libstore/derived-path.hh


    GENERATE_CMP(DerivedPathOpaque, me->path);
 };

+struct SingleDerivedPath;


We could use a high-level description and/or depiction of what the various kinds of "Paths" are, their relationship, and intended meaning.

roberth · 2023-07-31T13:28:59Z

src/libexpr/eval.hh

@@ -532,12 +532,12 @@ public:
    StorePath coerceToStorePath(const PosIdx pos, Value & v, NixStringContext & context, std::string_view errorCtx);

    /**
-     * Part of `coerceToDerivedPath()` without any store IO which is exposed for unit testing only.
+     * Part of `coerceToSingleDerivedPath()` without any store IO which is exposed for unit testing only.


What would the check be?

It would be reading the derivation to check that it in fact has the outputs requested. But do we want to improve the error message for this in this already large PR? I think the division of labor between these functions did not change, right? So improving this message seems orthogonal?

Happy to do afterwords, however.

#8780 made and assigned to self.

src/libstore/derived-path.cc

src/libstore/remote-store.cc

roberth · 2023-07-31T14:17:16Z

src/libstore/store-api.hh

+ * much as possible, but where the derivation resolution doesn't exists, leave
+ * as-is.
+ */
+SingleDerivedPath tryResolveDerivedPath(Store &, const SingleDerivedPath &, Store * evalStore = nullptr);


Still need to wrap my head around this one, but other than that, I've completed a review of the full diff.

I am not even sure if it is needed, tbh.

So it is just used by Installables::toDerivations (virtual method, multiple implementations). The idea is that we want to try to get the "one up" prior derivation, and fail if it is not built.

But maybe this is bad for the same reason --derivation is bad: be explicit, and pass what you mean instead.

Either way, maybe we should just just unconditionally build the thing, and its up to you to pass some sort of --must-be-already-built if one doesn't want that.

~~I removed it, and there was problem! It's in obsidiansystems@try-resolve-derived-path if we ever need it later.~~ it is actually needed later, so it is just part of the main PR now.

We want to be able to write down `foo.drv^bar.drv^baz`: `foo.drv^bar.drv` is the dynamic derivation (since it is itself a derivation output, `bar.drv` from `foo.drv`). To that end, we create `Single{Derivation,BuiltPath}` types, that are very similar except instead of having multiple outputs (in a set or map), they have a single one. This is for everything to the left of the rightmost `^`. `NixStringContextElem` has an analogous change, and now can reuse `SingleDerivedPath` at the top level. In fact, if we ever get rid of `DrvDeep`, `NixStringContextElem` could be replaced with `SingleDerivedPath` entirely! Important note: some JSON formats have changed. We already can *produce* dynamic derivations, but we can't refer to them directly. Today, we can merely express building or example at the top imperatively over time by building `foo.drv^bar.drv`, and then with a second nix invocation doing `<result-from-first>^baz`, but this is not declarative. The ethos of Nix of being able to write down the full plan everything you want to do, and then execute than plan with a single command, and for that we need the new inductive form of these types. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

tomberek

Re-reviewed. Nothing particularly surprising; additional changes will expand the scope too much. Looking forward to the final upcoming steps that bring in the user-facing changes.

Ericson2314 added the RFC Related to an accepted RFC label May 18, 2023

Ericson2314 requested review from thufschmitt and edolstra as code owners May 18, 2023 17:54

github-actions bot added new-cli Relating to the "nix" command repl The Read Eval Print Loop, "nix repl" command and debugger store Issues and pull requests concerning the Nix store with-tests Issues related to testing. PRs with tests have some priority labels May 18, 2023

Ericson2314 force-pushed the inductive-derived-path branch from 0dd1b78 to 91cb7fd Compare May 18, 2023 17:56

Ericson2314 assigned roberth May 18, 2023

This was referenced May 18, 2023

Tracking issue for RFC 92: Dynamic derivations #6316

Open

Dynamic derivations RFC 92 #4628

Merged

Ericson2314 commented May 18, 2023

View reviewed changes

src/libstore/store-api.hh Outdated Show resolved Hide resolved

Ericson2314 commented May 18, 2023

View reviewed changes

src/libstore/store-api.hh Outdated Show resolved Hide resolved

Ericson2314 commented May 18, 2023

View reviewed changes

src/libcmd/installables.cc Outdated Show resolved Hide resolved

Ericson2314 force-pushed the inductive-derived-path branch from 91cb7fd to 88cea0e Compare June 21, 2023 19:49

Ericson2314 force-pushed the inductive-derived-path branch from 88cea0e to 988beed Compare July 10, 2023 03:34

Ericson2314 commented Jul 10, 2023

View reviewed changes

src/libstore/derived-path.hh Outdated Show resolved Hide resolved

fricklerhandwerk reviewed Jul 10, 2023

View reviewed changes

Ericson2314 force-pushed the inductive-derived-path branch from 988beed to d736e6d Compare July 13, 2023 03:06

Ericson2314 mentioned this pull request Jul 13, 2023

Feature gate DownstreamPlaceholder::unknownCaOutput #8692

Merged

8 tasks

Ericson2314 force-pushed the inductive-derived-path branch 3 times, most recently from 744a8a0 to 220fcc1 Compare July 14, 2023 12:31

Ericson2314 marked this pull request as draft July 17, 2023 14:10

roberth reviewed Jul 17, 2023

View reviewed changes

Ericson2314 force-pushed the inductive-derived-path branch 2 times, most recently from ccfe59e to 88ba8d9 Compare July 20, 2023 19:48

Ericson2314 mentioned this pull request Jul 20, 2023

Give queryPartialDerivationOutputMap an evalStore parameter #8724

Merged

8 tasks

Ericson2314 force-pushed the inductive-derived-path branch from 88ba8d9 to bc86ee0 Compare July 20, 2023 20:20

Ericson2314 force-pushed the inductive-derived-path branch from bc86ee0 to 88d77c4 Compare July 21, 2023 12:54

Ericson2314 marked this pull request as ready for review July 21, 2023 12:55

Ericson2314 force-pushed the inductive-derived-path branch 2 times, most recently from bed2acb to 07c0d6b Compare July 21, 2023 13:19

tomberek reviewed Jul 24, 2023

View reviewed changes

roberth reviewed Jul 31, 2023

View reviewed changes

src/libstore/derived-path.cc Outdated Show resolved Hide resolved

Ericson2314 force-pushed the inductive-derived-path branch from 07c0d6b to 8470751 Compare July 31, 2023 14:03

roberth reviewed Jul 31, 2023

View reviewed changes

src/libstore/remote-store.cc Outdated Show resolved Hide resolved

roberth reviewed Jul 31, 2023

View reviewed changes

This comment was marked as resolved.

Sign in to view

Ericson2314 force-pushed the inductive-derived-path branch 4 times, most recently from 245a7a1 to dbc80df Compare August 4, 2023 01:19

Ericson2314 mentioned this pull request Aug 4, 2023

Improve API docs for EvalState::coerceToDerivedPath #8780

Open

Ericson2314 force-pushed the inductive-derived-path branch from dbc80df to 60b7121 Compare August 10, 2023 04:08

tomberek approved these changes Aug 11, 2023

View reviewed changes

tomberek merged commit 010dc79 into NixOS:master Aug 11, 2023
8 checks passed

Ericson2314 deleted the inductive-derived-path branch August 11, 2023 13:50

Ericson2314 added this to the dynamic-derivations stabilisation milestone Mar 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the Derived Path family of types inductive for dynamic derivations #8369

Make the Derived Path family of types inductive for dynamic derivations #8369

Ericson2314 commented May 18, 2023 •

edited by roberth

Ericson2314 commented May 18, 2023 •

edited

Ericson2314 May 18, 2023

fricklerhandwerk left a comment •

edited

fricklerhandwerk Jul 10, 2023

fricklerhandwerk Jul 10, 2023

dpulls bot commented Jul 14, 2023

roberth left a comment

dpulls bot commented Jul 21, 2023

tomberek Jul 24, 2023

roberth Jul 31, 2023

Ericson2314 Aug 4, 2023 •

edited

Ericson2314 Aug 4, 2023

roberth Jul 31, 2023

Ericson2314 Jul 31, 2023

Ericson2314 Jul 31, 2023

Ericson2314 Aug 4, 2023 •

edited

This comment was marked as resolved.

tomberek left a comment

Make the Derived Path family of types inductive for dynamic derivations #8369

Make the Derived Path family of types inductive for dynamic derivations #8369

Conversation

Ericson2314 commented May 18, 2023 • edited by roberth

Motivation

Context

Not in scope

Checklist for maintainers

Priorities

Ericson2314 commented May 18, 2023 • edited

Choose a reason for hiding this comment

fricklerhandwerk left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpulls bot commented Jul 14, 2023

roberth left a comment

Choose a reason for hiding this comment

dpulls bot commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Aug 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Aug 4, 2023 • edited

Choose a reason for hiding this comment

This comment was marked as resolved.

tomberek left a comment

Choose a reason for hiding this comment

Ericson2314 commented May 18, 2023 •

edited by roberth

Ericson2314 commented May 18, 2023 •

edited

fricklerhandwerk left a comment •

edited

Ericson2314 Aug 4, 2023 •

edited

Ericson2314 Aug 4, 2023 •

edited