`--derivation` considered harmful, remove it in lieu of `^` and new stuff #7261

Ericson2314 · 2022-11-04T14:45:04Z

Reasons it is bad

No single meaning

--derivation is a shared flag between all commands, but only some commands care about it at all:

src/nix/diff-closures.cc:        auto beforePath = Installable::toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, before);
src/nix/diff-closures.cc:        auto afterPath = Installable::toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, after);
src/nix/why-depends.cc:        auto packagePath = Installable::toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, package);
src/nix/why-depends.cc:        auto dependencyPath = Installable::toStorePath(getEvalStore(), store, Realise::Derivation, operateOn, dependency);

Other commands force one way or the other:

 $ git grep OperateOn src/nix
src/nix/develop.cc:                getEvalStore(), store, Realise::Nothing, OperateOn::Output, {installable});
src/nix/develop.cc:            for (auto & path : Installable::toStorePaths(getEvalStore(), store, Realise::Outputs, OperateOn::Output, {bashInstallable})) {
src/nix/run.cc:        auto outPaths = Installable::toStorePaths(getEvalStore(), store, Realise::Outputs, OperateOn::Output, installables);

Still other commands force something like --derivation by hand, like nix log:

nix/src/nix/log.cc

Lines 45 to 52 in 499e99d

    
           auto log = std::visit(overloaded { 
        
               [&](const DerivedPath::Opaque & bo) { 
        
                   return logSub.getBuildLog(bo.path); 
        
               }, 
        
               [&](const DerivedPath::Built & bfd) { 
        
                   return logSub.getBuildLog(bfd.drvPath); 
        
               }, 
        
           }, b.raw());

This makes for unpredictable behavior, which is a frustrating user experience.

Doesn't scale to CA derivations

In the CA derivation world, we cannot assume a path has one unique builder. This is true both for unstable floating CA derivations, and stable fixed CA derivations (fixed output derivations). --derivation fundamentally is meaningless, or non-determinstic (picking one of many builders) in an arbitrary way. This is no good.

nix build myFixedOutput0 # produces /nix/store/asdfasdfdasf-foo
nix build myFixedOutput1 # Also produces /nix/store/asdfasdfdasf-foo
nix log /nix/store/asdfasdfdasf-foo # does what?!

Doesn't scale to RFC 92

Currently, if one passes a BuiltPath derived path, --derivation (and nix log) always exacts the drvPath part. But with RFC 92 the output path could also be a derivation! This makes --derivation confusing for users.

It also leads to a loss of expressive power. Imagine if one wants --derivation for one argument but not another, e.g. for why-depends asking why a computed derivation depends on another derivation:

nix why-depends makeDrvs.anOutputWhichIsADrv gcc # oops cannot get gcc.drvPath which is what I want

What we should do instead

Finish implementing the new ^ syntax (Low level <drvPath>^<outputName> installable syntax to match existing <highLevelInstallable>^<outputNames> syntax #4543) so it always works to select an output for every type of installable.
Make foo.drvPath an installable so we always have a nice way to select the derivation for high-level installabes.
- Will also need to sort out Make NixStringContextElem::DrvDeep opt-in, and maybe deprecate it #7910
Stop special-casing .drv (Get rid of .drv special-casing for store path installable #7600) so we always have a nice way to select the derivation itself for low-level installablles.
Make new low level nix query-derivations command, which given a store path will return all the derivations, output that are known to produce it.
Delete --derivation, in all cases make semantics the actual path not an extracted drvPath.
- nix log foo should fail for consistency (!), because output paths don't have logs: derivations have logs. It can still have a nice "did you mean...?" error message leveraging the above.

BTW if the nix log UX feels fishy, #3813 is an suggestion by me (abandoned after @edolstra thought it was too much work) that we should have a notion of build attempts which logs are keyed on instead. The idea is that one can have a variety of successful or failing build attempts of the same derivation and we can remember all their logs, rather than replacing the derivation-keyed log every time their is a new attempt. CA derivation trust map entries could also point to the (successful) attempt they came from.

In this case, instead of saying nix log should work on derivation or an output path, it needs to be given an attempt. One can look up attempts from derivations or (derivation, output) pairs. If the latter path is obtainable, we can restrict ourselves to just attempts that succeeded.

fricklerhandwerk · 2022-11-23T10:18:26Z

Can you explain the difference between high-level and low-level installables?

Ericson2314 · 2022-11-23T12:52:21Z

Low level is store-only, so just store paths with or without ^. High level is everything else.

Ericson2314 · 2022-12-13T16:39:17Z

@Radvendii based on your nice fix in #7337, I thought you might be interested in this.

Ericson2314 · 2022-12-15T17:11:13Z

#7467 tracks some of the documentation work needed for this to have a good UX.

See the change log for user-facing details. Only two lines in the tests need changing, showing that the impact of this is pretty light. Progress towards NixOS#7261

See the change log for user-facing details. Only two lines in the tests need changing, showing that the impact of this is pretty light. Progress towards NixOS#7261 Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>

Ericson2314 · 2023-01-19T20:44:37Z

@7c6f434c you might take an interest in this.

The release notes document the change in behavior, I don't include it here so there is no risk to it getting out of sync. > Motivation >> Plumbing CLI should be simple Store derivation installations are intended as "plumbing": very simple utilities for advanced users and scripts, and not what regular users interact with. (Similarly, regular Git users will use branch and tag names not explicit hashes for most things.) The plumbing CLI should prize simplicity over convenience; that is its raison d'etre. If the user provides a path, we should treat it the same way not caring what sort of path it is. >> Scripting This is especially important for the scripting use-case. when arbitrary paths are sent to e.g. `nix copy` and the script author wants consistent behavior regardless of what those store paths are. Otherwise the script author needs to be careful to filter out `.drv` ones, and then run `nix copy` again with those paths and `--derivation`. That is not good! >> Surprisingly low impact Only two lines in the tests need changing, showing that the impact of this is pretty light. Many command, like `nix log` will continue to work with just the derivation passed as before. This because we used to: - Special case the drv path and replace it with it's outputs (what this gets rid of). - Turn those output path *back* into the original drv path. Now we just skip that entire round trip! > Context Issue NixOS#7261 lays out a broader vision for getting rid of `--derivation`, and has this as one of its dependencies. But we can do this with or without that. `Installable::toDerivations` is changed to handle the case of a `DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't need to do any extra work in that case. On this basis, commands like `nix {show-derivation,log} /nix/store/...-foo.drv` still work as before, as described above. Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>

Per the old FIXME, this flag was on too many commands, and mostly ignored. Now it is just on the commands where it actually has an effect. Per NixOS#7261, I would still like to get rid of it entirely, but that is a separate project. This change should be good with or without doing that.

fricklerhandwerk · 2023-02-28T10:33:43Z

Discussed in the Nix team meeting 2022-02-27:

consensus on the end goal and the proposed roadmap
idea approved

nixos-discourse · 2023-02-28T10:42:57Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-27-nix-team-meeting-minutes-36/25890/1

edolstra · 2023-02-28T12:21:37Z

I'm not convinced removing --derivation is a good idea. In general, commands should operate on output paths by default, since .drv files are an implementation detail that might even disappear in the future. For instance, nix path-info nixpkgs#hello should show info about the outputs of hello, since that's almost always what the user cares about. I shouldn't have to use some obscure syntax like nix path-info nixpkgs#hello^* to get the outputs. Instead, it's better in the rare case that I want to get info about the .drv, to ask for that explicitly using --derivation.

Ericson2314 · 2023-02-28T15:33:13Z

@edolstra This issue does not disagree that

nix path-info nixpkgs#hello

should continue to work.

What it says is that one should do

nix path-info nixpkgs#hello.drvPath

to get information about the derivation file itself.

fricklerhandwerk · 2023-02-28T15:42:21Z

.drv files are an implementation detail that might even disappear in the future.

And so are store paths, to some extent. But that's the reference type the Nix store happens to use, so we have to expose it at the nix store level. Maybe nix path-info should actually be nix store path-info? Why would the whole package management use case (for which I assume the top level nix command is the entry point) even care about paths except as an implementation detail of how environments are set up?

The release notes document the change in behavior, I don't include it here so there is no risk to it getting out of sync. > Motivation >> Plumbing CLI should be simple Store derivation installations are intended as "plumbing": very simple utilities for advanced users and scripts, and not what regular users interact with. (Similarly, regular Git users will use branch and tag names not explicit hashes for most things.) The plumbing CLI should prize simplicity over convenience; that is its raison d'etre. If the user provides a path, we should treat it the same way not caring what sort of path it is. >> Scripting This is especially important for the scripting use-case. when arbitrary paths are sent to e.g. `nix copy` and the script author wants consistent behavior regardless of what those store paths are. Otherwise the script author needs to be careful to filter out `.drv` ones, and then run `nix copy` again with those paths and `--derivation`. That is not good! >> Surprisingly low impact Only two lines in the tests need changing, showing that the impact of this is pretty light. Many command, like `nix log` will continue to work with just the derivation passed as before. This because we used to: - Special case the drv path and replace it with it's outputs (what this gets rid of). - Turn those output path *back* into the original drv path. Now we just skip that entire round trip! > Context Issue NixOS#7261 lays out a broader vision for getting rid of `--derivation`, and has this as one of its dependencies. But we can do this with or without that. `Installable::toDerivations` is changed to handle the case of a `DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't need to do any extra work in that case. On this basis, commands like `nix {show-derivation,log} /nix/store/...-foo.drv` still work as before, as described above. When testing older daemons, the post-build-hook will be run against the old CLI, so we need the old version of the post-build-hook to support that use-case. Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>

The release notes document the change in behavior, I don't include it here so there is no risk to it getting out of sync. > Motivation >> Plumbing CLI should be simple Store derivation installations are intended as "plumbing": very simple utilities for advanced users and scripts, and not what regular users interact with. (Similarly, regular Git users will use branch and tag names not explicit hashes for most things.) The plumbing CLI should prize simplicity over convenience; that is its raison d'etre. If the user provides a path, we should treat it the same way not caring what sort of path it is. >> Scripting This is especially important for the scripting use-case. when arbitrary paths are sent to e.g. `nix copy` and the script author wants consistent behavior regardless of what those store paths are. Otherwise the script author needs to be careful to filter out `.drv` ones, and then run `nix copy` again with those paths and `--derivation`. That is not good! >> Surprisingly low impact Only two lines in the tests need changing, showing that the impact of this is pretty light. Many command, like `nix log` will continue to work with just the derivation passed as before. This because we used to: - Special case the drv path and replace it with it's outputs (what this gets rid of). - Turn those output path *back* into the original drv path. Now we just skip that entire round trip! > Context Issue NixOS#7261 lays out a broader vision for getting rid of `--derivation`, and has this as one of its dependencies. But we can do this with or without that. `Installable::toDerivations` is changed to handle the case of a `DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't need to do any extra work in that case. On this basis, commands like `nix {show-derivation,log} /nix/store/...-foo.drv` still work as before, as described above. When testing older daemons, the post-build-hook will be run against the old CLI, so we need the old version of the post-build-hook to support that use-case. Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

maralorn · 2023-03-13T14:41:40Z

I just want to contribute, that this issue bit me today. I used nix copy /nix/store/...-foo.drv in a script of mine, which subsequently led to failures because that derivation was not actually copied to the host. Having the different paths /nix/store/...-foo.drv and /nix/store/...-foo do the same thing for nix copy but then using the flag --derivation to reintroduce their difference does not feel like intuitive UX to me.

Ericson2314 · 2023-03-14T02:33:20Z

@maralorn Happy to report that since #7600 was merged, that part is already done!

roberth · 2023-03-15T16:22:29Z

--derivation also makes it non-trivial to support why-depends chains that end in a source. It seems to require an otherwise unnecessary corner case.

nix why-depends --derivation does not understand that a derivation can depend on a source #8051

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 · 2024-01-08T19:47:02Z

This was discussed in today's Nix team meeting some more

Agreement we can do without new syntax at his time
Release notes must be good to cover cases
Backport making .drvPath work
--derivation can have error message with migration instructions
@edolstra: .drvPath and .inputDerivation seem ad hoc and imposes complexity on nixpkgs, and it's not as discoverable
@roberth: users don't need to learn --derivation
@Ericson2314: --expr and --file are very different from --derivation
@roberth: installable "modes" like --expr or --derivation can not be combined on the command line
@tomberek: we can address the discoverability concerns in the manual (referenced in the deprecation message) and see how people respond to the change
@Ericson2314: make sure we judge the feedback keeping in mind that the cost of having an unnecessary feature is a subtle long term cost
is this a regression considering (Redefine outputs in terms of language-level "package", not necessarily store-level derivation (RFC-92, and multi-drv packages, docs) #6507)? No.
- @roberth: maybe for scriptability if expressions decide to put the drvPath in other places and the goal is to retrieve the drv of an output. In this case I'd prefer a syntax like nixpkgs#hello^.., because that can be mixed with non-^.. installables. However, this seems not applicable (yet?), because all packages have a drvPath today.
@Ericson2314: maybe we should have a separation such as between nix-instantiate and nix-build
@edolstra: nix-instantiate is a nice orthogonal design, but not user friendly
@Ericson2314: derivations are a plumbing-side thing
@edolstra: nix why-depends needs something like --derivation
@edolstra: plumbing commands are not a priority for new-cli work
@Ericson2314: keep the porcelain simple for new users, which is easier if we have good plumbing commands to support the advanced use cases
- rationale is at --derivation considered harmful, remove it in lieu of ^ and new stuff #7261

There is still some disagreement, but we agree on some next steps:

Steps

Make .drvPath work
- back port it to 2.19
Document .drvPath as part of explaining --derivation
Add support for ^..
- ^.. / ^* can both be thought of as bonus/extra deriving path syntax.

Agreement on 1-3. Defer 4 until later.

Some sort of warning on --derivation indicating (a) we're debating it (b) issue to look at (c) some other ways to do the same thing, etc.

nixos-discourse · 2024-01-12T08:34:24Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-08-nix-team-meeting-minutes-114/38156/1

roberth · 2024-01-12T12:17:17Z

^* selects multiple paths, unlike any of the other syntax.
You could still construct it to be algebraic by throwing everything into a list, but then we'd lose the connection with the expression language, which does not, and must not do that.

It is therefore not part of the deriving path language, but a custom, CLI+scheduler-only language of deriving path sets. (sets: I don't think they're currently treated as lists)

nixos-discourse · 2024-04-18T02:35:10Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-implement-dependency-retrieval-primitive/43418/5

nixos-discourse · 2024-10-14T11:42:01Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-10-09-nix-team-meeting-minutes-185/54335/1

Ericson2314 added the feature Feature request or proposal label Nov 4, 2022

Ericson2314 changed the title ~~--derivation considered harmful, remove it in lieu of ^ and new commands~~ --derivation considered harmful, remove it in lieu of ^ and new stuff Nov 4, 2022

Ericson2314 mentioned this issue Nov 8, 2022

Redefine outputs in terms of language-level "package", not necessarily store-level derivation (RFC-92, and multi-drv packages, docs) #6507

Open

2 tasks

Ericson2314 added the UX The way in which users interact with Nix. Higher level than UI. label Nov 8, 2022

Ericson2314 mentioned this issue Nov 18, 2022

Low level <drvPath>^<outputName> installable syntax to match existing <highLevelInstallable>^<outputNames> syntax #4543

Merged

fricklerhandwerk added the new-cli Relating to the "nix" command label Nov 23, 2022

fricklerhandwerk mentioned this issue Dec 5, 2022

nix flake update --single <name> is more intuitive than nix flake lock --update #5110

Closed

Ericson2314 mentioned this issue Jan 14, 2023

Get rid of .drv special-casing for store path installable #7600

Merged

7 tasks

Ericson2314 mentioned this issue Jan 16, 2023

A command to add a gc root #7138

Open

Ericson2314 mentioned this issue Feb 5, 2023

Scope down --derivation to just the commands that use it #7754

Merged

7 tasks

thufschmitt added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Feb 27, 2023

roberth mentioned this issue Mar 15, 2023

nix why-depends --derivation does not understand that a derivation can depend on a source #8051

Open

andreabedini mentioned this issue Jun 16, 2023

nix eval --derivation ignores the --derivation flag #5910

Open

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jun 27, 2023

Get rid of the --derivation flag

e35385e

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jun 27, 2023

Get rid of the --derivation flag

4f712fe

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 mentioned this issue Jun 27, 2023

Get rid of the --derivation flag #8594

Draft

8 tasks

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jun 27, 2023

Get rid of the --derivation flag

2f3e9e9

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jul 31, 2023

Get rid of the --derivation flag

8c18ed7

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Oct 17, 2023

Get rid of the --derivation flag

05f4679

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Oct 23, 2023

Get rid of the --derivation flag

248993d

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Oct 23, 2023

Get rid of the --derivation flag

ff207e6

Since PR NixOS#7601, we've had enough flexibility in our types of installables that it is no longer needed. Progress towards NixOS#7261

Ericson2314 mentioned this issue Oct 10, 2024

Introduce nix derivation instantiate #11506

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`--derivation` considered harmful, remove it in lieu of `^` and new stuff #7261

`--derivation` considered harmful, remove it in lieu of `^` and new stuff #7261

Ericson2314 commented Nov 4, 2022 •

edited

Loading

Ericson2314 commented Nov 4, 2022 •

edited

Loading

fricklerhandwerk commented Nov 23, 2022

Ericson2314 commented Nov 23, 2022

Ericson2314 commented Dec 13, 2022

Ericson2314 commented Dec 15, 2022

Ericson2314 commented Jan 19, 2023

fricklerhandwerk commented Feb 28, 2023

nixos-discourse commented Feb 28, 2023

edolstra commented Feb 28, 2023

Ericson2314 commented Feb 28, 2023 •

edited

Loading

fricklerhandwerk commented Feb 28, 2023

maralorn commented Mar 13, 2023

Ericson2314 commented Mar 14, 2023

roberth commented Mar 15, 2023

Ericson2314 commented Jan 8, 2024

nixos-discourse commented Jan 12, 2024

roberth commented Jan 12, 2024

nixos-discourse commented Apr 18, 2024

nixos-discourse commented Oct 14, 2024

--derivation considered harmful, remove it in lieu of ^ and new stuff #7261

--derivation considered harmful, remove it in lieu of ^ and new stuff #7261

Comments

Ericson2314 commented Nov 4, 2022 • edited Loading

Reasons it is bad

No single meaning

Doesn't scale to CA derivations

Doesn't scale to RFC 92

What we should do instead

See also

Ericson2314 commented Nov 4, 2022 • edited Loading

fricklerhandwerk commented Nov 23, 2022

Ericson2314 commented Nov 23, 2022

Ericson2314 commented Dec 13, 2022

Ericson2314 commented Dec 15, 2022

Ericson2314 commented Jan 19, 2023

fricklerhandwerk commented Feb 28, 2023

nixos-discourse commented Feb 28, 2023

edolstra commented Feb 28, 2023

Ericson2314 commented Feb 28, 2023 • edited Loading

fricklerhandwerk commented Feb 28, 2023

maralorn commented Mar 13, 2023

Ericson2314 commented Mar 14, 2023

roberth commented Mar 15, 2023

Ericson2314 commented Jan 8, 2024

Steps

nixos-discourse commented Jan 12, 2024

roberth commented Jan 12, 2024

nixos-discourse commented Apr 18, 2024

nixos-discourse commented Oct 14, 2024

`--derivation` considered harmful, remove it in lieu of `^` and new stuff #7261

`--derivation` considered harmful, remove it in lieu of `^` and new stuff #7261

Ericson2314 commented Nov 4, 2022 •

edited

Loading

Ericson2314 commented Nov 4, 2022 •

edited

Loading

Ericson2314 commented Feb 28, 2023 •

edited

Loading