provide getDerivationEnvironment as a primop to reference nix develop drv from Nix code

[nrdxp]

Is your feature request related to a problem? Please describe.
Say I want to write a script that first enters a devShell with nix print-dev-env before executing its task. There is currently no way to make the *-env drv produced by nix develop and nix print-dev-env referencable in the Nix code so as to make the devshell a proper dependency of the script.

devShells: writeShellScript:
writeShellScript "f.sh" ''
  # not properly tracked as a dependency of this script text
  eval $(nix print-dev-env .#shell)
  
  # proposal to treat a shell as any other derivation and simply source it
  . ${builtins.devEnv devShells.shell}
  
  # rest of the script logic
''

This may seem like a non-issue since the act of running nix print-dev-env will simply eval and build the shell deriation at runtime, but what if we want a clean separation between buildtime and runtime for environments like CI that want to track the build step for caching, etc?

You might think that you could just place a comment with a reference to the outpath of the derivation itself, but what if that derivation is a heavy build and you don't actually need the binary, you just need its build env? Also, your script will be that much lighter if it doesn't have to depend on the nix binary to make a single call.

Describe the solution you'd like
A simple primop that basically executes getDerivationEnvironment (modified appropriately) on any derivation passed to it, returning the resulting environment drv seems like it should do the trick.

Describe alternatives you've considered
Perhaps there is already a way to do this that I'm just not aware of? If we carefully craft a function in Nix code that does the same thing as getDerivationEnvironment we could theoretically end up with the exact same result, but that seems more error prone.

Additional context

In divnix/std-action we make a clear separation between eval time, build time, and run time. The build time dependencies are tracked, and that's how we can ensure they are always sent to our binary cache, even in the event that the task runtime fails. We also track packages that are already built and skip builds if possible. If we don't have an accurate representation of the dependency graph, this logic can be less effective, where the goal is to do as little work as possible.

I am finding it tricky to solve this for some of our tasks at work that call nix print-dev-env from a script. For now I have a comment referencing the derivation, but this isn't exactly right since the derivation isn't the same as the one nix print-dev-env actually calls.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-would-you-like-to-see-improved-in-nix-cli-experience/24012/9

[thufschmitt]

I would actually be very much against that precise proposal, because I would like to push for the exact opposite: move the logic behind getDerivationEnvironment outside Nix proper: Despite the fact that it's currently mostly implemented in C++, it could be implemented purely in the Nix language. Doing so would make it customizable (it is currently bash-specific and stdenv.mkDerivation-specific) and prevent Nix to implicitly depend on implementation details of Nixpkgs.
I would however very much support a mechanism to let nix print-dev-env source an externally provided getDerivationEnvironment Nix function (in which case the current logic could live inside Nixpkgs's stdenv and be available for other use-cases).

If we carefully craft a function in Nix code that does the same thing as getDerivationEnvironment we could theoretically end up with the exact same result, but that seems more error prone.

It's actually not that hard. IIRC lorri ended-up with its own implementation, and so does streamNixShellIamge from Nixpkgs.

[infinisil]

This already exists in nixpkgs as .inputDerivation on any derivation built with mkDerivation, see NixOS/nixpkgs#95536

[nrdxp]

looks like inputDerivation is pretty similar but it is not technically the same derivation. Thanks though, I can at least use that for my scripts for now. I'm all for doing this is in straight Nix and having nix develop just source the nix file. I'd just like a mechanism that keeps the two in sync so that you can produce and reference the exact same drv from Nix as from the cli.

It's actually not that hard. IIRC lorri ended-up with its own implementation, and so does streamNixShellIamge from Nixpkgs.

I didn't think that it would be hard, just that it would randomly drift from the C++ implementation as time moves forward. Having the canonical implementation in Nix sounds reasonable though. I might even be able to implement it myself to close this.

We already have inputDerivation, but maybe we can just move the code for that into the Nix source tree so as not to make nixpkgs a hard dependency?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-would-you-like-to-see-improved-in-nix-cli-experience/24012/14

[nrdxp]

@infinisil, the inputDerivation doesn't seem to actually set the PATH for the buildInputs, so it's not really appropriate to source to enter a devshell as is after trying it.

edit
And on further testing, manually trying to set the PATH, SBT doesn't like the environment. I need the exact build environment of the package itself here or SBT has a fit.

[roberth]

# proposal to treat a shell as any other derivation and simply source it

A derivation can't be sourced. It's nothing more than an instruction to run any builder.
The fact that nix develop makes all sorts of assumptions about what's in a derivation is a layering violation, and its implementation should be phased out for something that builds a specific package attribute which produces a command that launches a development environment.

What you're looking for needs to be done at the Nixpkgs level. That's the component that is aware of what is inside some derivations, and it may perhaps let you query such things. Or you could assume that it's stdenv-based and just source stdenv.sh yourself that way.

If nix shells were implemented this way in the first place, your use case would have been trivially easy: just return that "specific package attribute". Let's call it pkg.devShellLauncher.

[nrdxp]

Well I've already almost reproduced the same derivation (same output, but not same derivation hash due to minor differences) as getDerivationEnvironment. Basically all I had to do was override the derivation args with the contents of the get-env.sh header from the C++. From there, in order to make it sourceable from another script I basically need to reproduce the behavior of nix print-dev-env in a nix build to produce a sourcable snippet as an output. Without that, I can still use derivation I've got to add the shell dependencies to the sourcing script, allowing my CI to properly build it during its build phase, but I still need nix in scope at runtime to call nix print-dev-env on this manufactured shell env to actually enter it.

What I got so far is without any dependency on nixpkgs by calling derivation on the overriden drvAttrs.

I was already thinking that maybe we should attach this at the attribute level as meta-data, to do basically exactly what you suggest. But devshells are so prominent in Nix, it seems to make sense to do this as a built in functionality. That is, basically every derivation has to be build, that build has some sort of environment, so every derivation could have a devShell attribute.

If the end result is referencable in Nix code, it seems to me that it matters less if the implementation is in Nix or C++.

[roberth]

But devshells are so prominent in Nix, it seems to make sense to do this as a built in functionality.

libc is also very prominent in Nix, but we don't build it in, so that we can change it out and customize it to fit our needs. This has worked out rather well, spawning an entire ecosystem of very successful projects, each maintaining their own libc without our involvement, to fit the needs of various libc-consuming niches.

We've grown accustomed to nix-shell's architectural mistake, but we can do better and the shell-related nix commands still have a role to play by enabling a proper interface. We just stop hardcoding the implementation, or perhaps provide the current hardcoded one as a fallback.

[nrdxp]

I'm not voting for one way or the other, just that we at least have a canonical implementation which the cli uses, but can also be referenced from in plain old Nix code.

Relying on all of nixpkgs feels like a mistake when its so simple to do it raw nix, but maybe its not, or maybe we can just have it live somewhere else.

[blaggacao]

I copy @roberth 's principled appreciations and the assertion that we've become path-dependent on architectural mistakes.

I would hope that this subject may even enter the Nixpkgs Architecture Team in its appropriate moment.

I can follow the argument of the nix (CLI) consuming a well defined interface but otherwise leave the implementation to nix-lang code.

The layer violation might be actually one of layer collapse. Such that there isn't a layer for canonical implementations in between nix & nixpkgs. A layer that in an alternative reality, one might recognize as nixpkgs.lib.

Being able to swap that for nix-community/nixpkgs.lib may convince us that we can have that separation if we want and hence the coupling isn't imperative.

That means an implementation in nixpkgs.lib seems accurate and concise.

[roberth]

A layer that in an alternative reality, one might recognize as nixpkgs.lib.

I suppose that would be an alternate reality where lib isn't just about "pure," non-derivation functions, but also a few things centered around setup.sh, or perhaps even the whole stdenv, but certainly mkDerivation.
So while your line of reasoning isn't wrong, let's not make dismantling the monorepo a prerequisite to defining a new package attribute. (And let's not dismantle it at all)

[blaggacao]

let's not make dismantling the monorepo a prerequisite

Not at all. Apologies for my impure speech: my argument is that nix-community/nixpkgs.lib is enough capability to (optionally) break the tight coupling in such a way, that the argument nixpkgs might be a too heavy dependency is no longer true.

[roberth]

I figured I can't just keep preaching about this idea forever, so here's some code to illustrate the idea.

• POC: devShell interface nixpkgs#206728

[nrdxp]

Is there some document already that explains why this is an architectural mistake? Also, how do you account for the fact that the passthru.devShell actually evaluates to something that we can source as a shell? Seems folks could just stuff non-sense there and break things, or inject a shell that wouldn't even be relavent to the current derivation. Here I think a built in functionality would actually be superior to guarantee that this is the actual build enviroment of a given package, and not some offshoot.

Although I accept that isn't always what we want either. Some times I want the build environment, sometimes I want a more convenient interface like numtide/devshell.

Assuming we can find some solutions though, I'm not against just evaluating a derivation attribute, but then I'm not entirely convinced that shells built into Nix is neccesarily a mistake either, just that the current implementation definitely is. Derivations already produce some meta attributes that are built in, not dependant on nixpkgs, such as the drvAttrs, et al.

It's possible that we need to mimic the flakes API here and do a passthru.devShells instead, where perhaps the default is a builtin attribute guaranteeing us a replica of the derivations build enviroment, but users can add others for various purposes?

[blaggacao]

So, the way I see it is that @nrdxp, you would like to see language level contracts on the drv that represents:

• the drv
• minus the actual build (aka builtins.unsafeDiscardOutputDependency)
• plus the environment (but different from cp ${drv.inputDerivation} $out/environment.sh - @nrdxp I didn't fully understand how this is not suitable, but I also didn't bother to compare. Can you pin-point the difference?)

actually evaluates to something that we can source as a shell?

A broken nix CLI -level contract is still a broken contract and under reasonable assumptions indicative of bad faith (or negligence), anyway. I'm not sure we need to consider bad faith when we establish contracts (nor negligence in nixpkgs.lib). So potentially a lib implementation doesn't materialize (in the scope of good faith and care) your proposed disadvantage.

It's possible that we need to mimic the flakes API here and do a passthru.devShells instead, where perhaps the default is a builtin attribute guaranteeing us a replica of the derivations build enviroment, but users can add others for various purposes?

That is an interesting idea and maybe NixOS/nixpkgs#206728 would want to consider and adopt or discard this?

[blaggacao]

Can you pin-point the difference?

I guess, it's a fair answer to reference (still very fresh): divnix/std@c54501e

[roberth]

@nrdxp

Is there some document already that explains why this is an architectural mistake?

Closest thing is my analysis of shell uses cases buried here #4715 (comment)

Also, how do you account for the fact that the passthru.devShell actually evaluates to something that we can source as a shell?

This can be part of the contract, as blaggacao suggests.

guarantee that this is the actual build enviroment of a given package

haskellPackages has the opposite problem where the build environment is not suitable as a development environment. That's why its users must specify nix develop .#my-hs-pkg.env.
There are no guarantees. Users may equally well stuff random values in drvPath and cause things to break.

@blaggacao

minus the actual build (aka builtins.unsafeDiscardOutputDependency)

My use of this in the proof of concept is as an implementation detail. A shell could equally well be constructed from the mkDerivation parameters without a .drv. That .drv won't be built anyway.

I'd like to focus on the interface alone: a pkg.devShell executable and perhaps a file "${pkg.devShell.shellData}/environment.sh" for sourcing.
Perhaps we could simplify the latter to "${pkg.devShell}/environment.sh". In this case the devShell derivation must produce both an executable and an environment.sh file.