Flakes UX: make it easy to evaluate an arbitrary expression in a context of a flake

[balsoft]

This is a design discussion issue, so feel free to give as much feedback and criticism as you like.

Preamble

In ye olde days of nix-shell, it was possible to easily evaluate expressions in the context of nixpkgs. Something like this:

nix-shell -p 'python3.withPackages (ps: with ps; [ numpy scipy ])'

Drops you into a shell in which there's a python3 executable with numpy and scipy available. This is really useful for ad-hoc scripting. You could also do something like

nix-shell -p 'hello.overrideAttrs (_: { src = ./.; })'

It was also possible to add alias p='nix-shell -p' to save some typing.

The flexibility is endless -- and all just a few keystrokes away!

Current situation

It is possible to do something similar with new, flaky interface:

nix shell --impure --expr 'with builtins.getFlake "nixpkgs"; with legacyPackages.${builtins.currentPlatform}; python3.withPackages (ps: with ps; [ numpy scipy ])'

However, this is a chore to do, and I'd much rather just run pip install numpy scipy and deal with all the usual stateful package management issues than type this monstrosity every time I need to try a python package.

Proposal

Add a way to easily evaluate expressions "in the context of" a flake, meaning with the top-level attributes and {packages,legacyPackages}.${currentPlatform} of that flake available in scope.

Possible implementations

I came up with three different ideas, all of which have some advantages and drawbacks.

(my current favourite) Allow flake fragments to be arbitrary expressions

Example: nix shell 'nixpkgs#(python3.withPackages (ps: with ps; [ numpy scipy ]))'

This feels like a natural extension of the already-existing flake-fragment-as-attrpath interpretation. It shouldn't be too difficult to implement, and doesn't break compatibility. It does however introduce some potential complexity to newcomers.

With alias p='nix shell' it is possible to do p 'nixpkgs#python3.withPackages (...)' , and some more keystrokes could be saved with a registry entry n -> nixpkgs: p 'n#python3.withPackages (...)'

Add a new flag, like --with

Example: nix shell --with nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This plays nicely with Nix' with keyword. It implies that some expression will be evaluated with some additional context.

Aliasing p='nix shell --with nixpkgs' can give experience identical to the status quo -- p 'python3.withPackages (...)'

Change the behaviour of --expr

Example: nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This breaks compatibility, but may be nicer than --with since --expr was not very usable with flakes anyways and it's not nice to have a lot of useless flags around.

[Radvendii]

+1 to nix shell nixpkgs#(<expression>) syntax

One thing I wonder about also is being able to pull multiple flakes into scope. (the equivalent of with x; with y;) Maybe with a syntax like [nixpkgs otherFlake]#(<expression>) Maybe once you get to this point it's better to make a flake.nix file and take them both in as inputs.

This may go too far down the road of complexity without enough real use-cases.I remember there was a use-case for this that came up for me a little while ago, but I don't remember what it was. I'll write it down if it ever comes up again.

EDIT: I talked it through with @balsoft and we're in agreement that my addition isn't a great move right now. It changes the semantics of flake paths away from being "urls" of sorts, which is a much bigger decision, even if we want to go there.

[thufschmitt]

I’m a big fan of the overall idea, that’s something I want to have available reasonably frequently, and it’s always a pain. I’m not sure what the UX should be though:

• The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax (like what OCaml has for modules), but it doesnt, (and for good reasons imho). So it’s a bit confusing. (and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs)
• --with <flakeref> sounds nice, but since it only makes sense with --expr, which isn’t very clean
• --expr <flakeref> <expr> isn’t really viable, because part of the point of --expr is to make it possible to use the nix command with non-flake things.

A fourth way could be to have a --expr-with <flakeref> <expr> flag. But it’s not utterly pretty either.

[balsoft]

My --with flag suggestion is actually --with <flakeref> <expr>, so like your --expr-with suggestion. But having --with <flakeref> --expr <expr> is also a possible solution.

The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax

I think the installables syntax is already different from usual Nix syntax. E.g. <flakeref>#<attrpath> will not evaluate to the same thing in actual Nix language as it does in the attributes, so I think some syntactic liberty could be allowed. But I also see your point, because it can be even more confusing for newcomers if there's even more strange syntax to learn and understand.

[Radvendii]

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

[thufschmitt]

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

Not exactly. Current nixpkgs#(foo) is (builtins.getFlake "nixpkgs").[Search path Magic]."(foo)" while with this proposal it would be (builtins.getFlake "nixpkgs").[Search path Magic].foo (so the attr name is different).
But that’s really nitpicking because that’s a really small edge-case (and the syntax is officially unstable anyways)

[edolstra]

For the new UI, we have to careful about adding features that are not discoverable or have a steep learning curve. For instance, in the example

nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

it's not obvious how a user would discover the existence of python3.withPackages or figure out that they need to write (ps: ...).

There's also the issue of cacheability. Currently we can cache flake output attributes like nixpkgs#pythonPackages.foo, but not arbitrary expressions like the above.

[zimbatm]

If we can translate CLI arguments to function calls, it's possible to build functions that work for common cases. This assumes some cooperation between nixpkgs and the nix CLI.

Eg: nix apply nixpkgs#python3.with --packages numpy,scipy

Could map to (builtins.getFlake "nixpkgs).legacyPackages.${builtins.currentSystem}.python3.with { packages = "numpy,scipy"; }

It makes me think of httpie since they also try to convert CLI to JSON data structures: https://httpie.io/docs#non-string-json-fields

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-22/16251/1

[bmabsout]

I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): nix shell nixpkgs/nixos-21.11#python39 --expr "withPackages (p: [p.numpy])" I expected python39's attributes to be in scope for the expression but it wasn't

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-for-python-packages/16575/4