nixos/flake: set up NIX_PATH and system flake registry automatically #254405

lf- · 2023-09-10T12:45:46Z

Currently there are a bunch of really wacky hacks required to get nixpkgs path correctly set up under flake configs such that nix run nixpkgs#hello and nix run -f '<nixpkgs>' hello hit the nixpkgs that the system was built with. In particular you have to use specialArgs or an anonymous module, and everyone has to include this hack in their own configs.

We can do this for users automatically.

I have tested these manually with a basic config; I don't know if it is even possible to write a nixos test for it since you can't really get a string-with-context to yourself unless you are in a flake context.

cc @RaitoBezarius

Description of changes

Things done

IreneKnapp

overall looks great, I want this!

IreneKnapp · 2023-10-02T08:30:35Z

nixos/modules/misc/nixpkgs.nix

+        default = false;
+
+        description = mdDoc ''
+          Whether to create /etc/nix/inputs/nixpkgs as a symlink and set


This could use more documentation about where the symlink points to. I mean, I assume it points to the path to the derivation that nixos is being built out of, but there must be an appropriately precise way to say that...

IreneKnapp · 2023-10-02T08:32:59Z

nixos/modules/misc/nixpkgs.nix

+        default = false;
+
+        description = mdDoc ''
+          Whether to set nixpkgs in the local flake registry and disable the


Similarly, I assume this points to the derivation nixos is built out of, but find some precise way to say that. Also specify explicitly whether this option relies on the /etc/nix/inputs/nixpkgs symlink to exist or not.

Also, it's called the system registry not the local registry. Since that term is vague and confusing and doesn't help a user to understand why they'd want this option turned on, by all means keep the present wording about "local" as well, but do use it in conjunction with the official name.

Ah, not quite the one nixos is built out of, which means my docs are bad. I will write better ones than I wrote while originally writing this whilst tired at nixcon tomorrow :)

Apparently I misunderstood what you wrote because it was 2am. This option doesn't require /etc/nix/inputs/nixpkgs to exist, and is completely independent of the other option.

it sure was 2am <3

lf- · 2023-10-02T21:43:28Z

Alright, I have pushed a change rewriting the documentation to have examples and to use more clear wording. I have also turned it on by default.

The reasoning for enabling it by default on flake configurations is that we do things that are suboptimal for closure size by default, for example environment.noXlibs = false;, documentation.enable = true;, and it seems much more important to make the out of the box experience better for machines that might be used interactively.

This change only affects systems built with flakes.

IreneKnapp

looks solid to me! note that this is by way of peer review, I'm not a committer, somebody else will need to approve it

IreneKnapp · 2023-10-02T22:29:20Z

nixos/modules/config/nix.nix

@@ -332,6 +332,16 @@ in
                always allowed to connect.
              '';
            };
+
+            flake-registry = mkOption {
+              type = types.str;


don't you mean nullOr str?

Yes, I didn't realize null was coerced to empty string.

IreneKnapp · 2023-10-02T22:31:27Z

nixos/modules/misc/nixpkgs.nix

+
+        description = mdDoc ''
+          Whether to create `/etc/nix/inputs/nixpkgs` as a symlink to the version of nixpkgs that
+          the NixOS system was built with and set `NIX_PATH` to include `/etc/nix/inputs/nixpkgs`.


just to be absolutely clear: the symlink points to a path in the store, right?

Yep. I will word this more carefully.

IreneKnapp · 2023-10-02T22:33:43Z

nixos/modules/misc/nixpkgs.nix

+
+        description = mdDoc ''
+          Whether to pin nixpkgs in the system-wide flake registry (`/etc/nix/registry.json`) to the
+          version of nixpkgs that the NixOS system has been built with.


so the understanding I get from this is that it replicates the protocol, URL, etc but pins based on rev. correct? I still feel like there could be a bit more wording to say so explicitly, but it's good as-is.

Nope. It is a string-with-context that introduces a simple dependency:

nix-repl> nixosConfigurations.micro.options.nix.registry.value.nixpkgs.to { path = "/nix/store/c8cz4q0y6jw9qvjwk45nf6pvwysj7yn6-source"; type = "path"; } nix-repl> :p builtins.getContext nixosConfigurations.micro.options.nix.registry.value.nixpkgs.to.path { "/nix/store/c8cz4q0y6jw9qvjwk45nf6pvwysj7yn6-source" = { path = true; }; }

ah! that's nice and simple. the clearest way to say that might be "by referencing it in the store"?

IreneKnapp · 2023-10-02T22:35:01Z

nixos/modules/misc/nixpkgs.nix

+          version of nixpkgs that the NixOS system has been built with.
+
+          This also sets a default to disable the global (online) flake registry due to
+          <https://github.com/NixOS/nix/issues/9087>.


ah good call

Bad call: unnecessary breaking change. Why not fix the bug instead?

I don't want to try getting anything into Nix after a one line change to fix nix-shell in shebangs by a former coworker continues to sit unreviewed for going on two years now, in spite of repeated prodding for reviews including by back channels.

I understand the Nix team has been improving things, and I think more prompt reviews generally happen now but I'm still not interested.

I can remove the workaround because the bug isn't that bad.

@lf- i have an interest in the shebangs features, but I am not finding the referenced shebang PR. (my own effort in bringing shebangs to nix-command took more than two years with 5189, so I understand the patience can get worn out)

i had it written down somewhere; it's NixOS/nix#5088

lf- · 2023-10-03T00:51:51Z

I have done a final pass through the docs. I am not sure if they are too verbose, but even if they are, usually we have the opposite problem and I have rarely thought that of module docs.

IreneKnapp · 2023-10-03T03:38:43Z

the latest revision of the docs looks great. thanks for indulging me! I know I tend to prefer more verbose docs than others do, but I feel that a lot of flake semantics are under-documented and I'd much rather not have to guess at these details.

Artturin · 2023-10-03T07:12:40Z

The diff looks very weird, are there formatting changes or something?

Related PR #197424

lf- · 2023-10-03T07:17:33Z

The diff looks very weird, are there formatting changes or something?

Related PR #197424

Yes. I moved the whole existing config= contents right one indent, put it in a mkMerge, and added my own things. When I rebased I did these changes manually because git had made such an absolute dog's breakfast of the diff and I couldn't make any sense of it.

I don't know what made git fail so hard on this PR's diff, but I am sorry it is annoying to review.

Artturin · 2023-10-03T07:28:52Z

Hiding the whitespace changes makes it better

rightmost icon

nixos/modules/misc/nixpkgs.nix

tomberek

blocking: /etc/nix/inputs/nixpkgs is awkward, please examine some other possibilities mentioned above
thought: having nixpkgs.flake.setNixPath interacting with nix.nixPath (same with nixpkgs.flake.setFlakeRegistry / nix.registry) might cause confusion. There are then two ways to do the same thing, which takes precedence, etc. Here i don't yet know of solid suggestion; there is merit to having it in both option sets.

nixos/modules/misc/nixpkgs.nix

…nfigs Currently there are a bunch of really wacky hacks required to get nixpkgs path correctly set up under flake configs such that `nix run nixpkgs#hello` and `nix run -f '<nixpkgs>' hello` hit the nixpkgs that the system was built with. In particular you have to use specialArgs or an anonymous module, and everyone has to include this hack in their own configs. We can do this for users automatically. I have tested these manually with a basic config; I don't know if it is even possible to write a nixos test for it since you can't really get a string-with-context to yourself unless you are in a flake context.

ethorsoe · 2024-02-13T11:44:28Z

While I like this change a lot, I understood that this breaks (at least works unexpectedly for) initializing new flake.lock files using nixpkgs name on such a system, am I wrong?

mjm · 2024-02-13T15:48:06Z

I didn't test this PR specifically, but I do have my config set up to point nixpkgs in the registry at my flake's nixpkgs input. And this does cause a surprising issue where if you create new flakes (maybe for a specific project), then if they use the nixpkgs registry alias, the lockfile will reference the store path rather than the github URL for nixpkgs.

This happens, for instance, with the default flake template:

$ cd $(mktemp -d)
$ nix flake init
$ nix build
warning: creating lock file '/tmp/tmp.cqX16SmdSg/flake.lock'
$ cat flake.lock
{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1707743206,
        "narHash": "sha256-AehgH64b28yKobC/DAWYZWkJBxL/vP83vkY+ag2Hhy4=",
        "path": "/nix/store/qn2bp2ipwdfsv5a4fp8sdk6kmibxhvsw-source",
        "rev": "2d627a2a704708673e56346fcb13d25344b8eaf3",
        "type": "path"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

I'm not sure if nix flake update will then use the registry alias or the path in the lockfile. Either of those seem undesirable in different ways. But if it uses the path in the lockfile, then overriding the registry alias by default for flake-based systems will probably cause surprising behavior for folks that create new flakes that don't specify a nixpkgs input explicitly.

lf- · 2024-02-13T18:33:05Z

Good grief, our default flake template is really really broken then. We should fix that.

lf- · 2024-02-13T19:01:19Z

Thanks for pointing this out; I wish to throw my computer into the sea. The remote flake registry should be removed imo or at least not used in lock file updates. NixOS/nix#7422

I've now found the previous attempt at this pr, which had a lot more respect for wrongly written flakes that use the registry for their inputs, a use case which should not be allowed imo. #197424

Fortunately the default template is here, so I'll send a pr to stop doing bad things in there at least: https://github.com/NixOS/templates/blob/master/trivial/flake.nix

But this needs to be fixed in nix flake update to issue a warning at least. Putting it at nixos in the system registry like the previous pr does is not an acceptable solution as it makes "get random packages" invocations different on NixOS; it's <nixos> all over again.

lf- · 2024-02-14T07:32:55Z

This is the PR from ages ago that should be merged to fix nix flake init NixOS/templates#67

ethorsoe · 2024-02-14T11:13:19Z

Makes me a bit uneasy that all references are to open unfinished discussions, but this is the experimental unstable new thing anyway so ymmv.

I have been recommending people to use aliases in their flakes on the main channel without any interjections so I would assume not everyone knows or agrees that they should not be used. I don't have strong opinion on the matter, but at least it would be good to standardize that opinion.

IreneKnapp · 2024-02-14T23:44:04Z

wow :( good research. what a distressing situation.

for sure, nix flake update needs to stop doing this ><

IreneKnapp · 2024-02-15T00:02:27Z

This is the PR from ages ago that should be merged to fix nix flake init NixOS/templates#67

wait, really? this looks Haskell-specific

lf- · 2024-02-15T00:23:50Z

This is the PR from ages ago that should be merged to fix nix flake init NixOS/templates#67

wait, really? this looks Haskell-specific

Clicked wrong, meant NixOS/templates#53

RaitoBezarius

This has been reviewed, scrutinized for a long time, this change makes a lot of sense and improves the statut-quo. Let's iterate from now on.

stale

No longer needed via NixOS/nixpkgs#254405

totoroot · 2024-02-29T08:12:25Z

Just for reference, in case someone is looking for the change that caused their flake-based NixOS systems to no longer rebuild.

Since I previously had nix.registry.nixpkgs.flake set in my flake config, I got the following error when trying to rebuild my system:

       error: The option `nix.registry.nixpkgs.to.path' has conflicting definition values:
       - In `/nix/store/1gpgihsi21bni03hy6i5ccw9lfrhnbqi-source/nixos/modules/config/nix-flakes.nix': "/nix/store/84q33bh989jr3hadc8jj2rsz6ax5cqyj-source"
       - In `/nix/store/1gpgihsi21bni03hy6i5ccw9lfrhnbqi-source/nixos/modules/misc/nixpkgs-flake.nix': "/nix/store/1gpgihsi21bni03hy6i5ccw9lfrhnbqi-source"
       Use `lib.mkForce value` or `lib.mkDefault value` to change the priority on any of these definitions.

The suggested use of lib.mkForce or lib.mkDefault is not advisable.
Instead just remove the manual NIX_PATH override in your config, as this is now handled automatically.

If you do not wish to do so, you can also disable the newly added option, as described in the release notes (look for NIX_PATH).

To disable this, set nixpkgs.flake.setNixPath and nixpkgs.flake.setFlakeRegistry to false.

IMO the documentation of this change in the release notes is good, but users might still look for this error in the issues and PRs of the nixpkgs repository, so I thought I'd mention it.

No longer needed since NixOS/nixpkgs#254405 got merged See NixOS/nixpkgs#254405 (comment) for more information

NixOS/nixpkgs#254405

Since <NixOS/nixpkgs#254405> was merged, `nix.nixPath` is set when you use `lib.nixosSystem`

nixos-discourse · 2024-04-21T18:35:48Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-a-system-nixpkgs-version-in-a-flake/43784/3

- Don't auto-optimise-store by default - We should avoid arbitrary "sane defaults" in this template, as nixpkgs already has pretty sane defaults nowadays - Simplify registry+nixPath - Instead of using a file as indirection (to avoid staleness), indirect it through the registry using `flake:nixpkgs` - Add a comment about NixOS/nixpkgs#254405, which all this by default for nixpkgs - Disable channels and flake registry by default

github-actions bot added 6.topic: nixos 8.has: documentation 8.has: changelog 8.has: module (update) labels Sep 10, 2023

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Sep 10, 2023

lf- force-pushed the jade/nix-path-flakes branch from 2078be3 to 3120b3e Compare October 1, 2023 02:36

ofborg bot added the 2.status: merge conflict label Oct 1, 2023

lf- force-pushed the jade/nix-path-flakes branch from 3120b3e to 8651130 Compare October 2, 2023 08:23

IreneKnapp reviewed Oct 2, 2023

View reviewed changes

ofborg bot removed the 2.status: merge conflict label Oct 2, 2023

lf- force-pushed the jade/nix-path-flakes branch 3 times, most recently from e0c648f to bc3b186 Compare October 2, 2023 21:42

lf- requested a review from IreneKnapp October 2, 2023 21:43

lf- changed the title ~~nixos/flake: options to set up registry and nixPath for flake configs~~ nixos/flake: setup NIX_PATH and system flake registry automatically Oct 2, 2023

IreneKnapp reviewed Oct 2, 2023

View reviewed changes

lf- force-pushed the jade/nix-path-flakes branch from bc3b186 to fd80a9f Compare October 3, 2023 00:50

Artturin requested a review from roberth October 3, 2023 07:11

Artturin reviewed Oct 3, 2023

View reviewed changes

nixos/modules/misc/nixpkgs.nix Outdated Show resolved Hide resolved

lf- force-pushed the jade/nix-path-flakes branch from fd80a9f to 0c919c2 Compare October 18, 2023 21:55

tomberek previously requested changes Oct 21, 2023

View reviewed changes

nixos/modules/misc/nixpkgs.nix Outdated Show resolved Hide resolved

lf- force-pushed the jade/nix-path-flakes branch from 0c919c2 to ec806b7 Compare November 11, 2023 03:50

github-actions bot added 8.has: documentation 8.has: changelog labels Feb 4, 2024

lf- force-pushed the jade/nix-path-flakes branch from 5ec755d to e456032 Compare February 4, 2024 04:44

ofborg bot added 10.rebuild-darwin: 0 and removed 10.rebuild-darwin: 1-10 labels Feb 4, 2024

lf- requested a review from infinisil February 13, 2024 03:57

RaitoBezarius approved these changes Feb 25, 2024

View reviewed changes

RaitoBezarius merged commit 5337ff6 into NixOS:master Feb 25, 2024
21 checks passed

Kidsan added a commit to Kidsan/nixos-config that referenced this pull request Feb 28, 2024

flake: remove manual nixpath override

22f6147

No longer needed via NixOS/nixpkgs#254405

totoroot added a commit to totoroot/dotfiles that referenced this pull request Feb 29, 2024

Remove manual registry entry for nixpkgs in flake

4b01299

No longer needed since NixOS/nixpkgs#254405 got merged See NixOS/nixpkgs#254405 (comment) for more information

lf- mentioned this pull request Feb 29, 2024

Building an ISO from a flake doesn't ensure that experimental-feature flakes is enabled, so <nixpkgs> does not evaluate #292465

Open

GaetanLepage added a commit to GaetanLepage/nix-config that referenced this pull request Mar 6, 2024

flake.nix: remove now useless nix.nixPath setting

39c272e

NixOS/nixpkgs#254405

name-snrl added a commit to name-snrl/nixos-configuration that referenced this pull request Apr 6, 2024

nix: remove nixPath option

cfa98cd

Since <NixOS/nixpkgs#254405> was merged, `nix.nixPath` is set when you use `lib.nixosSystem`

ryan4yin mentioned this pull request Apr 20, 2024

fix: option `nix.registry.nixpkgs.to.path' has conflicting definition ryan4yin/nix-config#123

Merged

zendo mentioned this pull request May 7, 2024

nix-channel: do not set empty nix-path when disabling channels #273170

Open

13 tasks

nixos/flake: set up NIX_PATH and system flake registry automatically #254405

nixos/flake: set up NIX_PATH and system flake registry automatically #254405

Conversation

lf- commented Sep 10, 2023

Description of changes

Things done

IreneKnapp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lf- commented Oct 2, 2023

IreneKnapp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lf- commented Oct 3, 2023

IreneKnapp commented Oct 3, 2023

Artturin commented Oct 3, 2023

lf- commented Oct 3, 2023 • edited

Artturin commented Oct 3, 2023

tomberek left a comment

Choose a reason for hiding this comment

ethorsoe commented Feb 13, 2024

mjm commented Feb 13, 2024

lf- commented Feb 13, 2024

lf- commented Feb 13, 2024

lf- commented Feb 14, 2024

ethorsoe commented Feb 14, 2024

IreneKnapp commented Feb 14, 2024

IreneKnapp commented Feb 15, 2024

lf- commented Feb 15, 2024

RaitoBezarius left a comment

Choose a reason for hiding this comment

totoroot commented Feb 29, 2024

nixos-discourse commented Apr 21, 2024

lf- commented Oct 3, 2023 •

edited