Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0049] Flakes #49

Closed
wants to merge 42 commits into from
Closed

[RFC 0049] Flakes #49

wants to merge 42 commits into from

Conversation

edolstra
Copy link
Member

Abstract: This RFC proposes a mechanism to package Nix expressions into composable entities called "flakes". Flakes allow hermetic, reproducible evaluation of multi-repository Nix projects; impose a discoverable, standard structure on Nix projects; and replace previous mechanisms such as Nix channels and the Nix search path.

Rendered

@samueldr
Copy link
Member

samueldr commented Jul 15, 2019

How will "channel advances" be published with flakes?

Flakes are looked up in a registry that maps identifiers such as nixpkgs to actual locations such as github:edolstra/nixpkgs/release-19.03. You can use such locations ("flake references") directly in the nix command:

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

If the "tested update" publish workflow is a branch on the main (nixpkgs) repository, it will become possible for contributors to accidentally push an untested, maybe even invalid, branch to the repo. Not a great idea.


Additionally, an orthogonal point.

Flakes replace channels [...]

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

Copy link
Member

@FRidh FRidh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RFC is well written. I do think we should aim at getting answers to the unresolved questions. I think the proposed direction is good from a usability point of view, and to get a bit of order in our expressions.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved

Flakes are motivated by a number of serious shortcomings in Nix:

* While Nix pioneered reproducible builds, sadly, Nix expressions are
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention how pure evaluation mode fits in and if/why that is not sufficient.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
GitHub archives, we cannot directly check that the contents match the
commit hash.

Note that lock files are only used at top-level: the `flake.lock` files
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider the situation where one wants to offer a package set consisting of stable third-party applications, where each application is defined in a separate Flake and has a lock file to guarantee its reproducibility. In that case the lock files of the applications are ignored, no longer guaranteeing the stability of the applications.

The idea of not considering the lock files is good in case of libraries, but in case of applications not always.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good point. We could simply always use the lock file of a dependency if it exist (so good practice would be not to commit lock files for library flakes). It gets tricky for flakes that provide both libraries and applications (like Nixpkgs)...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like something to consider carefully or it could become a big pain.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the difference between packages and apps, but what about having whether a lock file is considered be based on which set the reference is in?

Copy link
Member

@FRidh FRidh Jul 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think also that when a lock file is present, it should be used. It basically says "we provide a curated set, don't break it". It's then up to the community to ensure they don't use it incorrectly.

Now, in practice I think that is not something we cannot always rely on and so we want to be able to override/patch. Also, because upstreams tend to pin things too much; just look at the amount of patching we need with Python applications. We are given the attributeinputs, which is already imported, and won't allow us to override or patch the Flake when needed which, if correct, is a limitation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’re discussing things here that many other systems have stumbled over in the past.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lockfiles shouldn't be committed to repositories under any circumstance, unless it is a distribution repository. This is no different to vendor bundling, which is rightly opposed. It's for a distributor to decide how to combine dependencies.

I agree that lockfiles in dependencies should be ignored. If this principle were abandoned, it should be strictly opt-in.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could someone explain why lock files in dependencies should be ignored? Doesn't it break the reproducibility of evaluation if the inputs of the dependency got updated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without care, there is a tension between flakes offering composability on the one hand and reproducibility on the other. Ignoring dependent lockfiles is one way of reducing that tension: The flake.nix should specify the interface for the flake (eventually with version bounds or similar?) and the flake.lock should specify the current composition for people working on that project directly.

For example, I'd like us to eventually have a stdenv flake that abstracts out the bootstrapping logic and definition of mkDerivation from nixpkgs proper. When I'm developing on stdenv itself, I want some fixed definitions of things like gcc etc. so I have something to work with, but the entire point is that I ultimately want the caller to specify these! So when nixpkgs imports the stdenv flake, it should provide the pieces that make it all work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it might make sense, for a first draft, to say “lock files in dependencies are a hard error for now” and to just add that as a to-be-resolved-later question?

I'm not sure we need what @FRidh wants right now (and it's something that apparently only makes sense for applications, so not that well-explored for build systems at least), and this way is forwards-compatible with any future road we might end up choosing for handling lock files in dependencies.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
"id": "import-cargo",
"inputs": {},
"narHash": "sha256-mxwKMDFOrhjrBQhIWwwm8mmEugyx/oVlvBH1CKxchlw=",
"uri": "github:edolstra/import-cargo/c33e13881386931038d46a7aca4c9561144d582e"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While understandable for Nix, these uri's are still a bit odd for other potential consumers. Is there a reason for not putting a concrete https url to a tarball here? I suppose to be able to still use git after all. In any case, the motivation seems to be missing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because the github: scheme has some particular caching semantics, in particular using the fact that the ETag returned by GitHub is the Git revision. We can't assume that for https tarballs in general.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should come up with a better name for this scheme, then, and make it so the full URL is still included. I don’t think Nix should be coupled to GitHub (and as proposed this wouldn’t even work with GitHub enterprise). If we used attribute sets instead of URLs, this could be an attribute in the set.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why wouldn't it work with GitHub enterprise?

There is definitely something to be said for attrsets instead of URLs, but it means that we would have different syntaxes for flake references on the command line and in .inputs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, there’s no way to specify the domain GitHub is running on.

The command line is a good point. Could just be multiple arguments, maybe?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy. The advantages of this would be:

  • Separation of what is retrieved from how it is retrieved; github: and https://github.com/... are the same what, just being retrieved via a different method. But an URL is supposed to specify the what, not the how.
  • This avoids having to change github: URLs if the retrieval methods GitHub supports change.
  • Conversely, suppose some other hosting site https://example.com/ becomes popular in the future. If you want to add a more efficient way of retrieving Git repositories, with this model you'd be asking them to change to example:... to do so, which would break compatibility with older versions of Nix which don't understand example:....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be up to the user whether they want to use https or something else? Similar to how you get the option to clone using https or ssh in the github interface.

While a URI does indeed merely identify a resource a URL definitely most commonly specifies a protocol to be used for the retrieval. So I'd rather see a scheme where git://github.com/... is valid than having https urls being subverted non-obviously.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy.

Special-casing certain HTTP URLs seems much uglier than having a separate URI scheme. You would expect all resources in a particular URI scheme to be interpreted uniformly.

Also, from a UX perspective, having to figure out / type https://github.com/edolstra/patchelf/archive/master.tar.gz (and hoping that that's the special URL recognized by Nix) is worse than github:edolstra/patchelf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Special casing would be a very bad idea, because GitHub URLs don’t support all the features of git URLs. Like many other things in this rfc, I think we should be asking ourselves “is this feature absolutely critical for an initial release?” The answer to me in this case is quite clearly no. So I propose we drop GitHub URLs for now, in favour of git URLs, pending further, more focussed discussion.

zimbatm and others added 2 commits July 16, 2019 09:24
Co-Authored-By: Frederik Rietdijk <freddyrietdijk@fridh.nl>
Co-Authored-By: Frederik Rietdijk <freddyrietdijk@fridh.nl>
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
@edolstra
Copy link
Member Author

How will "channel advances" be published with flakes?

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

Ideally, we would merge nixpkgs-channels into nixpkgs. The only reason why we have a separate channels repo is to prevent people from pushing channels, but nowadays GitHub allows per-branch access control. So we can restrict these branches to a team that only includes the channel update script.

So nixpkgs/release-19.03 would refer to the untested 19.03 branch, while nixpkgs/nixos-19.03 would be the tested one.

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

Yes, exactly. The idea is that your NixOS system is itself a flake, so its flake.lock pins the exact version of Nixpkgs. FWIW a NixOS configuration flake looks like this:

{
  name = "eelco-configurations";

  edition = 201906;

  description = "My NixOS configurations";

  inputs =
    [ "nixpkgs/nixos-19.03"
      "dwarffs"
    ];

  outputs = inputs: rec {

    nixosConfigurations.my-server = inputs.nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ./configuration.nix
          inputs.dwarffs.nixosModules.dwarffs
        ];
    };

  };
}

which would be built by nixos-rebuild using

# nix build --profile /nix/var/nix/profiles/system ~/Misc/eelco-configurations:nixosConfigurations.my-server.config.system.build.toplevel

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
@lheckemann

This comment has been minimized.

@aanderse
Copy link
Member

@edolstra Am I correct in assuming that this will fix the issue where every machine in a nixops network is locked to the same channel?

Copy link
Member

@Profpatsch Profpatsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I’m missing most from the current version is a prior work section with how other package managers solve this problem and how it has influenced the design of flakes. Especially the design of flakes’s lock files.

There’s been much practical research in that area over the last few years.

packages.dwarffs =
with inputs.nixpkgs.packages;
with inputs.nixpkgs.builders;
with inputs.nixpkgs.lib;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s probably cleaner to a reader not in the know of nixpkgs if those are aliased instead of pulled in with with.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, what do you mean with aliased?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just something like:

let
  pkgs = inputs.nixpkgs.packages;
  inherit (inputs.nixpkgs) builders lib;

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
Copy link
Member

@alyssais alyssais left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m extremely wary of this. I don’t think that Nix should become a package registry, and I am extremely concerned by the lockfile issues others have raised. In particular, what should happen when dependencies pin to different versions of Nixpkgs, and the inability to patch flakes due to them not being exposed as derivations. I think it’s imperative that this RFC should not be merged until we have satisfactory solutions to these issues.

On the other hand, some things in here excite me a great deal. I love the syntax this will enable for the nix cli, for both pinned and unpinned versions. Channels and default.nix both have their own rather severe shortcomings, and I’m looking forward to seeing the back of them. This strikes me as an extremely complicated solution to that problem, but perhaps there is no simple one…


There are multiple registries:

* The global registry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a global registry at all? Could we not just, eg, provide hard-coded defaults for nix and nixpkgs, and let users do the rest? This would avoid becoming a package repository to any extent, which I think would be a very good thing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alyssais what would the defaults be?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe

  • nix = https://github.com/NixOS/nix stable
  • nixpkgs = https://github.com/NixOS/nixpkgs unstable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need a registry, but it seems important for UX. In fact it's more important here than in tools like Cargo, since the registry is used in a command line interface. We don't want to require people to type nix run github:NixOS/patchelf instead of nix run patchelf. Special-casing nixpkgs seems ugly, and it makes every other flake second-class compared to nixpkgs (which is bad since we should move away from the monorepo approach since it doesn't scale).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which is bad since we should move away from the monorepo approach since it doesn't scale

I disagree here, which is probably also the reason I'm a bit hesitant with the flakes approach in general. For me the monorepo approach is one of the major selling points of nixpkgs. Probably the selling point. It lowers the barrier to contribution a lot. It makes it very easy to find any information I want with a single grep. I can bisect any problem. I can test if my update breaks any known reverse dependencies. There is a central point of trust, even if that trust is very diffused at the moment.

I imagine a bunch of single-package repos loosely tied together with lockfiles will actually scale less. Why would I want patchelf to be in its own flake vs. in nixpkgs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

crates.io works very well so I don't think a registry needs to be a disaster. In any case,you don't have to use the registry. Of course then you have to type things like nix build github:NixOS/nixpkgs:... instead of nixpkgs:... all the time.

You could have the same ergonomics with a local registry only, with the nice commands for adding to it proposed in this rfc, though, with none of the need for us to maintain any sort of global registry.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately however the monorepo approach fundamentally doesn't scale: it shouldn't be necessary for everybody who just wants to share a NixOS module to have to add it to the nixpkgs repo.

I don’t want to argue this further in this thread, because I think it’s off-topic for the RFC, but I want to make clear that I still don’t agree and any such move would need to be discussed extensively at a later date before it were to happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

crates.io works very well so I don't think a registry needs to be a disaster. In any case,you don't have to use the registry. Of course then you have to type things like nix build github:NixOS/nixpkgs:... instead of nixpkgs:... all the time.

You mentioned crates.io a lot in your NixCon talk and it's obvious that this RFC took a lot of inspiration of what Rust is doing. I'd like to chime in here that the public often doesn't see a lot of the effort and sweat that it takes on parts of people to run crates.io, and the architectural problems that it comes with. This doesn't just include the actual infrastructure and the team of people it takes to run it, but also a lot of policy decisions on things like DMCA requests, name squatting, etc that introduce a lot of work that general users of the service aren't aware of either. It's easy to say "look, it runs really well" and not see the weekly meetings on policy updates. I got the feeling that everybody involved in developing NixOS was already pretty booked out. I feel that adding yet another thing to maintain to the list of NixOS infrastructure things would be an extremely bad idea.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The crates.io ecosystem is fundamentally a different maintenance approach to NixOS. There are no "Rust maintainers". Sure, there's people who work on the compiler or who are active in various working groups and teams (:wave:) but there's no centralised development in the way that NixOS requires it. Building a bunch of libraries is different from building an operating system distribution and while a global registry of libraries works well for one, it wouldn't scale at all for the other because it's the same people involved in most of the pieces. It just adds overhead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: the mono repo

As others have said previously, the fact that all the moving parts in NixOS are in a single place, can easily be bisected and tested against one another is a huge selling point for collaboration and also lowers the barrier to entry. NixOS is the first Linux distribution I've actually started working on, despite wanting and very much trying to get involved in both Debian and Fedora. I think it would be a shame if this were to change.

There's a relevant discussion/story from the Rust/ rustc repo about this. The repo itself is huge, unwieldy, takes ages to download etc. So it was proposed to remove certain components from it that are only tangentially related to the compiler core, such as rustdoc. Ultimately, it was decided to keep the repo as it is, because rustdoc is a component that frequently breaks with new PRs and so, it's common practise that if a compiler contribution breaks docs, that person then also fixes docs.

I think you could compare the development of rustc with the development of NixOS/nixpkgs. They are both mono repos, both part of an ecosystem with moving pieces around it, but strongly integrated i each other and benefit greatly from having all their parts kept in the same place.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
@edolstra
Copy link
Member Author

Yes, sounds good.

@edolstra edolstra closed this Jul 16, 2020
@edolstra edolstra added status: withdrawn and removed status: FCP in Final Comment Period labels Jul 16, 2020
@Mic92
Copy link
Member

Mic92 commented Jul 25, 2020

Since this RFC is not extensive enough to be used as documentation and the nix repository also does not provide any except for the source code I started writing a wiki article: https://nixos.wiki/wiki/Flakes

@arianvp
Copy link
Member

arianvp commented Jul 25, 2020 via email

@bqv
Copy link

bqv commented Jul 25, 2020

This is still an experimental feature

@arianvp
Copy link
Member

arianvp commented Jul 25, 2020 via email

@Mic92
Copy link
Member

Mic92 commented Jul 25, 2020

Why not make those docs part of the nix manual?

On Sat, Jul 25, 2020, 11:16 Jörg Thalheim @.***> wrote: Since this RFC is not extensive enough to be used as documentation and the nix repository also does not provide any except for the source code I started writing a wiki article: nixos.wiki/wiki/Flakes — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <github.com/notifications/unsubscribe-auth/AAEZNI3KE6RMCM7GE3VBTGTR5KPHFANCNFSM4ID3SJCQ> .

docbook... and I am not sure if those PRs are accepted.

@Ekleog
Copy link
Member

Ekleog commented Jul 27, 2020

Having a link to the wiki (that makes it very explicit it is an experimental feature) as part of the manual might make sense, I'd personally be wary of having the docs for experimental features that are expected to change alongside the docs for regular features — taking example on Rust that has an unstable book in addition to the regular (stable) rust book would seem like a good idea to me.

@Mic92
Copy link
Member

Mic92 commented Jul 27, 2020

Having a link to the wiki (that makes it very explicit it is an experimental feature) as part of the manual might make sense, I'd personally be wary of having the docs for experimental features that are expected to change alongside the docs for regular features — taking example on Rust that has an unstable book in addition to the regular (stable) rust book would seem like a good idea to me.

The wiki has has now a complete reference on the nix command: https://nixos.wiki/wiki/Nix_command
This allows to link to new commands as well as extending examples. I also started fixing some examples that are currently broken in --help.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/low-hanging-fruit-and-high-impact-tasks/21095/1

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/6

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-hard-coded-fields-and-their-rationale/44225/2

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-is-not-dying-please-dont-spread-fear-actively/44310/91

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment