Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0049] Flakes #49

Open
wants to merge 23 commits into
base: master
from

Conversation

@edolstra
Copy link
Member

commented Jul 15, 2019

Abstract: This RFC proposes a mechanism to package Nix expressions into composable entities called "flakes". Flakes allow hermetic, reproducible evaluation of multi-repository Nix projects; impose a discoverable, standard structure on Nix projects; and replace previous mechanisms such as Nix channels and the Nix search path.

Rendered

edolstra and others added 7 commits Jul 9, 2019
Rename 'epoch' -> 'edition'
This follows the Rust naming.
@samueldr

This comment has been minimized.

Copy link
Member

commented Jul 15, 2019

How will "channel advances" be published with flakes?

Flakes are looked up in a registry that maps identifiers such as nixpkgs to actual locations such as github:edolstra/nixpkgs/release-19.03. You can use such locations ("flake references") directly in the nix command:

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

If the "tested update" publish workflow is a branch on the main (nixpkgs) repository, it will become possible for contributors to accidentally push an untested, maybe even invalid, branch to the repo. Not a great idea.


Additionally, an orthogonal point.

Flakes replace channels [...]

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

@FRidh
Copy link
Member

left a comment

The RFC is well written. I do think we should aim at getting answers to the unresolved questions. I think the proposed direction is good from a usability point of view, and to get a bit of order in our expressions.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved

Flakes are motivated by a number of serious shortcomings in Nix:

* While Nix pioneered reproducible builds, sadly, Nix expressions are

This comment has been minimized.

Copy link
@FRidh

FRidh Jul 16, 2019

Member

Mention how pure evaluation mode fits in and if/why that is not sufficient.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
GitHub archives, we cannot directly check that the contents match the
commit hash.

Note that lock files are only used at top-level: the `flake.lock` files

This comment has been minimized.

Copy link
@FRidh

FRidh Jul 16, 2019

Member

Consider the situation where one wants to offer a package set consisting of stable third-party applications, where each application is defined in a separate Flake and has a lock file to guarantee its reproducibility. In that case the lock files of the applications are ignored, no longer guaranteeing the stability of the applications.

The idea of not considering the lock files is good in case of libraries, but in case of applications not always.

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 16, 2019

Author Member

Yes, good point. We could simply always use the lock file of a dependency if it exist (so good practice would be not to commit lock files for library flakes). It gets tricky for flakes that provide both libraries and applications (like Nixpkgs)...

This comment has been minimized.

Copy link
@ryantm

ryantm Jul 16, 2019

Member

This seems like something to consider carefully or it could become a big pain.

This comment has been minimized.

Copy link
@ryantm

ryantm Jul 16, 2019

Member

I'm not sure I understand the difference between packages and apps, but what about having whether a lock file is considered be based on which set the reference is in?

This comment has been minimized.

Copy link
@FRidh

FRidh Jul 16, 2019

Member

I think also that when a lock file is present, it should be used. It basically says "we provide a curated set, don't break it". It's then up to the community to ensure they don't use it incorrectly.

Now, in practice I think that is not something we cannot always rely on and so we want to be able to override/patch. Also, because upstreams tend to pin things too much; just look at the amount of patching we need with Python applications. We are given the attributeinputs, which is already imported, and won't allow us to override or patch the Flake when needed which, if correct, is a limitation.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 16, 2019

Member

We’re discussing things here that many other systems have stumbled over in the past.

This comment has been minimized.

Copy link
@hlandau

hlandau Jul 20, 2019

Lockfiles shouldn't be committed to repositories under any circumstance, unless it is a distribution repository. This is no different to vendor bundling, which is rightly opposed. It's for a distributor to decide how to combine dependencies.

I agree that lockfiles in dependencies should be ignored. If this principle were abandoned, it should be strictly opt-in.

This comment has been minimized.

Copy link
@Ninlives

Ninlives Jul 26, 2019

Could someone explain why lock files in dependencies should be ignored? Doesn't it break the reproducibility of evaluation if the inputs of the dependency got updated?

This comment has been minimized.

Copy link
@shlevy

shlevy Jul 29, 2019

Member

Without care, there is a tension between flakes offering composability on the one hand and reproducibility on the other. Ignoring dependent lockfiles is one way of reducing that tension: The flake.nix should specify the interface for the flake (eventually with version bounds or similar?) and the flake.lock should specify the current composition for people working on that project directly.

For example, I'd like us to eventually have a stdenv flake that abstracts out the bootstrapping logic and definition of mkDerivation from nixpkgs proper. When I'm developing on stdenv itself, I want some fixed definitions of things like gcc etc. so I have something to work with, but the entire point is that I ultimately want the caller to specify these! So when nixpkgs imports the stdenv flake, it should provide the pieces that make it all work.

This comment has been minimized.

Copy link
@Ekleog

Ekleog Jul 30, 2019

Member

Maybe it might make sense, for a first draft, to say “lock files in dependencies are a hard error for now” and to just add that as a to-be-resolved-later question?

I'm not sure we need what @FRidh wants right now (and it's something that apparently only makes sense for applications, so not that well-explored for build systems at least), and this way is forwards-compatible with any future road we might end up choosing for handling lock files in dependencies.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
# Unresolved questions
[unresolved]: #unresolved-questions

* How to handle the system type? Currently `x86_64-linux` is

This comment has been minimized.

Copy link
@FRidh

FRidh Jul 16, 2019

Member

I suppose the following two questions need to be answered:

  1. how to deal with cross-compilation? Do we think the approach we have now in Nixpkgs is the solution, i.e., a localSystem and crossSystem.
  2. detecting the current system type is impure. Is it an acceptable exception that Nix by default sets localSystem.

This comment has been minimized.

Copy link
@zimbatm

zimbatm Jul 17, 2019

Member

Injecting the localSystem is probably a necessity to make this practical. Maybe it can be a default argument instead of being attached to a builtin. Then if the user wants to cross-build or switch the localSystem they can always pass -A localSystem something-else or -A crossSystem ...

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 18, 2019

Author Member

One possibility is to extend the flake outputs function with a set of arguments, e.g.

outputs = { inputs, args }: 
  with import inputs.nixpkgs { system = args.system; };
  { ...
  };

The args.systems argument would be provided automatically by the user interface. It would alos be part of the cache key so evaluations of the same flake for different systems wouldn't collide with each other.

The alternative is to lazily return packages for all supported platform, e.g.

outputs = inputs: {
  packages.hello.x86_64-linux = ...;
  packages.hello.x86_64-darwin = ...;
};

(This is similar to what Nixpkg's Hydra jobset does.) Then the UI could automatically select a package for the local system (e.g. nix install flake:hello would select packages.hello.<my-system>).

This comment has been minimized.

Copy link
@mstone

mstone Aug 11, 2019

One piece of positive feedback: I can use uses for both approaches but happily the with import inputs.nixpkgs { system = ... } approach works today on Darwin just by hardcoding system = "x86_64-darwin";, at least for the simple flake I'm testing with!

"id": "import-cargo",
"inputs": {},
"narHash": "sha256-mxwKMDFOrhjrBQhIWwwm8mmEugyx/oVlvBH1CKxchlw=",
"uri": "github:edolstra/import-cargo/c33e13881386931038d46a7aca4c9561144d582e"

This comment has been minimized.

Copy link
@FRidh

FRidh Jul 16, 2019

Member

While understandable for Nix, these uri's are still a bit odd for other potential consumers. Is there a reason for not putting a concrete https url to a tarball here? I suppose to be able to still use git after all. In any case, the motivation seems to be missing.

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 16, 2019

Author Member

It's because the github: scheme has some particular caching semantics, in particular using the fact that the ETag returned by GitHub is the Git revision. We can't assume that for https tarballs in general.

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 16, 2019

Member

We should come up with a better name for this scheme, then, and make it so the full URL is still included. I don’t think Nix should be coupled to GitHub (and as proposed this wouldn’t even work with GitHub enterprise). If we used attribute sets instead of URLs, this could be an attribute in the set.

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 18, 2019

Author Member

Hm, why wouldn't it work with GitHub enterprise?

There is definitely something to be said for attrsets instead of URLs, but it means that we would have different syntaxes for flake references on the command line and in .inputs.

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 19, 2019

Member

Well, there’s no way to specify the domain GitHub is running on.

The command line is a good point. Could just be multiple arguments, maybe?

This comment has been minimized.

Copy link
@hlandau

hlandau Jul 20, 2019

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy. The advantages of this would be:

  • Separation of what is retrieved from how it is retrieved; github: and https://github.com/... are the same what, just being retrieved via a different method. But an URL is supposed to specify the what, not the how.
  • This avoids having to change github: URLs if the retrieval methods GitHub supports change.
  • Conversely, suppose some other hosting site https://example.com/ becomes popular in the future. If you want to add a more efficient way of retrieving Git repositories, with this model you'd be asking them to change to example:... to do so, which would break compatibility with older versions of Nix which don't understand example:....

This comment has been minimized.

Copy link
@toonn

toonn Jul 28, 2019

Shouldn't it be up to the user whether they want to use https or something else? Similar to how you get the option to clone using https or ssh in the github interface.

While a URI does indeed merely identify a resource a URL definitely most commonly specifies a protocol to be used for the retrieval. So I'd rather see a scheme where git://github.com/... is valid than having https urls being subverted non-obviously.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy.

Special-casing certain HTTP URLs seems much uglier than having a separate URI scheme. You would expect all resources in a particular URI scheme to be interpreted uniformly.

Also, from a UX perspective, having to figure out / type https://github.com/edolstra/patchelf/archive/master.tar.gz (and hoping that that's the special URL recognized by Nix) is worse than github:edolstra/patchelf.

zimbatm and others added 2 commits Jul 16, 2019
Update rfcs/0049-flakes.md
Co-Authored-By: Frederik Rietdijk <freddyrietdijk@fridh.nl>
Update rfcs/0049-flakes.md
Co-Authored-By: Frederik Rietdijk <freddyrietdijk@fridh.nl>
* `lastModified`: The commit time of the revision `rev`, in the
format `%Y%m%d%H%M%S` (e.g. `20181231100934`). Unlike `revCount`,
this is available for both Git and GitHub repositories, so it's
useful for generating (hopefully) monotonically increasing version

This comment has been minimized.

Copy link
@lheckemann

lheckemann Jul 16, 2019

Member

AFAIU, the ways monotonicity can be violated are:

  • Force pushes
  • Desynchronised clocks on different committers' machines
  • Timezone differences.

Naïvely, I'd say that the first two cannot be handled and we'll simply have to rely on best practice, and for the third that the dates should always be in UTC — but I have the impression that date/time stuff is always more complicated than I would have thought, so corrections/improvements on that suggestion would be very welcome.

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 16, 2019

Author Member

Yes, hence the "hopefully", it's the best that can be done without access to the revCount. Forced pushes would also break the monotonicity of revCount BTW.

The timestamp is in UTC (via git log --format=%ct).

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
# nix build .:defaultPackage

For backwards compatibility, it's possible to use non-flake Nix
expressions using `-f`, e.g. `nix build -f foo.nix foo.bar`.

This comment has been minimized.

Copy link
@lheckemann

lheckemann Jul 16, 2019

Member

-1 on putting backwards compatible cruft in a tool that we still have the freedom to change! AFAIK, the only thing the current nix build can do which nix-build can't is display a fancy progress bar, so IMHO nix-build should be preferred for non-flake expressions.

This comment has been minimized.

Copy link
@tadfisher

tadfisher Jul 20, 2019

Is there a consensus on deprecating or continuing the support of the "nix-*" commands? I prefer using the nix command because its arguments are consistent and the UX matches other tools such as git and cargo.

If we take your suggestion, should the -f argument be removed from other nix commands, such as nix eval? I personally think doing so would reduce nix's usability as a general-purpose build tool rather than as a flake builder.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 25, 2019

Member

Is there a consensus on deprecating or continuing the support of the "nix-*" commands? I prefer using the nix command because its arguments are consistent and the UX matches other tools such as git and cargo.

The nix command is not even stable yet, so nobody should use them for scripting. We can think about starting deprecation once we give any stability guarantees.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

I don't think we should remove -f because it's unrealistic to think that everything will be converted to flakes right away, and there are non-legacy uses of -f (e.g. nix eval -f ...).

@edolstra

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

How will "channel advances" be published with flakes?

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

Ideally, we would merge nixpkgs-channels into nixpkgs. The only reason why we have a separate channels repo is to prevent people from pushing channels, but nowadays GitHub allows per-branch access control. So we can restrict these branches to a team that only includes the channel update script.

So nixpkgs/release-19.03 would refer to the untested 19.03 branch, while nixpkgs/nixos-19.03 would be the tested one.

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

Yes, exactly. The idea is that your NixOS system is itself a flake, so its flake.lock pins the exact version of Nixpkgs. FWIW a NixOS configuration flake looks like this:

{
  name = "eelco-configurations";

  edition = 201906;

  description = "My NixOS configurations";

  inputs =
    [ "nixpkgs/nixos-19.03"
      "dwarffs"
    ];

  outputs = inputs: rec {

    nixosConfigurations.my-server = inputs.nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ./configuration.nix
          inputs.dwarffs.nixosModules.dwarffs
        ];
    };

  };
}

which would be built by nixos-rebuild using

# nix build --profile /nix/var/nix/profiles/system ~/Misc/eelco-configurations:nixosConfigurations.my-server.config.system.build.toplevel
and the contents of its lockfile. Because flakes are evaluated in pure
mode, this uniquely identifies the evaluation result.

Currently caching is only done for top-level attributes (e.g. for

This comment has been minimized.

Copy link
@lheckemann

lheckemann Jul 16, 2019

Member

"top-level attributes" is confusing here — should this be "attributes specified in a command"? Because as far as I can tell, packages.firefox is not a top-level attribute.

rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
the specified repository and `ref`.)

* Fetched flakes are registered as garbage collector roots, so running
the garbage collector while on an airplane will not ruin your day.

This comment has been minimized.

Copy link
@lheckemann

lheckemann Jul 16, 2019

Member

Do we want to specify a mechanism for removing/expiring these roots?

@lheckemann

This comment has been minimized.

Copy link
Member

commented Jul 16, 2019

❤️ overall in particular:

  • Evaluation caching
  • running the garbage collector while on an airplane will not ruin your day.

@aanderse

This comment has been minimized.

Copy link

commented Jul 16, 2019

@edolstra Am I correct in assuming that this will fix the issue where every machine in a nixops network is locked to the same channel?

@Profpatsch
Copy link
Member

left a comment

What I’m missing most from the current version is a prior work section with how other package managers solve this problem and how it has influenced the design of flakes. Especially the design of flakes’s lock files.

There’s been much practical research in that area over the last few years.

packages.dwarffs =
with inputs.nixpkgs.packages;
with inputs.nixpkgs.builders;
with inputs.nixpkgs.lib;

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 16, 2019

Member

It’s probably cleaner to a reader not in the know of nixpkgs if those are aliased instead of pulled in with with.

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 18, 2019

Author Member

Hm, what do you mean with aliased?

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 19, 2019

Member

Just something like:

let
  pkgs = inputs.nixpkgs.packages;
  inherit (inputs.nixpkgs) builders lib;
rfcs/0049-flakes.md Show resolved Hide resolved

* `checks`: A non-nested set of derivations built by the `nix flake
check` command, or by Hydra if a flake does not have a `hydraJobs`
attribute.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 16, 2019

Member

What’s a check?

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 19, 2019

Author Member

An arbitrary derivation that must build successfully in order for nix flake check to succeed.

This comment has been minimized.

Copy link
@domenkozar

domenkozar Jul 26, 2019

Member

How does it differ from hydraJobs then?

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

hydraJobs is Hydra-specific. Usually you should use checks.

Also, hydraJobs can be nested, which means it's impossible to efficiently query what jobs are defined by it. Hence checks cannot be nested.

rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Outdated Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved

where the hash `92a9...` is a fingerprint over the flake store path
and the contents of its lockfile. Because flakes are evaluated in pure
mode, this uniquely identifies the evaluation result.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 16, 2019

Member

For automation it’s often necessary to guarantee that no such resolving is done at some specific time, is there a way to populate the cache (and maybe even merge multiple existing caches)?

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

Hm, what do you mean with "resolving"? The existence or non-existence of the cache should not matter in general.

* The "edition" feature enables future Nix changes, including language
changes. For example, changing the parsing of multiline strings
(https://github.com/NixOS/nix/pull/2490) could be conditional on the
flake's edition.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 16, 2019

Member

This is a good practice. Rust does something similar with editions.

The key here is that the compiler (or evaluator in nix’s case) never loses the ability to work with older editions.

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 16, 2019

Member

How would parsing these work? With Cargo, it’s not like the TOML format is changing between editions. Does the evaluator have to try evaluating with each version of the syntax, starting with the newest, until it finds one that works?

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 19, 2019

Member

Good question, but do we really want to change the nix syntax in a backwards-incompatible way at this point? I think probably no.

If we don’t intend to change the syntax it’s just about the semantics of the fields, which is what editions are about I presume.

This comment has been minimized.

Copy link
@timokau

timokau Jul 19, 2019

Member

Good question, but do we really want to change the nix syntax in a backwards-incompatible way at this point? I think probably no.

There's https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 19, 2019

Author Member

@alyssais That's a good point, and is an argument in favor of using JSON/TOML for flake.nix.

This comment has been minimized.

Copy link
@domenkozar

domenkozar Aug 7, 2019

Member

Shouldn't edition just be evaluated lazily? That seems like a sane requirement.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

It can't be evaluated lazily. It has to be evaluated eagerly because the interpretation of all other attributes (potentially) depends on it.

@alyssais
Copy link
Member

left a comment

I’m extremely wary of this. I don’t think that Nix should become a package registry, and I am extremely concerned by the lockfile issues others have raised. In particular, what should happen when dependencies pin to different versions of Nixpkgs, and the inability to patch flakes due to them not being exposed as derivations. I think it’s imperative that this RFC should not be merged until we have satisfactory solutions to these issues.

On the other hand, some things in here excite me a great deal. I love the syntax this will enable for the nix cli, for both pinned and unpinned versions. Channels and default.nix both have their own rather severe shortcomings, and I’m looking forward to seeing the back of them. This strikes me as an extremely complicated solution to that problem, but perhaps there is no simple one…


There are multiple registries:

* The global registry

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 16, 2019

Member

Why do we need a global registry at all? Could we not just, eg, provide hard-coded defaults for nix and nixpkgs, and let users do the rest? This would avoid becoming a package repository to any extent, which I think would be a very good thing.

This comment has been minimized.

Copy link
@asymmetric

asymmetric Jul 17, 2019

@alyssais what would the defaults be?

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 18, 2019

Member

Maybe

  • nix = https://github.com/NixOS/nix stable
  • nixpkgs = https://github.com/NixOS/nixpkgs unstable

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 19, 2019

Author Member

We don't need a registry, but it seems important for UX. In fact it's more important here than in tools like Cargo, since the registry is used in a command line interface. We don't want to require people to type nix run github:NixOS/patchelf instead of nix run patchelf. Special-casing nixpkgs seems ugly, and it makes every other flake second-class compared to nixpkgs (which is bad since we should move away from the monorepo approach since it doesn't scale).

This comment has been minimized.

Copy link
@timokau

timokau Jul 20, 2019

Member

which is bad since we should move away from the monorepo approach since it doesn't scale

I disagree here, which is probably also the reason I'm a bit hesitant with the flakes approach in general. For me the monorepo approach is one of the major selling points of nixpkgs. Probably the selling point. It lowers the barrier to contribution a lot. It makes it very easy to find any information I want with a single grep. I can bisect any problem. I can test if my update breaks any known reverse dependencies. There is a central point of trust, even if that trust is very diffused at the moment.

I imagine a bunch of single-package repos loosely tied together with lockfiles will actually scale less. Why would I want patchelf to be in its own flake vs. in nixpkgs?

This comment has been minimized.

Copy link
@alyssais

alyssais Sep 3, 2019

Member

Regarding registries: the UX value of global registries is obvious, IMHO, from tools like Cargo or npm.

npm’s global registry has been a disaster. There’s been account compromises, typosquatting, name squatting in general, and useful names lie dormant because somebody grabbed them years ago.

I am strongly opposed to a global registry, especially to start with. If somebody wants to start a global registry outside of Nix to show its value, fair enough. Maybe we can adopt it. But we don’t need it for flakes, so I don’t think we should do it now.

This comment has been minimized.

Copy link
@alyssais

alyssais Sep 3, 2019

Member

A search engine for flakes might be a nice idea, however. Again, though, it can be external to Nix.

This comment has been minimized.

Copy link
@timokau

timokau Sep 4, 2019

Member

The reason you'd want that is because the patchelf package in nixpkgs is several years behind the upstream package

In this case why don't we update the nixpkgs version instead?

This comment has been minimized.

Copy link
@edolstra

edolstra Sep 4, 2019

Author Member

crates.io works very well so I don't think a registry needs to be a disaster. In any case,you don't have to use the registry. Of course then you have to type things like nix build github:NixOS/nixpkgs:... instead of nixpkgs:... all the time.

@timokau Patchelf was just an example, but in general you can't expect Nixpkgs to contain the whole world.

This comment has been minimized.

Copy link
@PyroLagus

PyroLagus Sep 4, 2019

Name squatting is also an issue on crates.io, unfortunately. Now, it doesn't have to be a disaster, but it certainly needs moderation and, ideally manual, verification before flake submissions. If the official global registry is small, and users can just add third-party registries to their systems, it shouldn't be bad. Namespacing would also help, but I feel like the official global registry should be limited to official NixOS stuff anyways.

automatically fetches this registry periodically. The check interval
is determined by the `tarball-ttl` option.

* The local registry `~/.config/nix/registry.json`.

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 16, 2019

Member

Should be $XDG_CONFIG_HOME/nix/registry.json.

This comment has been minimized.

Copy link
@lheckemann

lheckemann Jul 17, 2019

Member

Or local registries, searching $XDG_CONFIG_DIRS for nix/registry.json. That allows different levels of the config, e.g. system-wide with user overrides.

This comment has been minimized.

Copy link
@Profpatsch

Profpatsch Jul 17, 2019

Member

Though it shouldn’t be so complicated that it’s easy to miss configuration. So keeping it simple is key for less surprises.

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 9, 2019

Member

Maybe a single directory with multiple registries and a debug command to list the local registries?

I also think that allowing only a single local-FS registry can create annoyance in too many ways… Neither $NIX_PATH not overlays are that restrictive.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

The implementation uses $XDG_CONFIG_HOME in fact, but I didn't want to mention that here for simplicity.

@lheckemann I guess you meant $XDG_CONFIG_DIRS? That would be pretty useful.

This comment has been minimized.

Copy link
@lheckemann

lheckemann Sep 6, 2019

Member

I did indeed :) (fixed). And it's good for consistency, since we have that behaviour for nix.conf already with NixOS/nix#2501.

rfcs/0049-flakes.md Show resolved Hide resolved
rfcs/0049-flakes.md Show resolved Hide resolved
any non-derivation attributes. This also means it cannot be a nested
set! (The rationale is that supporting nested sets requires Nix to
evaluate each attribute in the set, just to discover which packages
are provided.)

This comment has been minimized.

Copy link
@timokau

timokau Jul 16, 2019

Member

So things like pythonPackages won't work anymore?

This comment has been minimized.

Copy link
@michaelpj

michaelpj Jul 16, 2019

This is awkward for flakes that want to provide many outputs structured in some way, e.g. haskellPackages, docs, developmentTools etc. It sounds like currently those would have to be non-packages outputs, which reduces the usefulness of packages quite a lot.

Here's an alternative suggestion: allow users to provide an "index filter". Much like other filters, this would be applied at each step of looking for packages to decide whether to index the derivations inside that attribute set. The default would be "packages" (and not any child attribute sets!) but a user could specify a more liberal filter if they wanted (which would let you include attribute sets inside packages, or maybe even in other outputs).

This comment has been minimized.

Copy link
@alyssais

alyssais Jul 17, 2019

Member

supporting nested sets requires Nix to
evaluate each attribute in the set, just to discover which packages
are provided.

Why?

This comment has been minimized.

Copy link
@edolstra

edolstra Jul 19, 2019

Author Member

When you encounter an attribute like pythonPackages, the only way to know that it's an attrset containing more packages rather than a package is by evaluating it.

This comment has been minimized.

Copy link
@arianvp

arianvp Jul 20, 2019

However turning haskellPackages into a flake itself would mean we don't need nesting right?

This comment has been minimized.

Copy link
@timokau

timokau Jul 21, 2019

Member

There would likely be circular dependencies between haskellPackages and the rest of nixpkgs though.

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 9, 2019

Member

Given that flake UI already requires a complicated variable amount of colons, maybe have something like subflakes so that github:NixOS/nixpkgs:pythonPackages:pygtk could work?


* Nix projects lack discoverability and a standard structure. For
example, it's just convention that a repository has a `release.nix`
for Hydra jobs and a `default.nix` for packages.

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 9, 2019

Member

Isn't default.nix and shell.nix assumed by enough tools to be a bit more than mere convention?

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

No, default.nix has no well-known structure. I've seen it contain a single package at top-level, an attrset of packages, and an attrset of arbitrary stuff (like functions).


<path>(\?<params)?

where `<path>` must refer to (a subdirectory of) a Git repository.

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 9, 2019

Member

Can we please support plain directories? Then we don't need to wait until at least Mercurial is supported, not to mention negotiating about Fossil, Pijul, Monotone…

evaluate each attribute in the set, just to discover which packages
are provided.)

* `defaultPackage`: A derivation used as a default by most `nix`

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 9, 2019

Member

What happens if packages.defaultPackage also exists? Check complains?

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

No, but that would be a good idea to check.

This comment has been minimized.

Copy link
@7c6f434c

7c6f434c Aug 16, 2019

Member

Maybe it is a good idea to specify that it should not exist then?


## Flakes

A flake is a Git repository that contains a file named `flake.nix` in

This comment has been minimized.

Copy link
@mstone

mstone Aug 10, 2019

Defining flakes explicitly as git repos seems like a useful stepping-stone toward a delightful future but I sense that it may even now be too restrictive for what I think even early-majority flake adopters will want, let alone what the long tail of users will want.

As a result, would it be possible to sketch out at least a little bit, here or elsewhere, where the current design is driven to depend exclusively on git for short-term convenience, expediency, or concreteness vs more long-term reasons so that we can think together about how the needs or aspirations those reasons represent might be addressed in non-git scenarios e.g., involving different network, security, compatibility, usability, or scale considerations?

This comment has been minimized.

Copy link
@mstone

mstone Aug 10, 2019

To add a couple of high-level use cases motivating the question beyond more abstract design value judgments about configurability and coupling:

  • while acknowledging the importance of human-meaningful flake metadata for good UX, as a user experimenting with flakes for the first time, I would like to be able to define flakes in local directories on my filesystem that are not yet version-controlled or that are version-controlled with something other than git

  • as a user who occasionally interacts with monorepos, some of them large, some of them not based on git, or some of them based on git but using it in non-conventional ways, I would like flakes to be less tightly coupled not only with git but also with the idea that relevant units of code live precisely in "git repositories made available by a network service via URLs"

  • finally, as a user interested in using nix flakes in diverse environments including on hardware or operating systems where git may not be available, as well as with colleagues who are not yet familiar with git or where software is being managed by other means, I would like the flakes design to be a little bit more agnostic about how flakes and their supporting resources are organized, aggregated, and transported even if awesome support for git and GitHub remain top-priority concerns.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 16, 2019

Author Member

There is nothing about flakes that makes them inherently tied to Git or GitHub. It's just that those are the only two mechanisms implemented at the moment. Mercurial and tarball support would be straightforward to add.

@Mic92 Mic92 referenced this pull request Aug 15, 2019
@nbp

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

tl;dr: This approach is trying to go too far too fast, it would be good to keep it in mind but better to do it incrementally. We should first solve Nixpkgs modularity issues such that Nixpkgs can converge to a collection of external flake resources, coming from the root of repositories instead of building a new format with known design flaws.

Flakes format is similar to what got proposed in #3 , which is having a syntax where the meta data can be manipulated without the derivation, and the derivation is just another attribute which live in this attribute set that we call a package.

However, the current proposal is going further than #3 as it proposed to have these files stored at the root of external packages. While #3 suggested to use a similar formalism as a replacement for the way we write packages.

As highlighted in the drawback, this solution fails to support overlay.

Also, as highlighted by @FRidh (comment), this fails to handle cases where one package is not a standalone entity, but a part of a bigger set of compatible packages, such as Python, Emacs, Haskell, kernel modules, Xlibs … This is a problem which had already multiple solutions proposed to fix this issue within the Nixpkgs fix-point, such as NixOS/nixpkgs#44196.

Today, if we want to use Nix to build one package, we create the following files release.nix (for having multiple configuration of the same package), default.nix (to mimick how it should look like if integrated in Nixpkgs), shell.nix (to select one configuration of release.nix to work with), module.nix (for NixOS configuration) and overlay.nix (to extend/patch Nixpkgs).

flakes.nix attempt to do the same with a single file, as being the unique Nix file to be edited. Technically this is easily doable without introducing a new package schema:

rec {
  jobs = {};
  package = { .. }: {};
  overlay = self: super: {};
  modules = {};
  _type = "flakes";
}

One of the problem added with the current flakes.nix proposal is the lack of naming of the package when used as an input of another. Nixpkgs gives us a central location where the fix-point is used to resolved the inputs of a package. With the lack of global names, a package would only be known as some other package input, and doing so adds a lot of complexity to external patch-ability, such as overlays, override and overrideDerivation offer.

Thus I am in favor with this principle, but against the concept of using directly other packages as inputs.

However, I would be in favor of including other packages through overlays, where each package is registered under global names. Thus to use the package foo, the package bar would have to import foo, then refer to it through an attribute in the global namespace. The package foo will register multiple names, such as foo, foo_1 and foo_1_5. The package bar would register bar.inputs.foo as an alias of foo_1_5 through the use of the imported locked file.

Therefore, I would recommend to first adopt this new formalism for writing packages inside Nixpkgs, before attempting/advertising to use flakes as a standalone format to be included at the root of repositories. In the ideal world, Nixpkgs should just be an authoritative archive of each project's flake.

Integrating it in Nixpkgs first will help us solve the issues of Haskell, Python packages … The problem we have with these is that one might want to define a package for all versions of python above Python 3, and thus appear in the associated package sets. Thus we need to solve NixOS/nixpkgs#44196 in a modular fashion.

Once the remaining issues within Nixpkgs are fixed, then we could promote it to external packages, as the universal package manager. This could later be done, by generating flakes out of Cargo.toml and requirements.txt files for example.

edolstra added 3 commits Aug 16, 2019
Remove mention of evaluation caching
This is an implementation detail, not really relevant here.
@edolstra

This comment has been minimized.

Copy link
Member Author

commented Aug 16, 2019

@nbp

This approach is trying to go too far too fast, it would be good to keep it in mind but better to do it incrementally.

I think the current proposal is pretty minimal: it's at its core just a mechanism for composing repositories containing Nix expressions. Except for some standardised outputs (like packages), it doesn't impose a lot of semantics about those Nix expressions.

On a related note: (@Ekleog)

if we're going to do a package manager in nix, we should do it right, and that includes at least solving for version constraints.

Doing version constraint resolution would go far beyond the MVP nature of the current flake implementation. Also, Nix has never done version constraint resolution so it would feel kind of alien. However it's something that could be added in a future iteration.

As highlighted in the drawback, this solution fails to support overlay.

Actually overlays should work fine, you just don't get automatic composition of the nixpkgs flake with flakes that provide overlays. But you can compose them explicitly, e.g.

inputs = [ "nixpkgs" "some-overlay" ];
outputs = { inputs }:
  ...
  import inputs.nixpkgs { 
    system = "x86_64-linux";
    overlays = [ inputs.some-overlay.overlay ]; 
  }
  ...
@nbp

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

This approach is trying to go too far too fast, it would be good to keep it in mind but better to do it incrementally.

I think the current proposal is pretty minimal: it's at its core just a mechanism for composing repositories containing Nix expressions.

Except that we have not yet solved this problem on Nixpkgs yet. Not until we still have non-extensible package-sets for languages or sets of libraries.

My point is to first apply our wisdom to Nixpkgs to the point where we do not see any extensibility/composability issues in Nixpkgs.

Actually overlays should work fine, you just don't get automatic composition of the nixpkgs flake with flakes that provide overlays. But you can compose them explicitly, e.g.

inputs = [ "nixpkgs" "some-overlay" ];
outputs = { inputs }:
  ...
  import inputs.nixpkgs { 
    system = "x86_64-linux";
    overlays = [ inputs.some-overlay.overlay ]; 
  }
  ...

Yes, it works, but it would not scale if all packages within Nixpkgs were to use this pattern. This literally imply one instance of Nixpkgs attribute set per instantiated package.

@nixos-discourse

This comment has been minimized.

Copy link

commented Aug 18, 2019

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/scaling-the-python-package-set-in-nixpkgs/3749/1

@Ekleog

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

@edolstra

Doing version constraint resolution would go far beyond the MVP nature of the current flake implementation. Also, Nix has never done version constraint resolution so it would feel kind of alien. However it's something that could be added in a future iteration.

I think it'd be a good idea to figure out now how we would do it later, even if it's left to do later: I can't see a nice syntax for adding version bounds later, with the current RFC.

IOW, I believe we should make this RFC by having full version constraint resolution in mind, and then only implement the “accept-only-one-version-and-allow-multiple-versions-of-the-same-package” (thereafter “pin-nixpkgs-like”) version constraint at the beginning.

The hard part of this is that nix sometimes (but not always, eg. python packages or glibc) allows for multiple versions of a single package to be used concurrently, so the previous art on version constraint resolution (that I know of) could not be just taken, and we'd have to think about what kinds of version constraints make sense in order to properly define the syntax for pin-nixpkgs-like constraints.

@globin globin referenced this pull request Aug 22, 2019
@domenkozar domenkozar referenced this pull request Aug 29, 2019
edolstra added 3 commits Aug 16, 2019
@edolstra

This comment has been minimized.

Copy link
Member Author

commented Aug 30, 2019

Update: I changed the inputs attribute to be an attrset rather than a list. This allows you to depend on multiple versions of a flake, e.g. in the nixos-homepage flake:

inputs.nixpkgsStable.uri = "nixpkgs/release-19.03";
inputs.nixpkgsUnstable.uri = "nixpkgs/master";

Non-flake inputs can now be specified like this:

inputs.grcov = {
  uri = github:mozilla/grcov;
  flake = false;
};

The outputs function should now look like this:

outputs = { self, nixpkgsUnstable, nixpkgsStable, nix, hydra }: ...;

i.e. it takes the evaluated flakes listed in inputs.

Also, the name attribute is gone because it no longer served a purpose (it's now up to the caller to specify an identifier for each dependency via the inputs attribute).

inputs = [ "nixpkgs" ];
outputs = inputs: rec {
outputs = { self, nixpkgs }: rec {

This comment has been minimized.

Copy link
@toonn

toonn Aug 31, 2019

Should the set be recursive? Just read yesterday about how recursive overlays should be avoided and the proper way is to use the self fixpoint.

This comment has been minimized.

Copy link
@edolstra

edolstra Sep 3, 2019

Author Member

No, it doesn't have to be recursive, you can also use self to refer to other outputs.

This comment has been minimized.

Copy link
@toonn

toonn Sep 3, 2019

So the advice about overlays, specifically about a "recursive attrset", doesn't apply?

This comment has been minimized.

Copy link
@edolstra

edolstra Sep 4, 2019

Author Member

Yes, because this is not an overlay and there is no override mechanism at the moment (so the rec is the same as self.outputs).

@Zimmi48
Copy link
Member

left a comment

This RFC looks terrific overall. I'm really excited as it will make it easier:

  • For projects to distribute an up-to-date package, even one that reflects the current development branch. For instance, this is already something that we do in Coq, combined with a Cachix binary cache, to facilitate testing compatibility with the latest version of Coq in CI.
  • To distribute a set of packages present in an ecosystem registry, either officially (for instance the Coq package registry is currently opam-based, but we could also provide a Nix version) or as a community project in nix-community (like the new MELPA mirror).

But I'd like to strongly encourage the following modifications to the text of the RFC:

  1. Do not use the terminology "registry" as you do in the text of this RFC because it will only bring confusion, as nixpkgs itself is already a registry (a package registry), more specifically a portable registry (cf. https://github.com/ipfs/package-managers/blob/master/docs/categories.md).

    What you propose here would be a kind of meta-registry, a registry containing other registries (nixpkgs, plus for instance a MELPA mirror, a Stackage mirror, the Coq package registry, etc.).
    OK some flakes may contain a single package, so this is a kind of hybrid, but not enough justification to use a confusing terminology.

  2. The RFC does not discuss the policy of the (meta-)registry at all (or very little). I think that's fine because the policy of the (meta-)registry should be the specific topic of a subsequent RFC. The current RFC should make this clear. Starting it without carefully designing its policy would be disastrous IMHO, but it is fine to delay its creation (or to make it only contain nixpkgs at the beginning) because flakes will still be useful (and very much testable) before such a (meta-)registry is created.

    For instance, who should be able to submit an addition to the (meta-)registry? I think only the official maintainers of a project should, with an exception for registry mirrors hosted in nix-community. Indeed, there is an important underlying trust issue. Once you accept a flake in the (meta-)registry, you do not control the updates that are made to it. If some random person is allowed to add a flake for any project without connection with the maintainers, then they can start inserting malicious code afterwards. When the maintainers of a project do not provide a flake for their package, then the only way to distribute this package using Nix should be through nixpkgs, because then each update can be properly reviewed.

    Anyway, this was just an example to highlight the importance of the (meta-)registry's policy. I do not want to start the debate now. All I want is that the RFC state that it will be the topic of a subsequent RFC.


The flakes mechanism seeks to address all these problems. It can be
seen as a "Cargo/NPM/... for Nix" except that it's built into Nix
rather than a separate tool.

This comment has been minimized.

Copy link
@Zimmi48

Zimmi48 Sep 5, 2019

Member

I suggest removing this sentence which does not clarify what flakes are at all. The comparison with language package managers does not stand because Nix is not a normal programming language, it is a packaging language.

This comment has been minimized.

Copy link
@Zimmi48

Zimmi48 Sep 5, 2019

Member

Let me insist on something because I read some really odd things while going back in the discussion:

thinking about making nix a package manager

Nix is already a package manager. Just because it doesn't conform with what you are used to with some package managers like npm and Cargo does not make it less a package manager. Package managers have an almost 30-year long history, with very varied design decisions. And even today, npm and Cargo (and RubyGems and PyPI) do not represent the only way to make a good modern package manager.

This comment has been minimized.

Copy link
@toonn

toonn Sep 5, 2019

Isn't the point of this comparison that flakes would make it easier to package language-specific package manager ecosystems for nix-the-non-language-specific-package-manager?

As an example, currently there's a couple GHCs in nixpkgs, if you need one that isn't you're out of luck. Flakes would decouple the GHCs needed by nixpkgs from the ones that can be made available. Flakes'd be more like stackage LTSes, available for as long as anyone needs them. (Note that I'm not saying it's impossible to pin nixpkgs to the latest revision that had a GHC or that other technical solutions exist. I am saying flakes'd turn this from a PITA into simply specifying the right flake.)

This comment has been minimized.

Copy link
@Zimmi48

Zimmi48 Sep 5, 2019

Member

Sure, it would make such a use simpler and this is one of the great things about flakes.

But this sentence as is can be interpreted in many ways. If everyone who reads it interprets it differently, it's better to just remove it. This RFC is overall very clear, but this sentence does not help.

@arianvp
Copy link

left a comment

I like this. As it also aligns with the usage of # in URIs which identify a "sub-resource of a resource"

However, I think it also introduces a bit of ambiguity.

We now separate github:blah/lol#foo to a "uri" github:blah/lol and a package path foo right? But according to the URI RFC the entirety github:blah/lol#foo should be treated as the URI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.