Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add overlays mechanism to Nixpkgs. #21243

Merged
merged 13 commits into from Jan 16, 2017
Merged

Add overlays mechanism to Nixpkgs. #21243

merged 13 commits into from Jan 16, 2017

Conversation

nbp
Copy link
Member

@nbp nbp commented Dec 17, 2016

This patch add a new argument to Nixpkgs default expression named "overlays".

By default, the value of the argument is either taken from the environment variable NIXPKGS_OVERLAYS,
or from the directory ~/.nixpkgs/overlays/. If the environment variable does not name a valid directory
then this mechanism would fallback on the home directory. If the home directory does not exists it will
fallback on an empty list of overlays.

The overlays directory should contain the list of extra Nixpkgs stages which would be used to extend the
content of Nixpkgs, with additional set of packages. The overlays, i-e directory, files, symbolic links
are used in alphabetical order.

The simplest overlay which extends Nixpkgs with nothing looks like:

self: super: {
}

More refined overlays can use super as the basis for building new packages, and self as a way to query
the final result of the fix-point.

An example of overlay which extends Nixpkgs with a small set of packages can be found at:
https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix

To use this file, checkout the repository and add a symbolic link to
the moz-overlay.nix file in ~/.nixpkgs/overlays directory.

Motivation

The goal of this patch is to provide an alternative to the pkgs.overridePackages function (#20927), which re-execute Nixpkgs fix-point with new inputs. Instead, the alternative would be use Nixpkgs top-level function, with an overlays attribute which would be a list with the same argument as what is today the argument given to pkgs.overridePackages.

Thus, in addition to provide a useful extensible layer to Nixpkgs third parties, such as Mozilla, or any other company. This would remove the existing leakage of Nixpkgs fix-point out of Nixpkgs. This feature would also be nicer for the security-updates branch (#10851), as the overlays would be under the fix-point on which the patching mechanism works on.

Things done
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"

@mention-bot
Copy link

@nbp, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Ericson2314, @errge and @edolstra to be potential reviewers.

@Ericson2314
Copy link
Member

Hmm I'll need to think more about the ramifications for security updates, but otherwise I'm confused by the motivation for this. To me, mkExtensible is one of the best parts of the fix-extends ideom, and one I hope to see used more widely, not less.

CC @cstrahan, who created that abstraction.

@nbp
Copy link
Member Author

nbp commented Dec 17, 2016

@Ericson2314, @cstrahan:

The security-updates branch is using the fixed function a first time within the fix-point for base packages, and then as a standalone function for patching packages, and as a guide for another fix-point for propagating the patches.

The makeExtensibleWithCustomName function create a new attribute which let you inject phases in the pipeline of fixed function. If we wanted to carry the security-updates mechanism for the pkgs.overridePackages, then we would have to add special cases for this function, which I consider to be more painful than useful.

By adding the overlays list of extra phases, you can easily combine orthogonal extensions of Nixpkgs, which surpass the pkgs.overridePackages which only take one extra phase. Moreover, the pkgs.overridePackages function can easily be emulated as demonstrated in #20927 (comment)

Thus, I see no need to keep the pkgs.overridePackages function.

Copy link
Contributor

@the-kenny the-kenny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some directory-resolution related questions.


<listitem>

<para>As the directory located at
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this option work with different users? Will only the overlay set of the user executing nixos-rebuild considered or will it always use root's?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current patch, it will take the overlays of the user which is executing the command.

With the above suggestion, none should be considered unless they are specified in the configuration.nix file, or any other module.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the directory <filename>~/.nixpkgs/overlays/</filename>.


<listitem>

<para>As argument of the imported attribute set. When importing Nixpkgs,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to hard-code this directory for nixos-rebuild from configuration.nix? I think that might be useful.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree.

This would be a terrible experience if one configuration.nix is not evaluated the same way across computer depending on the ~/.nixpkgs/overlays directory content.

I will add the overlays argument to nixos/modules/misc/nixpkgs.nix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed in commit 46ae6b5

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grammar: As an argument ...

teh
teh approved these changes Dec 26, 2016
Copy link
Contributor

@teh teh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really amazing work & a lot of power for such a small change! I'm very much in favour.

<title>Overlays</title>

<para>This chapter describes how to extend and change Nixpkgs content using
overlays. Overlays are used to add phases in the fix-point used by Nixpkgs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"phases" sounds like pre or post-phase. Is a "phase" one file (and one function), and executed in alphabetical order?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, one phase / layer is one overlay, which is a single function self: super: { ... }. Maybe I should use the term layer instead of phase?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "layer" would be better, too.

<title>Installing Overlays</title>

<para>Overlays are looked for in the following order, the first valid one is
considered, and all the rest are ignored:
Copy link
Contributor

@teh teh Dec 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would we ignore invalid files instead of throwing an error? This seems like a recipe for "silent, hard-to-debug" problems?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant is the fallback logic, we consider the environment variable NIXPKGS_OVERLAYS, and check if this is a directory. Then we do the same check for the $HOME/.nixpkgs/overlays.

I agree, we should throw if NIXPKGS_OVERLAYS is set, but the content does not validate.
But I do not think we should do anything if the $HOME/.nixpkgs/overlays directory does not exists.

<listitem>

<para>As a directory pointed by the environment variable named
<varname>NIXPKGS_OVERLAYS</varname>. This directory can contain symbolic
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does NIXPKGS_OVERLAYS support colon separated paths like PATH?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the moment no, I think this would be a nice addition in a future patch.

I also think being able to pull a Nix expression from a trusted remote (such as https://my.company/overlay.nix) would also be interesting.

@@ -88,6 +90,11 @@ following incompatible changes:</para>
<literal>networking.timeServers</literal>.
</para>
</listitem>

<listitem>
<para><literal>pkgs.overridePackages</literal> function no longer exists. Instead import Nixpkgs a second time using <literal>import pkgs.path {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very long space after "exists." Maybe a missing line break?

@Ericson2314
Copy link
Member

Note in 2278414#diff-19c9515018ff76dea736b57000a62d5bR49 I use overridePackages in a way that I worry would be hard to replicate with this.

@Ericson2314
Copy link
Member

More broadly, as I wrote in #16531 (comment) I think I thinking instead of having nixpkgs vs nixos, we should have nixpkgs vs "nixuser" vs "nixos". nixkpgs would have no config attrset, and no impure.nix, and basically just be packages. nixuser would have some sort of config system (probably module system), things like firefoxWrapper, dotfiles management if we ever get that.

I think splitting the pure and impure parts of this would be an excellent further justification for factoring out this "nixuser".

domenkozar
domenkozar previously requested changes Dec 28, 2016
Copy link
Member

@domenkozar domenkozar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.

I think documentation needs someone who speaks English natively to flesh it out.

For Overlays I think we should have a tool that manages them. Something similar to git submodule that would sync git repositories. We should decide to go this route from the first day, prohibiting manipulation of ~/.nixpkgs/overlays by hand.

<para></para>
<para>Nixpkgs is now extensible through <link
xlink:href="https://github.com/NixOS/nixpkgs/pull/21243">overlays</link>.
See the Nixpkgs manual for more information.</para>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a link to the manual with docs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@domenkozar How would you link to the new Nixpkgs documentation from the release notes?

<listitem>
<para><literal>pkgs.overridePackages</literal> function no longer exists.
Instead import Nixpkgs a second time using <literal>import pkgs.path {
overlays = [ <replaceable>...</replaceable> ]; }</literal>.</para>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not clear what pkgs is and what would be the contents of overlays. It's better to point to overlays section in the manual.

@teh
Copy link
Contributor

teh commented Jan 2, 2017

I gave this a spin and I'm not super convinced that a directory is the best public API (it's not terrible either). The additional overlays parameter however is awesome!

@domenkozar I think the docs are OK - do you have a specific thing in mind? Also, what kind of tool do you have in mind?

Both @nbp & @domenkozar - what do you think about splitting the filesystem API (~/.nixpkgs/overlays and NIXPKGS_OVERLAYS) into a separate branch so we can get the uncontroversial parts merged soon? I can do the work :)

Copy link
Contributor

@aneeshusa aneeshusa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall docs look good, this is mostly small grammar fixes.

Also a request for a special callPackage to make using overlays easier.


<title>Overlays</title>

<para>This chapter describes how to extend and change Nixpkgs content using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: content is a bit strange here, maybe use packages or just remove it

<title>Overlays</title>

<para>This chapter describes how to extend and change Nixpkgs content using
overlays. Overlays are used to add layers in the fix-point used by Nixpkgs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Only use one space between sentences

<section xml:id="sec-overlays-install">
<title>Installing Overlays</title>

<para>The set of overlays are looked for in the following order, only the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grammar: set of overlays is singular but are is for plurals; I would say The set of overlays is looked for in the following places.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would split this into two sentences between order and only.

openssh = super.openssh.override {
hpnSupport = true;
withKerberos = true;
kerberos = self.libkrb5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing semicolon

pkgs.overridePackages (self: super: ...)
</programlisting>

Should be replaced by:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: don't capitalize, use should

</orderedlist>
</para>

<para>For the second and third option, the directory contains either
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

..., the directory should contain Nix expressions defining the overlays. Each overlay can be a file, a directory containing a default.nix file, or a symlink to one of those. The expressions should follow the syntax described in ...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this section should describe any ordering guarantees (or lack thereof) about overlays retrieved from a directory.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the following paragraph right after to describe the ordering of overlays, which has an impact both when providing a list of overlays as arguments of Nixpkgs, or when using a directory.

The order of the overlay layers can influence the recipe of packages if two layers override the same recipe. In the case where overlays are loaded from a directory, these are loaded in alphabetical order.

following the syntax described in <xref
linkend="sec-overlays-layout"/>.</para>

<para>To install an overlay, using the last option. Clone the repository of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grammar: To install an overlay using the last option, you can clone the overlay's repository and add a symbolic link to it in the ...


<para>This chapter describes how to extend and change Nixpkgs content using
overlays. Overlays are used to add layers in the fix-point used by Nixpkgs
to bind the dependencies of all packages.</para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlays can also be used for adding packages, so I would make this a bit more general, maybe used by Nixpkgs to compose the set of all packages.


<para>As a directory pointed by the environment variable named
<varname>NIXPKGS_OVERLAYS</varname>. This directory can contain symbolic
links to Nix expressions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to mention symlinks here since it's described below.


<para>As the directory located at
<filename>~/.nixpkgs/overlays/</filename>. This directory can contain
symbolic links to Nix expressions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to mention symlinks here since it's described below.

@nbp
Copy link
Member Author

nbp commented Jan 14, 2017

@aneeshusa , Thanks a lot of the review, I will fix the nits tomorrow.

For the custom callPackage, I think that is a good idea, I will think about it and the consequences/applications with the security-updates branch.

@aneeshusa
Copy link
Contributor

FYI, just noticed I had made a typo in my example for a custom callPackage, I fixed it at #21243 (comment).

@nbp nbp dismissed domenkozar’s stale review January 15, 2017 16:18

All review comments are addressed. I am slowly working on a nix-overlay tool.

Copy link
Contributor

@aneeshusa aneeshusa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more grammar nits, and some confusion about the usage of callPackage here.

IDK why GitHub is showing these as outdated, they should still be valid comments at time of writing...

to the entry Nix expression of the overlay. These Nix expressions are
following the syntax described in <xref
linkend="sec-overlays-layout"/>.</para>
<para>For the second and third option, the directory should contain Nix expressions defining the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grammar: options

the overlay, and add a symbolic link to it in the
<filename>~/.nixpkgs/overlays/</filename> directory.</para>
<para>The order of the overlay layers can influence the recipe of packages if multiple layers override
the same recipe. In the case where overlays are loaded from a directory, these are loaded in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are loaded

alphabetical order.</para>

<para>To install an overlay using the last option, you can clone the overlay's repository and add
a symbolic link to in the <filename>~/.nixpkgs/overlays/</filename> directory.</para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

link to it in

<para>An overlay is a Nix expression, which is a function which accepts 2
arguments.</para>
<para>Overlays are expressed as Nix functions which accept 2 arguments and return a set of
packages</para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a period or colon at the end.

boost = super.boost.override {
python = self.python3;
};
rr = super.callPackage ./pkgs/rr {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this super.callPackage? I would think super.callPackage would use packages from super for dependencies, where the documentation says to use packages from self.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g. I would think we need to provide a custom callPackage for each self layer using
lib.callPackagesWith.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically you could use both, but this would get some extra meaning in the futur (security-update branch), because self.callPackage goes twice through the self attribute evaluation. The first one to resolve the callPackage function, and the second time to resolve the dependencies from self which are captured by the callPackage function.

Note that self is the same provided to all layers. So using self.callPackage is for the moment identical to super.callPackage, except for this extra hop through the fix-point.

I documented it as functions should be taken from super, because they already capture self if they have to.

Also note that this way, we make it syntactically verifiable, as anything which comes around the dependencies uses super, and anything which is use to find dependencies uses self.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable, but can you explain how functions already capture self if they need to? E.g. if I have an overlay that adds two new packages:

self: super: {
    # Let's say all deps are already in super,
    # so I can see this working with super.callPackages or self.callPackages
    foo = super.callPackage {};
    # But this depends on foo, which isn't in super
    bar = super.callPackage {}; 
}

How does the second super.callPackage get a reference to foo to provide to bar?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I tried it out and looked at the implementation and I now understand why this works, albeit the current behavior is slightly surprising to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self: super:

{
  callWithSelf = f: f self;
}

Any layer defined after could then do:

self: super:

{
  mySelfClone = super.callWithSelf (x: x);
}

@nbp
Copy link
Member Author

nbp commented Jan 15, 2017

Tested locally on top of nixpkgs-unstable channel:

  • ./maintainers/scripts/travis-nox-review-pr.sh nixpkgs-verify nixpkgs-manual nixpkgs-tarball nixpkgs-unstable
  • ./maintainers/scripts/travis-nox-review-pr.sh nixos-options nixos-manual

@nbp nbp merged commit 8366525 into NixOS:master Jan 16, 2017

<programlisting>
let
pkgs = import &lt;nixpkgs&gt; {}; in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noticed a double in (see next line) - @nbp if you have direct push permissions maybe a quick fix? Otherwise I'll try to mop it up with some other change.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed as part of commit 0214d94. Thanks!

@Ericson2314
Copy link
Member

Wait, do overlays currently apply to bootstrapping stages after all?

@@ -80,7 +83,7 @@ in let
boot = import ../stdenv/booter.nix { inherit lib allPackages; };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do want all bootstraping stages to include the overlay unconditionally, we could instead "partially apply" overlays to all packages (3 lines above the inherit lib nixpkgsFun;, but I cannot comment there because unchanged).

@nbp
Copy link
Member Author

nbp commented Jan 17, 2017

@Ericson2314

Wait, do overlays currently apply to bootstrapping stages after all?

Yeah, but I think it would makes sense to disable that as soon as we can move the bootstraping process in Nixpkgs fix-point, as this would no longer be needed. For the moment I would not recommend anybody to rely on overlays for changing the stdenv, except by doing it explicitly.

@teh Thanks, I will fix it later, if nobody does it before ;)

@Ericson2314
Copy link
Member

@nbp, well, to be clear, we can disable it in the meantime too. But that plan sounds fine to me.

@globin globin mentioned this pull request Jan 19, 2017
@grahamc grahamc mentioned this pull request Jan 23, 2017
17 tasks
@grahamc grahamc added the weekly label Jan 23, 2017
@aneeshusa aneeshusa mentioned this pull request Jan 28, 2017
7 tasks
@edolstra
Copy link
Member

IMHO we should not add more ad hoc environment variables like $NIXPKGS_OVERLAYS. Why not use the NIX_PATH mechanism for this?

@edolstra
Copy link
Member

edolstra commented Feb 1, 2017

I've changed $NIXPKGS_OVERLAYS to <nixpkgs-overlays>, and ~/.nixpkgs/overlays to ~/.config/nixpkgs/overlays in order to not add more non-XDG-compliant paths.

BTW, since the point of overlays is that they can be stacked easily, should the overlays in <nixpkgs-overlays> and ~/.config/nixpkgs/overlays be merged, rather than having the former override the latter?

@nbp
Copy link
Member Author

nbp commented Feb 1, 2017

@edolstra

BTW, since the point of oveprlays is that they can be stacked easily, should the overlays in <nixpkgs-overlays> and ~/.config/nixpkgs/overlays be merged, rather than having the former override the latter?

I honestly do not know yet what would be the most desirable behaviour.

My initial though was that we could replace the default configuration and test with a different one, which is interesting in terms of reproducibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment