-
-
Notifications
You must be signed in to change notification settings - Fork 13.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Overlay/overriding improvement plan #99100
Comments
For an ideal overlay interface, a change like NixOS/nix#4090 would be very beneficial |
I totally agree, there's way to many things to get wrong and little corners. When I first started using nix there were overlays and I had a pretty confused time. I will also add that on nixUnstable there isn't |
|
Those are just arbitrary arguments btw, they can even be |
Yep, they can be arbitrary arguments. But in nixUnstable I notice that they're not anymore because of the check https://github.com/NixOS/nix/blob/b721877b85bbf9f78fd2221d8eb540373ee1e889/src/nix/flake.cc#L260 |
Ahh that's only for flakes though. But neat, didn't know that. |
Anyways, I should start explaining how I think this can be solved: One of the main insights I had while thinking about this was that currently the override author is responsible for getting overrides correct. This is just how it is for how overlays currently work, you need to know how to modify the original However it doesn't have to be this way: By redesigning overlays slightly, we can shift this responsibility to the package set author. This means that whoever defines the package set will be specifying in Nix code how it should be overridden, such that the people that actually write overrides won't have to know about it. How can this be implemented though? You see, currently overlays are just applied by essentially using So what we do is to change the left: right:
left // builtins.mapAttrs (name: rval:
# If the left attribute defines a custom merging strategy
if left ? ${name}.overlayMerge
# use that to merge the two values together
then left.${name}.overlayMerge left.${name} rval
# Otherwise override completely (old behavior)
else rval
) right Now with this, you can have an initial overlay that defines a package set: self: super: {
set = {
foo = 10;
# Multiple definitions of this set should be merged together
overlayMerge = lval: rval: lval // rval;
};
} And another one that extends this set how you'd expect it: self: super: {
# Just like that!
set.bar = 20;
} Which with this new merging strategy will not give you {
bar = 20;
foo = 10;
overlayMerge = <LAMBDA>;
} instead! So just by changing the self: super: {
set = super.set // {
bar = 20;
};
} Now this is just a single insight I had. This allows us to solve the problem of overlays being very hard to get right, given that the package set author provides a good enough |
@infinisil I think you have a typo in your examples - or I'm really not following how you ended up with bar = 20 and foo = 10, since those values don't show up anywhere else. :) |
@chreekat Oh yes, fixed now, thanks! |
I ended up here after trying to track down broken haskell overrides. I finally realized that haskellPackages.override is just broken if you use overlays. I'm debating how to work around this now. To test, I just created three overlays (test..nix, test2.nix, test3.nix) with this pattern, but obviously changing the number to 1, 2, and 3 in the corresponding overlay:
Then I look in the repl:
Intuitively, all six should be "seen". I'm debating how to work around this (I'm definitely not a nix expert). Does anyone have a patched or alternate override available for Haskell? This situation is frustrating, because overlays seem like a great way to organize local packages on top of nixpkgs, but it requires robust merging procedures to be in place. UPDATE To answer my own question, I've realized that I've been through this maze before, and there are a cluster of issues and blog posts addressing this point, with various recommendations for using override/overrides/extend/composeExtensions, most of which is undocumented. Also, along the way, the existing haskell docs have apparently been pulled from the nixpkgs manual and moved to a spam site. Ugh. UPDATE 2 The definitive answer seems to be in issue 44718. Here is my test written that way:
This works, but that issues seems to have been closed without being updating the documentation, which, in any case (as I indicated above) seems to have been removed from the manual. Also, to me this is just boilerplate, I'd love an explanation of what this is actually doing. Thanks. |
@infinisil now that I've found a solution for overlaying overrides in the current system, I'm looking at your proposal. I think I understand what you are proposing. How would this look for haskellPackages? It seems like if haskellPackages.callPackage, haskellPackages.override, haskellPackages.override.overrides, haskellPackages.extend, and composeExtensions were documented, and that documentation recognized overlays as the new normal, then people could achieve successful overlays. I guess I'm saying that your proposal will give authors more license to avoid telling us what is going on. For example, what really is the specification for haskellPackages.callPackage? We are in a situation where the creation and evolution of the nix language and nix programming practice has polluted the space in which we define packages with magic attributes that control the process. If we are going to continue down that road, then your solution has merit... but it makes me uncomfortable. In summary, I think the overlay mechanism is a game-changing achievement for packaging, but nothing will work properly if the authors of magic hooks assume that people can intuit the proper use of several layers of untyped higher order functions without documentation within the nixpkgs manual. As it stands, I think we all need to read the source code to gain a firm understanding. Maybe that is the reality? That would be a solution, but in that case the nixpkgs documentation should link directly to source code and call that out as the specification. I do appreciate that you are trying to solve the issue override/overlay issue, so please take my comments constructively. A big thumbs up for putting your ideas out and working toward a solution. |
I marked this as stale due to inactivity. → More info |
Not stale imho. |
Note that I started the Nixpkgs Architecture Team which should add some traction to this issue. It is being tracked here |
Nixpkgs overlays/overrides are way too hard to get right. Here's a seemingly innocent example:
But it has problems:
pkgs.pythonPackages
andpkgs.python2.pkgs
won't contain the package, even though it should point topkgs.python2Packages
Here's a Haskell example:
But it has problems:
.extend
doesn't properly work with.override
, which other overlays might use, see "haskellPackages.extend" and "haskellPackages.override" are incompatible #26561pkgs.haskell.packages.ghc883
won't contain the new package.extend
is quadratic. Every new.extend
access evaluates all previous overlays again.And this is just two package sets. There's also:
pkgs.linuxPackages
set, which useslib.makeExtensible
for allowing overrides with.extend
, but not with.override
lib.makeScope
mechanism, which creates a new scope that can be extended with.overrideScope'
, which works very much the same as.extend
. Don't use.overrideScope
though, that has its arguments flipped around.pkgs.appendOverlays
which works very much the same aspkgs.extend
, except for multiple overlayspkgs.perlPackages
doesn't support overriding with scoped overlays at all. You need to usesuper.perlPackages.<pkg>
to refer to a previous perl package, orself.perlPackages.<pkg>
for final ones.override
,.overrideAttrs
and.overrideDerivation
, and it's hard to know which one is right.overrideAttrs (old: { version = "1.2"; })
and wonder why it doesn't override the versionself
andsuper
is very tricky. And sometimes very intuitive. In fact, usingself
to avoid infinite recursion can sometimes even introduce inconsistencies.It's just a big mess!
This issue should serve as a place to track this problem and how it can be improved. I myself have thought about this a bunch, and will soon elaborate my idea to fix all these problems in a subsequent comment.
The text was updated successfully, but these errors were encountered: