Deprecate packageOverrides?

[lheckemann]

packageOverrides is replaced by overlays as far as functionality goes, and has a number of disadvantages:

• Learning/documentation overhead. We have 3 distinct things with the word override in them: packageOverrides, pkg.override introduced by callPackage, and pkg.overrideAttrs introduced by stdenv.mkDerivation. This is very confusing to beginners (it was to me as well, and we didn't have overlays at the time). Additionally, packageOverrides shares a space with overlays while being less powerful.
• Unwieldy composition. I seem to recall there being some trick involving something like self: let super = self.pkgs; to get an approximation of what overlays provide, but it inrtoduces a fixpoint in each layer AFAIU. I personally don't know exactly how this works, but overlays are pretty clear in this respect.

For this reason, I think it would be nice to deprecate packageOverrides (remove it from docs in favour of overlays, add a warning when nixpkgs is evaluated with config.packageOverrides set) and eventually remove it completely, leaving perhaps only an explanatory error message. The warning/error messages should explain how to turn a packageOverrides function into an overlay to ease use of existing third-party documentation.

Any opinions on whether this is a good idea at all, further practical considerations, or any other relevant commentary welcome!

[grahamc]

cc @nbp

[nbp]

If I do not make any mistake this should almost be a backward compatible overlay for adding the feature back:

self: super:

(super.config.packageOverrides or (super: {})) super

The remaining issue is about modifying any bootstrap packages. This should be fixed by including all the stages of the bootstrap as attribute names within the main fix-point.

[lheckemann]

Given the current implementation of packageOverrides, yes, I think that should work :D

[lheckemann]

Opened PR #43560 to implement this.

[jcrben]

Overlays seem tricky to use correctly. Therefore I think deprecation at this point could on net cause more UX damage than non-deprecation. I'm thinking about the long-term user I met at a meetup a couple months ago who hadn't migrated yet. 🤔

Right now, they aren't showing up for me on my mac in nix-env -q (despite adding the directory to my NIX_PATH; originally was using single file approach) altho I can install them OK, including with nix-shell or nix-build (might be better on NixOS machine). (prolly just user error related to a package not having the proper attributes (more at https://gist.github.com/LnL7/570349866bb69467d0caf5cb175faa74#gistcomment-2655668))

Looks like #26487 (comment) ran into similar search problems altho I don't understand the fix. Admittedly search isn't absolutely necessary since I can read the overlay files. And #25264 has a long discussion about how to get overlays to be picked up by tools, but I don't see any sign of commits or docs to streamline the situation. Altho I guess https://nixos.wiki/wiki/Overlays has most of it?

Maybe hardening overlays and triaging / resolving the open issues on them (e.g., #24907 #28558 ) before deprecating?

And the existing docs on overlays could maybe be better as there are nuances that could to be called out a bit more strongly, e.g. single file needs to return a list while overlays in a directory are functions.

[ckauhaus]

I've converted an overlay to a packageOverrides construct lately because the overlay took too much memory to evaluate. It seems that a complete pkgs attrset mut be constructed on each fixpoint iteration. packageOverrides are more compact on memory usage exactly because they involve a local fixpoint.

This is by no way meant to stop overlays, but it should be taken into consideration before deprecating packageOverrides too early.

[lheckemann]

packageOverrides is implemented as a stage of nixpkgs which behaves just like an overlay, so I'm not sure I understand how this makes a difference.

Although at this point it almost seems to me like removing overlays again might be better 🙈

[ckauhaus]

I got random GC errors like "out of memory" or "too many heap sections" while using overlays (something similar to #40456). After converting overlays to packageOverrides, the errors disappeared. Actually no idea if this was just luck or a real solution.

[asymmetric]

I currently use packageOverrides to declare a set of packages in my ~/.config/nixpkgs/config.nix (explained here).

How would I do that - declaratively - with overlays? i.e. I don't want to have to run nix-env for each package in the overlay, but rather have a "meta-package" of sorts that bundles all the packages I want.

[lheckemann]

@asymmetric As documented in the warning in my PR https://github.com/NixOS/nixpkgs/pull/43560/files#diff-69df24a05ec3d577cb638113e9837ae1R119 :
take only the value of packageOverrides, and replace pkgs: with pkgs; with self: super: with self;, then put it in a .nix file in ~/.config/nixpkgs/overlays.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[lheckemann]

As mentioned on the issue, will do an RFC eventually

[stale]

I marked this as stale due to inactivity. → More info

[ShamrockLee]

Is this still revalent?

[lheckemann]

Yes

[asymmetric]

@lheckemann if this is still relevant (which I think it is), then maybe it should be turned into an RFC. I don't think @edolstra's mild opposition to this should stop you from writing it, for the record :)

Otherwise, it should be left stale and closed, eventually.

[lheckemann]

As mentioned on the issue, will do an RFC eventually

for some value of eventually…

[stale]

I marked this as stale due to inactivity. → More info

[ThibautMarty]

In the meantime, we may change the documentation without deprecating packageOverrides, what do you think?

I was helping a newcomer that used pkgs.buildEnv to install a set of packages at once. The method documented here uses packageOverrides and not overlays. His package set didn't contain a package that he patched in an overlay but the the original one which was confusing for him.
The fact that a newcomer seems to be able to create an overlay but is guided to use packageOverrides seems wrong to me.

I guess the use of packageOverrides here (and in other parts of the documentation) is for historical reasons only, so I would be in favor of replacing it with overlays.
I can make a patch if no one objects.

[YoshiRulz]

So to clarify, should this config.nix section

packageOverrides = pkgs: {
	myPackages = pkgs.buildEnv {
		name = "my-packages";
		paths = [ pkgs.hello ];
	};
};

be replaced with

final: prev: {
	myPackages = prev.buildEnv {
		name = "my-packages";
		paths = [ prev.hello ];
	};
}

as an overlay my-packages.nix? It seems to work at least.

[lheckemann]

@YoshiRulz yes, that looks mostly correct (though usually you'd want to get hello, being a package, from final so that it can still be affected by later overlays).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-is-fixed-point-used-for-overriding-nixpkgs-packages/35476/4