Python's packageOverrides isn't composable

[twhitehead]

Issue description

@FRidh the packageOverrides function argument approach to overriding package used in the base python function (here is the 2.7 one) doesn't compose in overlays. Specifically, if you follow the directions in the nixpkgs manual only the last declared override winds up being applied.

I haven't actually thought of a solution to this yet, other than determining that it is pretty much impossible to work around as you can only get a hold of a closed version of the previous package set via python.pkgs. Perhaps looking into the haskell infrastructure might give some directions (haven't looked at it in a while, but the fixpoint stuff looks like it came from there)?

Steps to reproduce

Put the following into ~/.config/nixpkgs/overlays/1.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      one = "definition from 1.nix";
    };
  };
}

and this into ~/.config/nixpkgs/overlays/2.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      two = "definition from 2.nix";
    };
  };
}

Ideally both of these would get applied. This is not the case though as you can see from the following

nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'

error: attribute 'one' missing, at (string):1:28

nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'

"definition from 2.nix"

If you move the 2.nix file aside then you see the override from 1.nix but not 2.nix

mv ~/.config/nixpkgs/overlays/2.nix{,-inactive}
nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'

"definition from 1.nix"

nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'

error: attribute 'two' missing, at (string):1:28

[FRidh]

Yep, this is a known issue, one which bothers me as well. I think we should pull the creation of the package set out of the interpreters.

As a workaround, keep your overrides as an attribute, something like pythonOverrides. The next overlay can then use composeExtensions.

[FRidh]

Related #44196

[twhitehead]

Thanks for the suggestions and links.

If I'm understanding where you are going with your comment, you are saying if the interpreter package pulled in the package set instead of producing it, you could then override the package set in the standard way with overlays.

The interpreter would automatically pick up changes because it would be closed over the final package set. You could suck the final interpreter in the package set if you just re-exported it for compatibility without creating any loops.

Cheers! -Tyson

[FRidh]

Exactly that!

[FRidh]

if the interpreter package pulled in the package set instead of producing it

But, having the overrides is enough:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
  python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}

overlay_b.nix:

self: super: {
  pythonOverrides = lib.composeExtensions (selfPython: superPython: {
    pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  }) super.pythonOverrides;
}

(not tested)

[twhitehead]

Thanks. I implemented that and it works well. 👍

Unless you would like to leave it open, feel free to close this issue.

[Shados]

For a more standard-nixos version of said workaround, here's more or less what I'm using:

python-override-setup.nix

{ config, lib, pkgs, ... }:
{
  # Setup for https://github.com/NixOS/nixpkgs/issues/44426 python
  # overrides not being composable...
  nixpkgs.overlays = lib.mkBefore [
    (self: super: let
      pyNames = [
        "python27" "python34" "python35" "python36" "python37"
        "pypy"
      ];
      overriddenPython = name: [
        { inherit name; value = super.${name}.override { packageOverrides = self.pythonOverrides; }; }
        { name = "${name}Packages"; value = super.recurseIntoAttrs self.${name}.pkgs; }
      ];
      overriddenPythons = builtins.concatLists (map overriddenPython pyNames);
    in {
      pythonOverrides = pyself: pysuper: {};
      # The below is just a wrapper for clarity of intent, use like:
      # pythonOverrides = buildPythonOverrides (pyself: pysuper: { ... # overrides }) super.pythonOverrides;
      buildPythonOverrides = newOverrides: currentOverrides: super.lib.composeExtensions newOverrides currentOverrides;
    } // listToAttrs overriddenPythons)
  ];
}

some-module.nix

{ config, lib, pkgs, ...}:
{
  nixpkgs.overlays = [
    (self: super: {
      pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
        some-package = "definition from some-package.nix";
      }) super.pythonOverrides;
    })
  ];
}

another-module.nix

{ config, lib, pkgs, ...}:
{
  nixpkgs.overlays = [
    (self: super: {
      pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
        another-package = "definition from another-package.nix";
      }) super.pythonOverrides;
    })
  ];
}

Some notes

• This does work, for at least some values of 'work'...
• The use of mkBefore is to ensure that this is independent of the default order of expression evaluation.
• I've a sneaking suspicion I've done something terrible somewhere in there, so please do tell if so.

[danbst]

This should be solved with #54266

$ code='with import ./. { overlays = [
    (self: super: {
        pythonPackages.one = "definition from 1.nix";
    })
    (self: super: {
        pythonPackages.two = "definition from 2.nix";
    })
]; }'

$ nix-instantiate --eval -E "$code; pythonPackages.one"
"definition from 1.nix"

$ nix-instantiate --eval -E "$code; pythonPackages.two"
"definition from 2.nix"

[deliciouslytyped]

I'm not quite sure about this, but .override can be used with a function which takes the previous argument set: .override (a: { stuff = doSomething a.oldStuff}) so I think in theory one could write a function that overrides the previous arguments with a recursive merge? It's been a while but that sounds close to what I did here: https://github.com/deliciouslytyped/nix-ghidra-wip/blob/43b2207729db83f84c0812ff9a054907d5ddc222/packages.nix#L6

[infinisil]

Actually this PR of mine solves this issue: #67422

It allows overrides of the form

python3.pkgs.overrideScope' (self: super: {
  # ...
})

Without discarding previous changes.

And this with very little code changed.

[DavHau]

if the interpreter package pulled in the package set instead of producing it

But, having the overrides is enough:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
  python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}

overlay_b.nix:

self: super: {
  pythonOverrides = lib.composeExtensions (selfPython: superPython: {
    pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  }) super.pythonOverrides;
}

(not tested)

Thanks, with this i got it working. Though, it needed a few modifications to work. Here the corrected version in case anyone needs it:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py = superPython.py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
}

overlay_b.nix:

self: super: rec {
  pythonOverrides = self.lib.composeExtensions super.pythonOverrides (selfPython: superPython: {
    pytest = superPython.pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  });
  python36 = super.python36.override { packageOverrides = pythonOverrides; };
}

[deliciouslytyped]

I forgot about this thread and I haven't read carefully enough to tell if this helps here, but maybe it's relevant; https://discourse.nixos.org/t/makeextensibleasoverlay/7116

[DavHau]

I'm now using the following function for mach-nix to merge an arbitrary list of pythonOverrides:

  mergeOverrides = with pkgs.lib; overrides:
    if length overrides == 0
    then a: b: {}  # return dummy overrides
    else
      if length overrides == 1
      then elemAt overrides 0
      else
        let
          last = head ( reverseList overrides );
          rest = reverseList (tail ( reverseList overrides ));
        in
          composeExtensions (mergeOverrides rest) last;

[andersk]

@DavHau mergeOverrides = lib.foldr lib.composeExtensions (self: super: { })

[deliciouslytyped]

Maybe we should put that in lib (or not) :p

https://github.com/deliciouslytyped/nix-rootedoverlay/blob/84f14bae677d4cb9a311bdedc066e0c91d4f0703/rooted.nix#L35

[FRidh]

@deliciouslytyped that was proposed in #33258

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/weird-overlay-behavior-with-python-packages/14469/7

[spease]

To save others the substantial amount of trouble I had to deal with trying to get the various suggestions in this thread to work, if you're using mach-nix, the easiest way to deal with overrides is to avoid this entirely and instead use its built-in override methodology:
https://github.com/DavHau/mach-nix/blob/master/examples.md#simplified-overrides-_-argument

[stale]

I marked this as stale due to inactivity. → More info

[centromere]

Not stale.

[DylanRJohnston-FZ]

I've just encoutered this issue when trying to override the broken pyopenssl package on aarch64-darwin for the awscli2 package.

As you can see here

let
  py = python3.override {
    packageOverrides = self: super: {
      awscrt = super.awscrt.overridePythonAttrs (oldAttrs: rec {
        version = "0.13.11";
        src = self.fetchPypi {
          inherit (oldAttrs) pname;
          inherit version;
          sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E=";
        };
      });
    };
  };

It creates its own override of python3 meaning any of my attempts to mark pyopenssl is not broken and skip the test phase of twisted fail because the overrides are not composable.

Would the awscli2 derivation need to be adjusted to make use of the composeExtensions method above?

Right now I've worked around it by forking the derivation entirely and then overlaying that instead. It really sucks that this is outside of the usually pleasant overlay system.

[DylanRJohnston-FZ]

If you change the awscli2 definition to use overrideScope instead it can now be properly overriden from an overlay.

# awscli2.nix
let
  pypkgs = python3.pkgs.overrideScope
    (self: prev: {
      awscrt = prev.awscrt.overridePythonAttrs (oldAttrs: rec {
        version = "0.13.11";
        src = self.fetchPypi {
          inherit (oldAttrs) pname;
          inherit version;
          sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E=";
        };
      });
    });
in

# overlay.nix
self: prev: {
  python3 = prev.python3 // {
    pkgs = prev.python3.pkgs.overrideScope (self: prev: {
      pyopenssl = prev.pyopenssl.overrideAttrs (_: { meta.broken = false; });
      twisted = prev.twisted.overridePythonAttrs (_: { doCheck = false; });
    });
  };
}

I don't currently see any examples in nixpkgs of using overrideScope with python packages, but it looks like it works.

[spease]

This seems like a definite usability hurdle. Three different overrides required to set one parameter on two Python packages.

[dbaynard]

👍 on the overrideScope solution. Here's how I'm using it.

I have two functions that each return sets of python packages.

1. modifiedPackagesOverlay takes existing packages and changes properties; its final and prev are sets of python3Packages.
2. newPackagesOverlay adds new packages; here final and prev correspond to top level packages, and python dependencies have to be specified via python3Packages.

The result is an overlay, exported as part of a flake. The python3 that results from applying the overlay can be overridden in a similar manner.

{
  overlays.python = final: prev:
    let
      pythonPkgsScope = prev.python3.pkgs.overrideScope (prev.lib.composeManyExtensions [
        (_: _: newPackages final prev)
      ]);
    in
    {
      python3 = prev.python3 // {
        pkgs = pythonPkgsScope;

        packageOverrides = lib.warn ''
          `python3.packageOverrides` does not compose;
          instead, manually replace the `pkgs` attr of `python3` with `python3.pkgs.overrideScope` applied to the overrides.
        ''
          prev.python3.packageOverrides;
      };

      python3Packages = pythonPkgsScope;
    };
}

This overlay also works if applied after one which uses pythonOverrides, but not if applied before.

[Edit: turns out setting python3Packages = final.python3.pkgs now breaks MacOS evaluation… I'm not sure why, but this does work.]

[DavHau]

The nixos module system could probably be used to solve this problem nicely.

[WizardUli]

If you change the awscli2 definition to use overrideScope instead it can now be properly overriden from an overlay.

# awscli2.nix
let
  pypkgs = python3.pkgs.overrideScope
    (self: prev: {
      awscrt = prev.awscrt.overridePythonAttrs (oldAttrs: rec {
        version = "0.13.11";
        src = self.fetchPypi {
          inherit (oldAttrs) pname;
          inherit version;
          sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E=";
        };
      });
    });
in

# overlay.nix
self: prev: {
  python3 = prev.python3 // {
    pkgs = prev.python3.pkgs.overrideScope (self: prev: {
      pyopenssl = prev.pyopenssl.overrideAttrs (_: { meta.broken = false; });
      twisted = prev.twisted.overridePythonAttrs (_: { doCheck = false; });
    });
  };
}

I don't currently see any examples in nixpkgs of using overrideScope with python packages, but it looks like it works.

This doesn't seem to work for me. I have the following overlay applied to nixpkgs

final: prev: {
  python3 = prev.python3 // {
    pkgs = prev.python3.pkgs.overrideScope (python-final: python-prev: {
      solidpython2 = python-final.callPackage ./solidpython2.nix { };
    });
  };
}

where solidpython2.nix is somethig like

{ buildPythonPackage, fetchPypi }:
buildPythonPackage rec {
  ............
}

and then I have this

{
  environment.systemPackages = with pkgs; [
    blah
    blah
    
    (python3.withPackages (ps: with ps; [
      blah
      blah
      solidpython2
    ]))
  ];
}

which ends in error: undefined variable 'solidpython2'.

[rskew]

@WizardUli It looks like withPackages doesn't pick up the overriden python3.pkgs, instead it keeps referring to pythonPackages from the let block:

nixpkgs/pkgs/development/interpreters/python/default.nix

Lines 100 to 101 in 2ad9cc7

	withPackages = import ./with-packages.nix { inherit buildEnv pythonPackages;};
	pkgs = pythonPackages;

(although I don't understand how passthruFun works and how it would be updated to pick up an overriden pkgs)

I've had success with this kind of usage:

    (python3.withPackages (_: with python3.pkgs; [
      solidpython2
    ]))

[WizardUli]

I've had success with this kind of usage:

    (python3.withPackages (_: with python3.pkgs; [
      solidpython2
    ]))

That's exactly what I've ended up using.

[FRidh]

You can just use the python.buildEnv function which withPackages uses under the hood.

[matshch]

By the way, there is now pythonPackagesExtensions attribute (added in #91850) that contains a list of overlays automatically applied to the Python packages set, and as a bonus it works for all Python versions simultaneously.