Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python's packageOverrides isn't compossible #44426

Open
twhitehead opened this issue Aug 3, 2018 · 9 comments

Comments

@twhitehead
Copy link
Contributor

commented Aug 3, 2018

Issue description

@FRidh the packageOverrides function argument approach to overriding package used in the base python function (here is the 2.7 one) doesn't compose in overlays. Specifically, if you follow the directions in the nixpkgs manual only the last declared override winds up being applied.

I haven't actually thought of a solution to this yet, other than determining that it is pretty much impossible to work around as you can only get a hold of a closed version of the previous package set via python.pkgs. Perhaps looking into the haskell infrastructure might give some directions (haven't looked at it in a while, but the fixpoint stuff looks like it came from there)?

Steps to reproduce

Put the following into ~/.config/nixpkgs/overlays/1.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      one = "definition from 1.nix";
    };
  };
}

and this into ~/.config/nixpkgs/overlays/2.nix

self: super:
rec {
  python27 = super.python27.override {
    packageOverrides = python-self: python-super: {
      two = "definition from 2.nix";
    };
  };
}

Ideally both of these would get applied. This is not the case though as you can see from the following

nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'
error: attribute 'one' missing, at (string):1:28
nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'
"definition from 2.nix"

If you move the 2.nix file aside then you see the override from 1.nix but not 2.nix

mv ~/.config/nixpkgs/overlays/2.nix{,-inactive}
nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.one'
"definition from 1.nix"
nix-instantiate --eval -E 'with import <nixpkgs> { }; pythonPackages.two'
error: attribute 'two' missing, at (string):1:28
@FRidh

This comment has been minimized.

Copy link
Member

commented Aug 4, 2018

Yep, this is a known issue, one which bothers me as well. I think we should pull the creation of the package set out of the interpreters.

As a workaround, keep your overrides as an attribute, something like pythonOverrides. The next overlay can then use composeExtensions.

@FRidh

This comment has been minimized.

Copy link
Member

commented Aug 4, 2018

Related #44196

@twhitehead

This comment has been minimized.

Copy link
Contributor Author

commented Aug 9, 2018

Thanks for the suggestions and links.

If I'm understanding where you are going with your comment, you are saying if the interpreter package pulled in the package set instead of producing it, you could then override the package set in the standard way with overlays.

The interpreter would automatically pick up changes because it would be closed over the final package set. You could suck the final interpreter in the package set if you just re-exported it for compatibility without creating any loops.

Cheers! -Tyson

@FRidh

This comment has been minimized.

Copy link
Member

commented Aug 9, 2018

Exactly that!

@FRidh

This comment has been minimized.

Copy link
Member

commented Aug 9, 2018

if the interpreter package pulled in the package set instead of producing it

But, having the overrides is enough:

overlay_a.nix:

self: super: {
  pythonOverrides = selfPython: superPython: {
    py.overridePythonAttrs (oldAttrs: {
      pname = "foo";
    });
  };
  python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}

overlay_b.nix:

self: super: {
  pythonOverrides = lib.composeExtensions (selfPython: superPython: {
    pytest.overridePythonAttrs (oldAttrs: {
      pname = "foobar";
    });
  }) super.pythonOverrides;
}

(not tested)

@twhitehead

This comment has been minimized.

Copy link
Contributor Author

commented Aug 11, 2018

Thanks. I implemented that and it works well. 👍

Unless you would like to leave it open, feel free to close this issue.

@dotlambda dotlambda referenced this issue Aug 21, 2018

Merged

home-assistant: 0.75.2 -> 0.76.1 #45414

7 of 9 tasks complete
@Shados

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2018

For a more standard-nixos version of said workaround, here's more or less what I'm using:

python-override-setup.nix

{ config, lib, pkgs, ... }:
{
  # Setup for https://github.com/NixOS/nixpkgs/issues/44426 python
  # overrides not being composable...
  nixpkgs.overlays = lib.mkBefore [
    (self: super: let
      pyNames = [
        "python27" "python34" "python35" "python36" "python37"
        "pypy"
      ];
      overriddenPython = name: [
        { inherit name; value = super.${name}.override { packageOverrides = self.pythonOverrides; }; }
        { name = "${name}Packages"; value = super.recurseIntoAttrs self.${name}.pkgs; }
      ];
      overriddenPythons = builtins.concatLists (map overriddenPython pyNames);
    in {
      pythonOverrides = pyself: pysuper: {};
      # The below is just a wrapper for clarity of intent, use like:
      # pythonOverrides = buildPythonOverrides (pyself: pysuper: { ... # overrides }) super.pythonOverrides;
      buildPythonOverrides = newOverrides: currentOverrides: super.lib.composeExtensions newOverrides currentOverrides;
    } // listToAttrs overriddenPythons)
  ];
}

some-module.nix

{ config, lib, pkgs, ...}:
{
  nixpkgs.overlays = [
    (self: super: {
      pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
        some-package = "definition from some-package.nix";
      }) super.pythonOverrides;
    })
  ];
}

another-module.nix

{ config, lib, pkgs, ...}:
{
  nixpkgs.overlays = [
    (self: super: {
      pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
        another-package = "definition from another-package.nix";
      }) super.pythonOverrides;
    })
  ];
}

Some notes

  • This does work, for at least some values of 'work'...
  • The use of mkBefore is to ensure that this is independent of the default order of expression evaluation.
  • I've a sneaking suspicion I've done something terrible somewhere in there, so please do tell if so.

@lopsided98 lopsided98 referenced this issue Nov 10, 2018

Closed

gst_all_1: make extensible #49723

3 of 9 tasks complete
@danbst

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2019

This should be solved with #54266

$ code='with import ./. { overlays = [
    (self: super: {
        pythonPackages.one = "definition from 1.nix";
    })
    (self: super: {
        pythonPackages.two = "definition from 2.nix";
    })
]; }'

$ nix-instantiate --eval -E "$code; pythonPackages.one"
"definition from 1.nix"

$ nix-instantiate --eval -E "$code; pythonPackages.two"
"definition from 2.nix"
@deliciouslytyped

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

I'm not quite sure about this, but .override can be used with a function which takes the previous argument set: .override (a: { stuff = doSomething a.oldStuff}) so I think in theory one could write a function that overrides the previous arguments with a recursive merge? It's been a while but that sounds close to what I did here: https://github.com/deliciouslytyped/nix-ghidra-wip/blob/43b2207729db83f84c0812ff9a054907d5ddc222/packages.nix#L6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.