pythonPackages: ensure all derivations provide python modules #194205

FRidh · 2022-10-03T10:25:43Z

This adds a test to ensure no new uses of buildPythonApplication can be added to python-packages.nix.

Python packages can be grouped into two groups: 1) applications and 2) packages providing importable modules. In python-packages.nix we only want to have 2). 1) should be in the top-level package set.

Description of changes

Things done

mweinelt · 2022-10-03T11:13:16Z

Wondering if the eval failure is the result of the one remaining application package: openrazer-daemon. If so, this is pretty hard to glean from the trace.

#194095

FRidh · 2022-10-03T11:32:40Z

Ha, no, this test fails because I forgot about setuphooks. In this case the wrapPython hook:

error: wrapPython should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.

I wonder how to exempt those...

We really need types in Nixpkgs so we can check them and filter based on them.

SuperSandro2000 · 2022-10-03T18:12:59Z

Is there an eval penalty to this? and if so how high would it be?

FRidh · 2022-10-04T08:33:33Z

Is there an eval penalty to this? and if so how high would it be?

Additional eval load is negligible for this.

We really need types in Nixpkgs so we can check them and filter based on them.

Other solutions:

whitelist
mark setup hooks as such and allow hooks

Some packages to consider as well:

python (the interpreter), I think we could use toPythonModule on it.
recursivePthLoader. Should get rid of this.

FRidh · 2022-10-25T13:40:28Z

~~Cross is broken with this change.~~

This adds a test to ensure no new uses of `buildPythonApplication` can be added to `python-packages.nix`. Python packages can be grouped into two groups: 1) applications and 2) packages providing importable modules. In `python-packages.nix` we only want to have 2). 1) should be in the top-level package set. To achieve this, all setup hooks need to be marked as being a setup hook. For the setup hooks in the Python packages set this is done by creating a new builder, `makePythonHook`. Because there were issues with splicing, the file importing all the hooks is converted to an extension. All non-packages were moved out of `python-packages.nix` into `python-packages-base.nix`. The `keep` argument to `makeScopeWithSplicing was cleaned up as well; there is no need to keep this one manually in sync reducing the risk of breaking cross-compilation.

FRidh · 2022-10-26T12:54:54Z

Everything seems to be OK now. I am surprised to see rebuilds though.

FRidh · 2022-10-27T07:02:26Z

Some packages now get a duplicate package in a closure. I guess an incorrect hook is being used.

- Don't call from a different scope - Disable catching of conflicts because we do get a (installation-time only) conflict due to overrides in awscli.

This is the recommended way. It won't actually matter here though.

FRidh · 2022-10-27T07:45:07Z

Bytecode has differences in filenames, but that would be fixed if we would build wheels as intermediate step and use CA then.

andrevmatos · 2022-10-28T14:40:39Z

This seems to have broken neovim-remote on the stdenv call in doCheck.

FRidh · 2022-10-28T17:20:23Z

This seems to have broken neovim-remote on the stdenv call in doCheck.

#198281

nixos-discourse · 2022-11-01T12:24:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/i-cant-update-my-system-due-to-buildpythonpackage/22886/7

fixes NixOS#198319

Artturin · 2022-11-06T23:12:10Z

this broke the fix i did in #196052 (comment)

before 33d12e5:

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/hvb9yxgv1133cfhxxd869sibldvv2vdx-python3-3.10.7.drv» «derivation /nix/store/v880cnh4ml7czmivfbk3cdh93hz9yvbn-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/n8l59iparx98yfw8g5ydqmzmk3fdic75-setuptools-setup-hook.drv» «derivation /nix/store/7vyhynlaxlva1rpvfl0hpdl3qywi9qmc-pip-install-hook.drv» «derivation /nix/store/0l76r5pf69smrl631qa801sjkyb8isk6-python-namespaces-hook.sh.drv» «derivation /nix/store/zipz2sj6ncyllb2z53p0fmsirq601rj7-python-output-dist-hook.drv» ]

after 33d12e5:

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/bhz39ds4v02hn6x4py4mzjyilw4a589h-python3-aarch64-unknown-linux-gnu-3.10.7.drv» «derivation /nix/store/q0pdz5z18bfra7841qwdqq9i3g5na3w7-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/x9b2q51spifxvzsxj90if1h4nb617pys-setuptools-setup-hook.drv» «derivation /nix/store/zsh90vnqf3zblmrlyh8py5hxky2bk78a-pip-install-hook.drv» «derivation /nix/store/a4cyp1fcsnlppzrwwbk1ka5xhbj52hnm-python-namespaces-hook.sh.drv» «derivation /nix/store/2hc97nsdi375cv13014c4lardpxk2mgw-python-output-dist-hook.drv» ]

Artturin · 2022-11-17T21:48:03Z

pkgs/development/interpreters/python/python-packages-base.nix

+  inherit toPythonModule toPythonApplication;
+  inherit buildSetupcfg;
+
+  python = toPythonModule python;


toPythonModule is probably discarding the splicing and that's why the fix is broken

Will test later

`lib.extends hooks pythonPackagesFun` includes python which we want to splice what this fixes: ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/39dkb51rciw6zwg0c2c44gpmpjapddxc-python3-aarch64-unknown-linux-gnu-3.10.12.drv» ``` to ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/21ldw2dp26xvv9iyxn9x77a8yh4waqz5-python3-3.10.12.drv» ``` before NixOS#194205 the keep python used to work by accident because self was passed from __splicedPackages https://github.com/NixOS/nixpkgs/pull/196052/files#diff-44ce3495c4f983ce64dd47c86a9d3e77bad210b2709c098a3806998dcd9b000bR213

`lib.extends hooks pythonPackagesFun` includes python which we want to splice what this fixes: ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/39dkb51rciw6zwg0c2c44gpmpjapddxc-python3-aarch64-unknown-linux-gnu-3.10.12.drv» ``` to ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/21ldw2dp26xvv9iyxn9x77a8yh4waqz5-python3-3.10.12.drv» ``` Before NixOS#194205 The keep python used to work by accident because self was passed from __splicedPackages https://github.com/NixOS/nixpkgs/pull/196052/files#diff-44ce3495c4f983ce64dd47c86a9d3e77bad210b2709c098a3806998dcd9b000bR213 But now it does not work because overrideAttrs is used on python in python-packages-base.nix Preferably we would splice the hooks too but we cannot do that until [Package sets within derivations (i.e. python3.pkgs) are not spliced](NixOS#211340) is fixed, because people often use `python3.pkgs.wrapPython` in `nativeBuildInputs` (it's correct but python3.pkgs should be python3Packages to get splicing.

`lib.extends hooks pythonPackagesFun` includes python which we want to splice what this fixes: ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/39dkb51rciw6zwg0c2c44gpmpjapddxc-python3-aarch64-unknown-linux-gnu-3.10.12.drv» ``` to ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/21ldw2dp26xvv9iyxn9x77a8yh4waqz5-python3-3.10.12.drv» ``` Before NixOS#194205 The keep python used to work by accident because self was passed from __splicedPackages https://github.com/NixOS/nixpkgs/pull/196052/files#diff-44ce3495c4f983ce64dd47c86a9d3e77bad210b2709c098a3806998dcd9b000bR213 But now it does not work because overrideAttrs is used on python in python-packages-base.nix Preferably we would [ splice the hooks too ](NixOS#228139) but we cannot do that until [Package sets within derivations (i.e. python3.pkgs) are not spliced](NixOS#211340) is fixed, because people often use `python3.pkgs.wrapPython` in `nativeBuildInputs` (it's correct but python3.pkgs should be python3Packages to get splicing.

fixes NixOS#198319

FRidh requested a review from mweinelt October 3, 2022 10:25

github-actions bot added the 6.topic: python label Oct 3, 2022

FRidh mentioned this pull request Oct 3, 2022

python3Packages: Use buildPythonPackage within the package set #194100

Merged

13 tasks

FRidh changed the base branch from staging to master October 25, 2022 11:10

FRidh force-pushed the ensure branch from 160feec to 29c6dbd Compare October 25, 2022 13:39

FRidh requested a review from jonringer as a code owner October 25, 2022 13:39

ofborg bot added 8.has: clean-up 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 25, 2022

FRidh force-pushed the ensure branch 4 times, most recently from 7e97f7e to 9566aa4 Compare October 26, 2022 12:01

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 26, 2022

FRidh force-pushed the ensure branch from 9566aa4 to 24dd69b Compare October 26, 2022 12:53

FRidh added 2 commits October 27, 2022 09:36

git-remote-codecommit: fix build

0170e88

- Don't call from a different scope - Disable catching of conflicts because we do get a (installation-time only) conflict due to overrides in awscli.

awscli: override best practice

2f43e21

This is the recommended way. It won't actually matter here though.

ofborg bot requested a review from zaninime October 27, 2022 07:48

FRidh merged commit 0c5c895 into NixOS:master Oct 27, 2022

FRidh deleted the ensure branch October 27, 2022 08:03

gador pushed a commit to gador/nixpkgs that referenced this pull request Nov 4, 2022

platformio: work around fallout from NixOS#194205

a3b7b77

fixes NixOS#198319

Artturin reviewed Nov 17, 2022

View reviewed changes

Artturin mentioned this pull request Apr 14, 2023

beautifulsoup4: fix cross #226121

Merged

12 tasks

Artturin mentioned this pull request Jul 24, 2023

python: don't use keep for hooks, so they will be spliced #228139

Draft

12 tasks

Artturin mentioned this pull request Jul 26, 2023

python3: splice python within python3Packages.callPackage #245475

Merged

12 tasks

rtimush pushed a commit to rtimush/nixpkgs that referenced this pull request Sep 21, 2023

platformio: work around fallout from NixOS#194205

60830f7

fixes NixOS#198319

natsukium mentioned this pull request Jan 24, 2024

python311Packages.recursive-pth-loader: rename from recursivePthLoader #279384

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages: ensure all derivations provide python modules #194205

pythonPackages: ensure all derivations provide python modules #194205

FRidh commented Oct 3, 2022 •

edited

Loading

mweinelt commented Oct 3, 2022 •

edited

Loading

FRidh commented Oct 3, 2022 •

edited

Loading

SuperSandro2000 commented Oct 3, 2022

FRidh commented Oct 4, 2022

FRidh commented Oct 25, 2022 •

edited

Loading

FRidh commented Oct 26, 2022

FRidh commented Oct 27, 2022

FRidh commented Oct 27, 2022

andrevmatos commented Oct 28, 2022

FRidh commented Oct 28, 2022

nixos-discourse commented Nov 1, 2022

Artturin commented Nov 6, 2022

Artturin Nov 17, 2022

pythonPackages: ensure all derivations provide python modules #194205

pythonPackages: ensure all derivations provide python modules #194205

Conversation

FRidh commented Oct 3, 2022 • edited Loading

Description of changes

Things done

mweinelt commented Oct 3, 2022 • edited Loading

FRidh commented Oct 3, 2022 • edited Loading

SuperSandro2000 commented Oct 3, 2022

FRidh commented Oct 4, 2022

FRidh commented Oct 25, 2022 • edited Loading

FRidh commented Oct 26, 2022

FRidh commented Oct 27, 2022

FRidh commented Oct 27, 2022

andrevmatos commented Oct 28, 2022

FRidh commented Oct 28, 2022

nixos-discourse commented Nov 1, 2022

Artturin commented Nov 6, 2022

Artturin Nov 17, 2022

Choose a reason for hiding this comment

FRidh commented Oct 3, 2022 •

edited

Loading

mweinelt commented Oct 3, 2022 •

edited

Loading

FRidh commented Oct 3, 2022 •

edited

Loading

FRidh commented Oct 25, 2022 •

edited

Loading