buildPython*: store dist (wheel/sdist) in dist output

[FRidh]

Store the intermediate artifacts. In time, we should build, install and test in separate derivations as that reduces circular dependencies, avoids rebuilds when fixing tests, and makes it possible to use the wheels for creating say virtualenv's.

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[FRidh]

Note this will roughly double the amount of data in the cache.

[SuperSandro2000]

There is often an error log in the output like for py:

Check whether the following modules can be imported: iniconfig
pythonOutputDistPhase
Executing pythonOutputDistPhase
Finished executing pythonOutputDistPhase
pythonRemoveBinBytecodePhase
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
pythonOutputDistPhase
Executing pythonOutputDistPhase
Finished executing pythonOutputDistPhase
patching script interpreter paths in /nix/store/kk2xcy6y89586r8qwy80x6snbs41x05z-python3.10-py-1.11.0
checking for references to /build/ in /nix/store/kk2xcy6y89586r8qwy80x6snbs41x05z-python3.10-py-1.11.0...
find: ‘/nix/store/hp1fdmx4ff9irrysasl2q8phxiz9w897-python3.10-py-1.11.0-dist’: No such file or directory
strip is /nix/store/n95cd4q1dqzdvsiy1hmrkx9shwi3n4sh-gcc-wrapper-11.3.0/bin/strip
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
pythonCatchConflictsPhase
pythonRemoveBinBytecodePhase
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: py
pythonOutputDistPhase
Executing pythonOutputDistPhase
Finished executing pythonOutputDistPhase

and the build fails because two of the ofborg evaluators run out of inodes:

error: creating file '/nix/store/b9l6z3w0rnlfgs8adc5imgg6vwdbz4rl-source/tls/s2n_client_cert.c': No space left on device

@ofbrog eval

[FRidh]

Yes something expects the output to be built before our phase is executed. Moved to preFixupPhases.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-build-python-package-for-use-in-non-nix-environments/21587/2

[jonringer]

I'm late, but I think this is a good first step in getting a source -> dist -> package workflow going.

[trofi]

Bisect claims that adbc59c buildPython*: store dist (wheel/sdist) in dist output dendered cvise and python3Packages.scipy to fail builds in staging as:

@nix { "action": "setPhase", "phase": "installCheckPhase" }
running install tests
/nix/store/hlihc9dxb0rk56ln3k7f2394h6jb3ayv-stdenv-linux/setup: line 1399: pushd: dist: No such file or directory

[FRidh]

yeah we should pushd to another folder in that package

[FRidh]

Fix in #191327

[trofi]

I still see the same cvise build failure on staging and on python-updates:

$ nix build --no-link -f. cvise
...
@nix { "action": "setPhase", "phase": "pythonOutputDistPhase" }
pythonOutputDistPhase
Executing pythonOutputDistPhase
mv: cannot stat 'dist': No such file or directory

Not sure if I'm looking at the wrong branch or the failure is expected.

[FRidh]

cvise needs fixing, probably needs format="other"; and all these dont* removed

[trofi]

Thank you! Proposed possible fix as #191461

[trofi]

cvise was already fixed in 370cce6

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-is-buildpythonpackage-passthru-not-in-my-result-folder/21852/4