buildPythonPackage: recompile bytecode for reproducibility

[FRidh]

Due to a change in pip the unpacked wheels are no longer reproducible.
We recompile the bytecode to cleanup this error.
#81441

Motivation for this change

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[jonringer]

@GrahamcOfBorg build python3Packages.requests
@GrahamcOfBorg build azure-cli

[misuzu]

pycrypto is not building:

$ nix-build . -A pkgs.python37Packages.pycrypto
these derivations will be built:
  /nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv
building '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv'...
Sourcing python-recompile-bytecode-hook.sh
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating pycrypto.egg-info
writing pycrypto.egg-info/PKG-INFO
writing dependency_links to pycrypto.egg-info/dependency_links.txt
writing requirements to pycrypto.egg-info/requires.txt
writing top-level names to pycrypto.egg-info/top_level.txt
writing manifest file 'pycrypto.egg-info/SOURCES.txt'
reading manifest file 'pycrypto.egg-info/SOURCES.txt'
writing manifest file 'pycrypto.egg-info/SOURCES.txt'
Copying pycrypto.egg-info to build/bdist.linux-x86_64/wheel/pycrypto-3.9.7-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/pycrypto-3.9.7.dist-info/WHEEL
creating 'dist/pycrypto-3.9.7-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pycrypto-3.9.7.dist-info/METADATA'
adding 'pycrypto-3.9.7.dist-info/WHEEL'
adding 'pycrypto-3.9.7.dist-info/top_level.txt'
adding 'pycrypto-3.9.7.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Finished executing setuptoolsBuildPhase
installing
Executing pipInstallPhase
/build/dist /build
Processing ./pycrypto-3.9.7-py3-none-any.whl
Requirement already satisfied: pycryptodome in /nix/store/6qjnk13bsxhcqgbqw1qsz1y0l8cxh5bv-python3.7-pycryptodome-3.9.7/lib/python3.7/site-packages (from pycrypto==3.9.7) (3.9.7)
Installing collected packages: pycrypto
Successfully installed pycrypto-3.9.7
/build
Finished executing pipInstallPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7
strip is /nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7/lib
patching script interpreter paths in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7
checking for references to /build/ in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
pythonRecompileBytecodePhase
*** Error compiling '/nix/store/6xmiwapzvgx4a444b9qaz5ldfv2jqw1c-python3-3.7.7/lib/python3.7/site-packages/sitecustomize.py'...
PermissionError: [Errno 13] Permission denied: '/nix/store/6xmiwapzvgx4a444b9qaz5ldfv2jqw1c-python3-3.7.7/lib/python3.7/site-packages/__pycache__/sitecustomize.cpython-37.opt-2.pyc.140737341708784'
builder for '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv' failed with exit code 123
error: build of '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv' failed

[jonringer]

maybe exclude the top level of site-packages?

[jonringer]

actually, figured out root cause, some packages don't ship any .py's, and in that case, it errors out

[FRidh]

Ah yes, pycrypto, my dummy package providing pycryptodome. Would be nice if we could finally get rid of it.

[FRidh]

Thanks for the feedback, it's building now.

[FRidh]

Oh, now we have another failure

$ nix-build -A python2.pkgs.setuptools
...
option -j not recognized
usage: python compileall.py [-l] [-f] [-q] [-d destdir] [-x regexp] [-i list] [directory|file ...]

arguments: zero or more file and directory names to compile; if no arguments given,
           defaults to the equivalent of -l sys.path

options:
-l: don't recurse into subdirectories
-f: force rebuild even if timestamps are up-to-date
-q: output only error messages
-d destdir: directory to prepend to file paths for use in compile-time tracebacks and in
            runtime tracebacks in cases where the source file is unavailable
-x regexp: skip files matching the regular expression regexp; the regexp is searched for
           in the full path of each file considered for compilation
-i file: add all the files and directories listed in file to the list considered for
         compilation; if "-", names are read from stdin
builder for '/nix/store/9hrz8wjl5ij1psq4b47yj0gbc4w3p3wm-python2.7-setuptools-44.0.0.drv' failed with exit code 1

Seems like Python 2 version does not have -j

[FRidh]

gtk-doc fails with

pythonRecompileBytecodePhase
find: ‘/nix/store/i5cla9rwy1kaakwy7k4vdjgrlw6qawrj-gtk-doc-1.32/lib/python3.8/site-packages’: No such file or directory
builder for '/nix/store/7clc8qpva22qcq06w0fw9hiccicfziiv-gtk-doc-1.32.drv' failed with exit code 1

https://hydra.nixos.org/build/122153227

For now I'll pass

[FRidh]

Issue tracking regressions #90516.

[FRidh]

@adisbladis @DavHau you may want to pass dontUsePythonRecompileBytecode = true; in poetry2nix and mach-nix. I don't expect trouble with Python 3 but there are already some Python 2 regressions. Depending on how many there are it may be disabled altogether for Python 2.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-builds-keep-pyc-in-store-dockertools/8784/6