Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pypy: Remove bootstrap python from closure #88454

merged 1 commit into from May 23, 2020


Copy link

adisbladis commented May 20, 2020

Motivation for this change

Closure size reduction by ~52M.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits

cc @emilazy

@adisbladis adisbladis requested a review from FRidh as a code owner May 20, 2020
@ofborg ofborg bot added the 6.topic: python label May 20, 2020
@ofborg ofborg bot requested a review from andersk May 20, 2020
Copy link

emilazy commented May 20, 2020

FWIW, the dependencies are from shebang lines in various stdlib files that can be executed directly (the ROT-13 module functions stand-alone, the CGI module outputs test data when ran as a script, etc.). There are a lot of these that reference #!/usr/bin/env python3 currently that aren't getting patched out, too. An alternate approach would be patching them correctly; something like:

diff --git a/pkgs/development/interpreters/python/pypy/default.nix b/pkgs/development/interpreters/python/pypy/default.nix
index 888c79b16dc..09a5fd60843 100644
--- a/pkgs/development/interpreters/python/pypy/default.nix
+++ b/pkgs/development/interpreters/python/pypy/default.nix
@@ -53,9 +53,6 @@ in with passthru; stdenv.mkDerivation rec {
   hardeningDisable = optional stdenv.isi686 "pic";
-  # Remove bootstrap python from closure
-  dontPatchShebangs = true;
   C_INCLUDE_PATH = makeSearchPathOutput "dev" "include" buildInputs;
   LIBRARY_PATH = makeLibraryPath buildInputs;
   LD_LIBRARY_PATH = makeLibraryPath (filter (x : x.outPath != or "") buildInputs);
@@ -150,6 +147,13 @@ in with passthru; stdenv.mkDerivation rec {
     cp ${../} $out/lib/${libPrefix}/${sitePackages}/
+  preFixup = ''
+    # Fix shebangs in standard library files to avoid bringing the
+    # bootstrap Python into the package closure.
+    find $out/lib/${libPrefix} -name '*.py' \
+      -exec sed -i "1 s|^#!.*/python[23]\?$|#!$out/bin/${executable}|" {} ';'
+  '';
   inherit passthru;
   enableParallelBuilding = true;  # almost no parallelization without STM

(Untested since I don't feel like staring at a Mandelbrot set for several hours right now...)

Copy link

FRidh commented May 21, 2020

I'm fine with either.

Might also want to add the python3 to disallowedReferences.

@adisbladis adisbladis force-pushed the adisbladis:pypy-closure-size branch from 5adcd4a to 203f382 May 23, 2020
Copy link
Member Author

adisbladis commented May 23, 2020

I feel much better about a simple dontPatchShebangs than trying to manually patch the shebangs correctly, it's not like these scripts are particularly useful to execute directly and if you want to do so I'd recommend using pypy -m ....

@adisbladis adisbladis merged commit 19b2898 into NixOS:master May 23, 2020
1 check was pending
1 check was pending
grahamcofborg-eval Checking new out paths
Copy link

emilazy commented May 23, 2020

Note that all these shebangs are patched correctly using the CPython derivations, and indeed it goes out of its way to achieve this:

  # ...

  preFixup = stdenv.lib.optionalString (stdenv.hostPlatform != stdenv.buildPlatform) ''
    # Ensure patch-shebangs uses shebangs of host interpreter.
    export PATH=${stdenv.lib.makeBinPath [ "$out" bash ]}:$PATH

  # ...

I personally think that leaving unpatched dangling shebang references in an installed package is unfortunate, but it's true that people aren't terribly likely to run these. The pypy -m solution also won't work for running the test CGI script, though that's even more marginal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.