python's pip downloads prebuilt binaries that don't work

[jkozak]

Describe the bug

The python pip utility installs python packages. These can include binary packages. By default where possible these binaries are fetched from a cache provoking the familiar NixOS issues with downloaded binaries.

A fairly easy way to address this would be to make pip default to --no-binaries :all:

Steps To Reproduce

Steps to reproduce the behaviour, using ninja as an example:

/tmp$ nix-shell -p python3Packages.pip python3Packages.setuptools

[nix-shell:/tmp]$ pip install --target pip ninja
Collecting ninja
  Using cached ninja-1.10.2.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
Installing collected packages: ninja
Successfully installed ninja-1.10.2.2
WARNING: You are using pip version 21.0.1; however, version 21.3 is available.
You should consider upgrading via the '/nix/store/32q6ryrb860sksdi1al5djg3pgcpq92l-python3-3.8.11/bin/python3.8 -m pip install --upgrade pip' command.

[nix-shell:/tmp]$ PYTHONPATH=pip pip/bin/ninja --help
Traceback (most recent call last):
  File "pip/bin/ninja", line 8, in <module>
    sys.exit(ninja())
  File "/tmp/pip/ninja/__init__.py", line 51, in ninja
    raise SystemExit(_program('ninja', sys.argv[1:]))
  File "/tmp/pip/ninja/__init__.py", line 47, in _program
    return subprocess.call([os.path.join(BIN_DIR, name)] + args)
  File "/nix/store/32q6ryrb860sksdi1al5djg3pgcpq92l-python3-3.8.11/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/nix/store/32q6ryrb860sksdi1al5djg3pgcpq92l-python3-3.8.11/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/nix/store/32q6ryrb860sksdi1al5djg3pgcpq92l-python3-3.8.11/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip/ninja/data/bin/ninja'

Expected behaviour

ninja should install and work normally. Running the same commands, but adding --no-binary :all: to the pip install command line builds the binaries locally and the result works, as seen

/tmp$ nix-shell -p python3Packages.pip python3Packages.setuptools

[nix-shell:/tmp]$ rm -rf pip

[nix-shell:/tmp]$ pip install --target pip --no-binary :all: ninja
Collecting ninja
  Using cached ninja-1.10.2.2.tar.gz (26 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: ninja
  Building wheel for ninja (PEP 517) ... done
  Created wheel for ninja: filename=ninja-1.10.2.2-cp38-cp38-linux_x86_64.whl size=120038 sha256=5f554cf49e106ec77ea06f96855978b5bfeda829180b138cd9adda017a6c41bc
  Stored in directory: /home/jk/.cache/pip/wheels/97/69/8f/691e6fa272110aae4fe2827310fba229acdf8a9e6b54ee9376
Successfully built ninja
Installing collected packages: ninja
Successfully installed ninja-1.10.2.2
WARNING: You are using pip version 21.0.1; however, version 21.3 is available.
You should consider upgrading via the '/nix/store/32q6ryrb860sksdi1al5djg3pgcpq92l-python3-3.8.11/bin/python3.8 -m pip install --upgrade pip' command.

[nix-shell:/tmp]$ PYTHONPATH=pip pip/bin/ninja
[nix-shell:/tmp]$ PYTHONPATH=pip pip/bin/ninja --help
usage: ninja [options] [targets...]

if targets are unspecified, builds the 'default' target (see manual).

options:
  --version      print ninja version ("1.10.2.git.kitware.jobserver-1")
  -v, --verbose  show all command lines while building

  -C DIR   change to DIR before doing anything else
  -f FILE  specify input build file [default=build.ninja]

  -j N     run N jobs in parallel (0 means infinity) [default=26 on this system]
  -k N     keep going until N jobs fail (0 means infinity) [default=1]
  -l N     do not start new jobs if the load average is greater than N
  -n       dry run (don't run commands but act like they succeeded)

  -d MODE  enable debugging (use '-d list' to list modes)
  -t TOOL  run a subtool (use '-t list' to list subtools)
    terminates toplevel options; further flags are passed to the tool
  -w FLAG  adjust warnings (use '-w list' to list warnings)

Additional context

Add any other context about the problem here.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.151, NixOS, 21.05.3801.83667ff60a8 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(root): `"nixos-21.05.3801.83667ff60a8, nixos-unstable-21.11pre323695.9aeeb7574fb"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[stale]

I marked this as stale due to inactivity. → More info

[MatrixManAtYrService]

Can this be un-stale? it's a real bummer.

[heitorPB]

For --no-binary :all: to work, you need to have the build dependencies available, this can be a compiler with other libraries.

Is there a better workaround for this issue?

[MatrixManAtYrService]

@heitorPB agreed. I've found that in this case adding the following to my flake.nix convinces pre-commit to build ruff, which works.

        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.pre-commit
            pkgs.rustc  # for ruff
            pkgs.cargo  # for ruff
          ];
          shellHook = ''
            export PIP_NO_BINARY="ruff"
            '';
        };
      });

But it seems like it would be much better to somehow get it to use pkgs.ruff instead.

[heitorPB]

Oh, that's a neat trick @MatrixManAtYrService! I will copy that =p I had the exact same issue (astral-sh/ruff#1699).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-there-a-bsd-equivalent-of-coretools/27257/6