Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv broken on nixos-19.09 #73254

Closed
zupo opened this issue Nov 11, 2019 · 20 comments
Closed

pipenv broken on nixos-19.09 #73254

zupo opened this issue Nov 11, 2019 · 20 comments

Comments

@zupo
Copy link
Contributor

zupo commented Nov 11, 2019

Describe the bug

I'm trying to pipenv install requests in a nix shell, but pipenv fails with ModuleNotFoundError: No module named 'pip._internal.main'.

To Reproduce
Steps to reproduce the behavior:

  1. I have the following shell.nix, pointing to the most recent commit in nixos-19.09 branch:
{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
    sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}
  1. I run nix-shell --pure --run 'pipenv install requests'
  2. I see pipenv failing:
$ nix-shell --pure --run 'pipenv install requests'
Installing requests…
Adding requests to Pipfile's [packages]…
✔ Installation Succeeded 
Installing dependencies from Pipfile.lock (444a6d)…
An error occurred while installing certifi==2019.9.11 --hash=sha256:e4f3620cfea4f83eedc95b24abd9cd56f3c4b146dd0177e83a21b4eb49e21e50 --hash=sha256:fd7c7c74727ddcf00e9acd26bba8da604ffec95bf1c2144e67aff7a8b50e6cef! Will try again.
An error occurred while installing chardet==3.0.4 --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691! Will try again.
An error occurred while installing idna==2.8 --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c! Will try again.
An error occurred while installing requests==2.22.0 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31! Will try again.
An error occurred while installing urllib3==1.25.6 --hash=sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398 --hash=sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:00
Installing initially failed dependencies…
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:02
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
[pipenv.exceptions.InstallError]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/play/.venv/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]

Expected behavior

If, in my shell.nix, I use the latest commit from master branch, then pipenv works as expected:

-     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
-     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
+     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
+     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
$ nix-shell --pure --run 'pipenv install requests'
Installing requests…
Adding requests to Pipfile's [packages]…
✔ Installation Succeeded 
Installing dependencies from Pipfile.lock (444a6d)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Additional context

MacOS Mojave, nix 2.3.1.

I found the following two PRs on master that touch pkgs/development/tools/pipenv/default.nix:

I tried cherry-picking them on top of nixos-19.09 and release-19.09, but pipenv install requests still fails:

@worldofpeace
Copy link
Contributor

cc @jonringer @FRidh

@jonringer
Copy link
Contributor

I think this was fixed on master, I'll try to find the PR

@jonringer
Copy link
Contributor

try doing `nix-shell --pure --run 'unset PYTHONPATH; pipenv install requests'

@jonringer
Copy link
Contributor

im going to defer to @FRidh on this one, as he's more familiar with pipenv + nixpkgs

@zupo
Copy link
Contributor Author

zupo commented Nov 12, 2019

I think this was fixed on master, I'll try to find the PR

Yes, it is fixed on master, the PRs that fixed it are linked in the issue body above.

@zupo
Copy link
Contributor Author

zupo commented Nov 12, 2019

try doing `nix-shell --pure --run 'unset PYTHONPATH; pipenv install requests'

Fantastic, this does in fact solve the problem!

So the fix is:

@FRidh
Copy link
Member

FRidh commented Nov 12, 2019

I've backported the commits to 19.09 just now.

Typically you should use nix run instead of nix-shell. No need for unsetting PYTHONPATH in that case.

@FRidh FRidh closed this as completed Nov 12, 2019
@zupo
Copy link
Contributor Author

zupo commented Nov 12, 2019

I've backported the commits to 19.09 just now.

Fantastic, thanks!

Typically you should use nix run instead of nix-shell. No need for unsetting PYTHONPATH in that case.

I used nix-shell --run as a minimal example to showcase the bug. I am using nix-shell with direnv to prepare a local Python development environment on my macbook. Should I use nix run instead of nix-shell for this? What documentation should I read?

@FRidh
Copy link
Member

FRidh commented Nov 12, 2019

nix-shell executes setup hooks. You should now for yourself whether for your use case the setup hooks are needed for the packages you load. Given nix-shell resembles a nix-build environment, this is often not what you want when you just want to use those packages for development.

@zupo
Copy link
Contributor Author

zupo commented Nov 12, 2019

I've confirmed the backports work for me by pointing the URL to the most recent commit in https://github.com/NixOS/nixpkgs/commits/release-19.09:

-     url = https://github.com/nixos/nixpkgs/archive/2d896998dc9b1b0daeb8a180dc170733f1225678.tar.gz;
-     sha256 = "1vj3bwljkh55si4qjx52zgw7nfy6mnf324xf1l2i5qffxlh7qxb6";
+     url = https://github.com/nixos/nixpkgs/archive/465e405dc9eda8d5f2b88af9e4555fcb4bfc1874.tar.gz;
+     sha256 = "096crqig0srjq8q6z7655yb46jmjj4zvjn0x1brwls7ibg8vv3gc";

Would it make sense to have a test that would try to run pipenv install <something> after the build is completed so we catch a potential future regression like this one? I can give it a stab.

@FRidh
Copy link
Member

FRidh commented Nov 12, 2019

How would we run such a test? If you want to run it in a nix-build then you need to prefetch dependencies and install from a folder.

@jonringer
Copy link
Contributor

Would it make sense to have a test that would try to run pipenv install after the build is completed so we catch a potential future regression like this one? I can give it a stab.

Unfortunately that can't really be done in a sandbox environment. Would need to prefetch the dependencies like @FRidh states.

You could have a testScript which does the installation steps. But you would need to mention it in the expression, and it would have to be ran manually, not as part of the build. Not particularly elegant, but it's more than nothing.

@zupo
Copy link
Contributor Author

zupo commented Nov 12, 2019

Ah, of course, and the sandbox does not have network access either, so yes, installing requests wouldn't work.

What could potentially work is installing pyjokes (which does not have dependencies) from a local tarball that was put in place before the checkPhase. I don't know if this is possible.

Yet another option is to ship a dummy python package with the pipenv pkg, that we can install locally with pipenv install -e path/to/dummy/package.

@zupo zupo mentioned this issue Nov 13, 2019
10 tasks
FRidh pushed a commit that referenced this issue Nov 13, 2019
This is to prevent regressions such as #73254
by using pipenv to install a simple Python package, thus testing
that pipenv was built correctly.

Many thanks to @infinisil and @domenkozar for hand-holding!
@zupo
Copy link
Contributor Author

zupo commented Nov 20, 2019

I've tried the process from the body of this issue, against latest nixos-19.09 commit, and pipenv install still fails. This is what I did:

$ cd /tmp && mkdir test && cd test
$ cat shell.nix
{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/08e503bac6decf9a3e01b79ab1f9788bf76380b1.tar.gz;
    sha256 = "1xf26fd1ahmz1m3si431xx8q8fs4xkbs6h4cznndgrr3hlafdgvg";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}
$ nix-shell --pure
[nix-shell:/private/tmp/test]$ pipenv install wheel
Creating a Pipfile for this project…
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (fcf4b1)!
Installing dependencies from Pipfile.lock (fcf4b1)…
An error occurred while installing wheel==0.33.6 --hash=sha256:10c9da68765315ed98850f8e048347c3eb06dd81822dc2ab1d4fde9dc9702646 --hash=sha256:f4da1763d3becf2e2cd92a14a7c920f0f00eca30fdde9ea992c836685b9faf28! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/nix/store/czrwm7f44cbivraj0z2ccr06ll5iliyw-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/.local/share/virtualenvs/test-6C_DLGjE/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]
ERROR: ERROR: Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

What am I doing wrong?

@zupo
Copy link
Contributor Author

zupo commented Nov 20, 2019

Same as before, doing the same, but taking the last commit on master, works fine:

-     url = https://github.com/nixos/nixpkgs/archive/08e503bac6decf9a3e01b79ab1f9788bf76380b1.tar.gz;
-     sha256 = "1xf26fd1ahmz1m3si431xx8q8fs4xkbs6h4cznndgrr3hlafdgvg";
+     url = https://github.com/nixos/nixpkgs/archive/0af613569c9056b2fa1c95b4a99dcb41219bb6dd.tar.gz;
+     sha256 = "07g3mlh0vfbz24v1dqwa0b72wkqky5zjp5yvbmy5cflirz12izzs";
$ cd /tmp && mkdir test && cd test
$ cat shell.nix
{ pkgs ? import (builtins.fetchTarball {
    url = https://github.com/nixos/nixpkgs/archive/0af613569c9056b2fa1c95b4a99dcb41219bb6dd.tar.gz;
    sha256 = "07g3mlh0vfbz24v1dqwa0b72wkqky5zjp5yvbmy5cflirz12izzs";
  }) {}
}:
pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [ pkgs.python37Full pkgs.pipenv];
}
$ nix-shell --pure
[nix-shell:/private/tmp/test]$ pipenv install wheel
Creating a Pipfile for this project…
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (fcf4b1)!
Installing dependencies from Pipfile.lock (fcf4b1)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

[nix-shell:/private/tmp/test]$ 

@jonringer
Copy link
Contributor

[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/Users/zupo/.local/share/virtualenvs/test-6C_DLGjE/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]

unfortuantely this just means that it's picking up pip from the environment:

[nix-shell:~/projects/nixpkgs]$ pipenv install wheel
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (fcf4b1)…
An error occurred while installing wheel==0.33.6 --hash=sha256:10c9da68765315ed98850f8e048347c3eb06dd81822dc2ab1d4fde9dc9702646 --hash=sha256:f4da1763d3becf2e2cd92a14a7c920f0f00eca30fdde9ea992c836685b9faf28! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: []
[pipenv.exceptions.InstallError]: ['Traceback (most recent call last):', '  File "/home/jon/.local/share/virtualenvs/nixpkgs-9itlgSZt/bin/pip", line 6, in <module>', '    from pip._internal.main import main', "ModuleNotFoundError: No module named 'pip._internal.main'"]
ERROR: ERROR: Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[nix-shell:~/projects/nixpkgs]$ echo PYTHONPATH
PYTHONPATH

[nix-shell:~/projects/nixpkgs]$ echo $PYTHONPATH
/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages:/nix/store/jvyqrqykg1ln1nwpb5qa9ds12xwbas71-python3.7-certifi-2019.6.16/lib/python3.7/site-packages:/nix/store/4c3z5r6yxsf2cxwwyazhdn92xixn4j5b-python3-3.7.5/lib/python3.7/site-packages:/nix/store/pbsbgky77l7s6sbq32ac2bsh0mqhbw7s-python3.7-setuptools-41.2.0/lib/python3.7/site-packages:/nix/store/mi4khdvzgabzg170an0rjc61cw02sr2f-python3.7-pip-19.2.3/lib/python3.7/site-packages:/nix/store/4rvrlhpfllhfiwczgd0ic0mgfsw86jhf-python3.7-virtualenv-16.4.3/lib/python3.7/site-packages:/nix/store/jyfcmsj081wc1hj6cgwvgpd27h4v6aqn-python3.7-virtualenv-clone-0.5.3/lib/python3.7/site-packages:/nix/store/75h4r36mgdwflk2rx7mq165iyvy8kvq6-pipenv-2018.11.26/lib/python3.7/site-packages:/nix/store/jvyqrqykg1ln1nwpb5qa9ds12xwbas71-python3.7-certifi-2019.6.16/lib/python3.7/site-packages:/nix/store/4c3z5r6yxsf2cxwwyazhdn92xixn4j5b-python3-3.7.5/lib/python3.7/site-packages:/nix/store/pbsbgky77l7s6sbq32ac2bsh0mqhbw7s-python3.7-setuptools-41.2.0/lib/python3.7/site-packages:/nix/store/mi4khdvzgabzg170an0rjc61cw02sr2f-python3.7-pip-19.2.3/lib/python3.7/site-packages:/nix/store/4rvrlhpfllhfiwczgd0ic0mgfsw86jhf-python3.7-virtualenv-16.4.3/lib/python3.7/site-packages:/nix/store/jyfcmsj081wc1hj6cgwvgpd27h4v6aqn-python3.7-virtualenv-clone-0.5.3/lib/python3.7/site-packages

[nix-shell:~/projects/nixpkgs]$ unset PYTHONPATH

[nix-shell:~/projects/nixpkgs]$ pipenv install wheel
Installing wheel…
Adding wheel to Pipfile's [packages]…
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (fcf4b1)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Although, this should have been fixed through backports, re-opening as this is still an issue on release-19.09

@jonringer jonringer reopened this Nov 20, 2019
@FRidh
Copy link
Member

FRidh commented Nov 21, 2019

PYTHONPATH is set because of the setup hook. That's just a consequence of using nix-shell when a buildPython* package is present. People should stop using nix-shell because for most things it is not the tool that should be used.

@FRidh FRidh closed this as completed Nov 21, 2019
@zupo
Copy link
Contributor Author

zupo commented Nov 21, 2019

@FRidh: I'm using https://direnv.net/ with use_nix to build my python project's local development environment. What should I be using instead or what should I be doing differently?

@worldofpeace
Copy link
Contributor

@FRidh: I'm using https://direnv.net/ with use_nix to build my python project's local development environment. What should I be using instead or what should I be doing differently?

I think direnv's use_nix probably uses nix-shell. So you might not be able to use direnv.

@zupo
Copy link
Contributor Author

zupo commented Nov 23, 2019

@zimbatm: sorry to drag you into this, but I wonder if you have any advice for how to use direnv without nix-shell, or if that wouldn't make sense at all?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants