Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long env path names cause weird errors after activiting virtual environment #997

Closed
HaraldNordgren opened this issue Jan 5, 2017 · 20 comments · Fixed by #1579
Closed

Long env path names cause weird errors after activiting virtual environment #997

HaraldNordgren opened this issue Jan 5, 2017 · 20 comments · Fixed by #1579

Comments

@HaraldNordgren
Copy link

@HaraldNordgren HaraldNordgren commented Jan 5, 2017

After creating a virtualenv in the path

/home/harald/git-repos/sandbox/stackoverflow/41454203-how-do-you-import-the-library-being-tested-when-running-a-nosetest/env

I could not run the env's pip,

nosetest/env$ ./bin/pip
bash: ./bin/pip: /home/harald/git-repos/sandbox/stackoverflow/41454203-how-do-you-import-the-l: bad interpreter: Permission denied

and running just pip gives me the system's pip, with all of its packages.

@yan12125
Copy link

@yan12125 yan12125 commented Jan 5, 2017

Not virtualenv's fault. Shebangs have length limits, see http://www.in-ulm.de/~mascheck/various/shebang/.

This is basically a duplicate of #994 - virtualenv are broken with fragile kernel shebang parsing.

@HaraldNordgren
Copy link
Author

@HaraldNordgren HaraldNordgren commented Jan 5, 2017

Where is the shebang written to file?

And is there maybe some workaround we could employ to make virtual environments more sturdy?

@yan12125
Copy link

@yan12125 yan12125 commented Jan 5, 2017

Where is the shebang written to file?

Do you mean the location of the shebang or the codes that generate a shebang? 1. In the first line of the file bin/pip 2. In distlib

And is there maybe some workaround we could employ to make virtual environments more sturdy?

I don't think there are simple workarounds. A possibility is writing a user space shebang parser just like py.exe on Windows.

@HaraldNordgren
Copy link
Author

@HaraldNordgren HaraldNordgren commented Jan 5, 2017

Using pip as an example, one thing we could do is move the current pip to .pip, remove the shebang line and create a new pip file with the following content:

#!/bin/bash
/path/to/env/bin/python /path/to/env/bin/.pip $@

Basically, wrap each program in a shell script calling the original file, passing on all of its arguments.

That will get around the problem of long shebangs.

And if we quote these paths in Bash we could probably solve the problem with space characters in paths as well.

@yan12125
Copy link

@yan12125 yan12125 commented Jan 6, 2017

OK maybe my definition for "workarounds" is too strict. I thought it shouldn't break existing things. In this case, using /bin/bash usually works but not always. Some systems do not install bash in /bin/ (FreeBSD install it at /usr/local/bin/bash), and some systems do not provide bash at all. (Most Android uses /system/bin/sh)

A similar idea was proposed 3 years ago: pypa/pip#923 (comment). Apparently no PyPA core developers are interested in this approach :(

@HaraldNordgren
Copy link
Author

@HaraldNordgren HaraldNordgren commented Jan 6, 2017

No, you're right. The workaround shouldn't break things. As long as env lives in the same place in every Unix system, this should work just as well:

#!/usr/bin/env sh
"/path/to/env/bin/python" "/path/to/env/bin/.pip" $@

It's nice to see that someone else brought up the same idea. However, @cellularmitosis phrased it in negative terms back then, so it only makes sense that no one took it seriously. It doesn't seem like any of the developers actually commented on it.

Let us give it a positive spin this time around and present it as waterproof solution to long paths and spaces. I'm not exactly sure how I would go about to putting this in the virtualenv source code though.

@yan12125
Copy link

@yan12125 yan12125 commented Jan 6, 2017

Looks better. If it's possible to use a single file, I believe it will be ready for a pull request!

@HaraldNordgren
Copy link
Author

@HaraldNordgren HaraldNordgren commented Jan 9, 2017

@yan12125 I created a pull request.

@vsajip
Copy link

@vsajip vsajip commented May 28, 2017

I've updated distlib to handle long paths and paths with spaces. I didn't use Harald Nordgren's patch directly (it had some problems - e.g. no tests) but the approach is the same - using '/bin/sh' as the executable. pip / virtualenv maintainers might want to test after locally vendoring the current version of the distlib repository.

@cellularmitosis
Copy link

@cellularmitosis cellularmitosis commented May 28, 2017

@HaraldNordgren ah, I should have been clearer that my dismay was directed at Apple, not at the virtualenv guys! On Linux, this isn't an issue at all, and I had assumed this was an insurmountable shortcoming of OS X shebang parsing. Glad to see there's a solution!

@Imaskar
Copy link

@Imaskar Imaskar commented Feb 11, 2020

Is it really fixed?
I just run into the same issue using latest pip 20.0.2 and virtualenv 20.0.1 and python 3.7.4 . And it still generates the long path.

@gaborbernat gaborbernat reopened this Feb 11, 2020
@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 11, 2020

It seems here we've regressed as we're not using distlib to generate the paths.

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Feb 11, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-2-2020-02-11) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

@Imaskar
Copy link

@Imaskar Imaskar commented Feb 13, 2020

Thank you for acting quick!
I tried 20.0.2 and it worked. It generates full shebang on macOS which supports long paths, and short on Linux.

@MartinKosicky
Copy link

@MartinKosicky MartinKosicky commented Apr 1, 2020

Well instead of pip... you can use python -m pip. That helped me

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Apr 1, 2020

@MartinKosicky you can, but should no longer be needed.

@MartinKosicky
Copy link

@MartinKosicky MartinKosicky commented Apr 1, 2020

Ok I am a bit confused. I am using venv probabbly and that is why I can't see this fix. This is virtualenv, which is something else?

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Apr 1, 2020

Yeah, we're very much different from venv. One advantage of using virtualenv is that we can fix issues faster than venv. There's more detail on differences here #1366 (comment)

@MartinKosicky
Copy link

@MartinKosicky MartinKosicky commented Apr 1, 2020

very hard to google something about virtualenv because it always gives me venv... even the standard python pages call it virtualenv, but the module is called venv :D

@gaborbernat
Copy link
Contributor

@gaborbernat gaborbernat commented Apr 1, 2020

IMHO it should be calling it virtual environments, not virtualenv. If you find virtualenv in venv doc then probably open an issue under http://bugs.python.org/

@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
7 participants