Shebang length exceeded in pip executable

[ajgillis]

On Linux, the pip executable is a shell script. At the top of this pip shell script is a shebang (#!) line that specifies the absolute path to the python interpreter.

If the absolute path to the python interpreter in the is very long (deeply nested and/or large path names), it may exceed the maximum length allowed for a shebang line. When the shebang length limit is exceeded, pip fails with the error: "bad interpreter: No such file or directory"

The maximum length for a shebang line is limited in the kernel by BINPRM_BUF_SIZE, set in /usr/include/linux/binfmts.h. On the Linux machines I looked at, this limit is set to 128.

A work-around for this problem is to not run pip directly, but run the python interpreter and pass the pip script to python as the source to execute, or run the pip library module (python -m pip).

It would be nice if pip was not affected by this limit. Usually this happens when creating a virtualenv with a long absolute path, but it could also happen with non-standard python installation path

[timgraham]

I hit this on some Jenkins jobs when the build name + params were long enough to create paths long enough trigger it. Thanks for the nice explanation.

[piotr-dobrogost]

Another way is to run pip with python -m pip.

[Yasumoto]

Adding another +1 as a teammate hit this today as well.

(Not sure on how to improve this, so will defer to others will better ideas :)

[ajgillis]

@piotr-dobrogost thanks, edited original comment to say that explicitly.

[barraponto]

Yeah, but keep in mind you've got to call the right python when calling python -m pip from within a virtualenv.

[EricFalkenberg]

It's possible to write a wrapper for said shell script that will temporarily make the shebang max length longer.

#!/bin/bash
script="$1" 
shebang=$(head -1 "$script")
interp=( ${shebang#\#!} )        # use an array in case a argument is there too
# now run it
exec "${interp[@]}" "$script"

The only other course of action would be to edit binfmts.h and recompile your kernel which is a bit silly.

[dcampbe]

Within a virtualenv you could do these to patch the shebangs from bash.

sed -i "1s|.*|#!/usr/bin/env python|" `which pip`
sed -i "1s|.*|#!/usr/bin/env python|" `which easy_install`

Might be able to script this to apply to all python shebanged files in bin.

[zjffdu]

Hit this issue recently and found this ticket, +1 for fixing it

[barraponto]

suggestion: instead of creating a sh wrapper for pip, just create an alias for python -m pip in the activate script.

[larsxschneider]

As a word around you could create a hardlink from your nested job directory to /tmp/[randomid] and then work on there, no?

[ajgillis]

Often /tmp/ is on a separate partition, so a hardlink will not work in that case. A symlink should work.

From: Lars Schneider [mailto:notifications@github.com]
Sent: Thursday, February 18, 2016 5:15 AM
To: pypa/pip
Cc: Gillis, Andrew
Subject: Re: [pip] Shebang length exceeded in pip executable (#1773)

As a word around you could create a hardlink from your nested job directory to /tmp/[randomid] and then work on there, no?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1773#issuecomment-185712522.

Spirent Communications e-mail confidentiality.

This e-mail contains confidential and / or privileged information belonging to Spirent Communications plc, its affiliates and / or subsidiaries. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution and / or the taking of any action based upon reliance on the contents of this transmission is strictly forbidden. If you have received this message in error please notify the sender by return e-mail and delete it from your system.

Spirent Communications plc
Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.
Tel No. +44 (0) 1293 767676
Fax No. +44 (0) 1293 767677

Registered in England Number 470893
Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United Kingdom.

Or if within the US,

Spirent Communications,
27349 Agoura Road, Calabasas, CA, 91301, USA.
Tel No. 1-818-676- 2300

[ssbarnea]

Are there any plans on fixing this issue? I guess more and more people will hit this ugly bug.

Pleaser remember that Jenkins is migrating to pipelines and that git flow generates really long branch names. This means that if you use git flow and trying to build all branches using Jenkins, you will reach this bug as soon you start to create the feature/.... branches.

[dstufft]

This is a limitation of the OS. The Linux kernel has a limit on how long a hashbang can be before it just stops reading them.

[ssbarnea]

This is a well known limitation and not a recently introduced bug with probable zero chances of being changes in the foreseeable future.

This makes me think that @EricFalkenberg workaround could be implemented as the default behaviour of pip.

For the moment we managed to avoid this by calling the modules directly but as we all know not all python applications are ready to be used like this.

[andyfeller]

It's apparent this is a recurring issue and deserving of some kind of action. At the very least this scenario should be documented within the pip user guide as an alternative of using pip install in the cases where scripts' interpreters/hashbangs are too long.

I've spent more time than reasonable stumbling around this problem just to run into this issue by happenstance.

[ssbarnea]

This is related to pypa/virtualenv#596

[tundratim]

I am looking into submitting a patch to Linux to make the max shebang line a lot bigger. We'll see if the Linux devs accept it. The 128 char limit is ancient.

[ssbarnea]

While I have nothing against changing this limit you should be aware that even if is accepted on a fast track you will not expect it to reach 50% of the userbase in the next 5-10 years. People do not upgrade servers the same way they upgrade they iOS devices ;)

[tundratim]

@ssbarnea I am well aware of this, but I'm not terribly worried about the existing base (which has had to solve this problem one way or the other). The concern I have is forward looking. With all the automation being used today (often in a really sloppy way), shebangs longer than 128 are starting to be a problem.

[ssbarnea]

@tundratim Be sure you post links to the threads, I will do my best to support these changes. You are right that we will see more and more occurrences of this, and the use of virtual environments is probably the primary source for these.

[domenkozar]

In Nix, we use following workaround to do the parsing in user space: https://github.com/shlevy/long-shebang

[andyfeller]

Long term fixes and work arounds aside, is there any reason why this scenario shouldn't be explicitly documented in the pop user guide?

Sent from my iPhone

On Aug 4, 2016, at 7:05 AM, Domen Kožar notifications@github.com wrote:

In Nix, we use following workaround to do the parsing in user space: https://github.com/shlevy/long-shebang

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[pfmoore]

@andyfeller No real reason, other than I'm not sure what we'd say (without making it sound like a much bigger issue than it is). Best approach would be for someone to raise a PR, and then we can thrash out the details there.

FWIW, personally, I'd like to see it being clear that:

1. This only affects Unix systems (and not all of those)
2. That it's a system limitation rather than a pip bug
3. That it's only going to happen when Python is on a long path (virtualenvs being the most likely scenario)
4. That it's easy to work around by using python -m pip

[andyfeller]

I wholeheartedly agree on all points. I just want to help others not waste time and get frustrated over a likely hurdle. Thanks!

Sent from my iPhone

On Aug 4, 2016, at 8:39 AM, Paul Moore notifications@github.com wrote:

@andyfeller No real reason, other than I'm not sure what we'd say (without making it sound like a much bigger issue than it is). Best approach would be for someone to raise a PR, and then we can thrash out the details there.

FWIW, personally, I'd like to see it being clear that:

This only affects Unix systems (and not all of those)
That it's a system limitation rather than a pip bug
That it's only going to happen when Python is on a long path (virtualenvs being the most likely scenario)
That it's easy to work around by using python -m pip
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[lukeschlather]

In virtualenv, creating your virtualenvs with the --relocatable flag helps with this issue, since part of what it does is change the shebangs to use /usr/bin/env python.

The main issue with relocatable seems to be that it needs to be re-run every time you install a new package in the virtualenv. If pip/virtualenv could be modified so that --relocatable is more stable/ the default behavior, that might solve this (or at least the most common case.)

[kamikaze]

got this problem on jenkins servers

[RikHeijdens]

I also ran into this issue on Jenkins with virtualenv. Running pip as module does the trick: python -m pip ....

[tundratim]

On 11/10/2016 05:41 PM, Rik Heijdens wrote:

I also ran into this issue on Jenkins with virtualenv. Running pip as module does the trick: |python -m pip ...|.

—
You are receiving this because you were mentioned.

Excellent tip, thanks.

---

Tim Daneliuk tundra@tundraware.com
PGP Key: http://www.tundraware.com/PGP/

[tgamblin]

FYI, in Spack we have since March been using a similar trick to what the nix guys came up with, called sbang. It's is written in pure bash and doesn't require compilation. On install, we patch all overlong shebangs to look like this:

#!/bin/bash /path/to/sbang
#!/long/path/to/real/interpreter with arguments
... rest of script ...

This requires sbang to be installed in a suitably shallow directory. We just install it with the package manager (Spack) so it is always at the top of a deep hierarchy.

I'm not sure about the best way to apply this type of workaround to the pip / virtualenv case. Seems like you would want virtualenv or pip to depend on sbang, so that sbang is installed somewhere shallow. Then you would want virtualenv to know that it's creating a deep virtual environment, patch the various scripts in its bin directory, and let the virtualenv's pip know about the shallow sbang, so it can patch scripts to use the shallow sbang with the deep interpreter. You can't just put sbang in the virtualenv, as then it would be too deep.

The approach above would work, but it may not be so easy, considering virtualenv doesn't seem to want to fix this (see #596). I don't see how you'd solve this with pip alone.

[ngaya-ll]

This is not just a Linux issue; Mac OS also imposes a limit of 512 characters.

[pradyunsg]

@pfmoore @dstufft This would be a setuptools + virtualenv + os issue, correct?

AFAICT, there's nothing that pip can really do about this; at best we can document the workaround in a section in the User Guide. If that's what we want to do, I'll happily make a PR for the docs.

[dstufft]

There are some possible work arounds, like using a redirector binstub or something of the sort.

[pradyunsg]

In Nix, we use following workaround to do the parsing in user space: https://github.com/shlevy/long-shebang

@dstufft Something along these lines?

[dstufft]

Yea. Not sure how that'd actually work out yet or if it would be acceptable fix but we should at least explore it first.

[pfmoore]

As I noted above, we use distlib to write our wrappers, and hence it's an issue that should probably be fixed in distlib - in fact, I believe there have been some changes related to this in distlib.

IMO, unless someone can find code in pip itself that writes out the shebang, this issue should be closed in favour of an issue on distlib (if the problem hasn't already been fixed there). However, I don't have the time at the moment to confirm this, I'm afraid.

@dstufft do you know of a reason why this would be something to fix in pip rather than distlib?

[pradyunsg]

IMO, unless someone can find code in pip itself that writes out the shebang

In pip/wheel.py:72.

I believe there have been some changes related to this in distlib.

Last month this happened.

---

This probably means pip just needs to switch to using ScriptMaker from a newer distlib?

[pradyunsg]

Okay, that previous comment was preliminary.

IIUC, pip doesn't write out the scripts on it's own, it does delegate that part to ScriptMaker. pip would just need to upgrade the vendored distlib, when it's released with the fix for long shebangs, which is already there in the master branch of distlib.

@pfmoore Can this issue be closed or do we want to track the fix in distlib being released and vendored by keeping this issue open?

[pfmoore]

IMO we can close this issue.