Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shebang length exceeded in pip executable #596

Closed
ajgillis opened this issue Apr 25, 2014 · 16 comments

Comments

Projects
None yet
10 participants
@ajgillis
Copy link

commented Apr 25, 2014

When a new virtualenv is created, there are a number of things installed into that virtualenv. One of these things is the python package installer, pip.

On Linux, the pip executable is a shell script. At the top of this pip shell script is a shebang (#!) line that specifies the python interpreter in the virtualenv. This line contains the absolute path to the python interpreter.

If the absolute path to the python interpreter in the virtualenv is very long (deeply nested and/or large path names), it may exceed the maximum length allowed for a shebang line.

The maximum length for a shebang line is limited in the kernel by BINPRM_BUF_SIZE, set in /usr/include/linux/binfmts.h. On the Linux machines I looked at, this limit is set to 128.

So, when the path, to where a python virtualenv is created, becomes too long, using that virtualenv's pip will not work. It will fail with the error: "bad interpreter: No such file or directory"

A work-around for this problem is to not run pip directly, but by run the virtualenv's python interpreter and pass the pip script to python as the source to execute.

@d6e

This comment has been minimized.

Copy link

commented Oct 13, 2014

I am also experiencing this issue.

@sholsapp

This comment has been minimized.

Copy link

commented Nov 21, 2014

Me too.

@sblask

This comment has been minimized.

Copy link

commented Feb 9, 2015

Thanks for the workaround!

@pfmoore

This comment has been minimized.

Copy link
Member

commented Feb 9, 2015

Note that the wrapper scripts are actually generated by setuptools (if you install from sdist) or distlib (if you install from wheel). So any fix for this issue should really be requested from those projects.

But if it's an OS limitation, maybe there simply isn't a viable fix? I seem to remember that at one time, Perl used a magic incantation to execute scripts (googles a bit...) Yes, something like this (translated for Python)

#!/bin/sh
eval 'exec /the/long/interpreter/path/to/python $0 ${1+"$@"}'

You'd need some extra stuff so that Python didn't try to execute the eval-exec line, but that might work.

If anyone wanted to propose something like that as a feature request for setuptools and distlib, that'd be great.

@b-long

This comment has been minimized.

Copy link

commented Sep 28, 2015

So, when the path, to where a python virtualenv is created, becomes too long, using that virtualenv's pip will not work.

It would be really nice if running virtualenv on a system in which this problem occurs, the command would output some warning (or even an error) that being in a directory with such a long path name is essentially not supported by the defaults at the moment.

@jantman

This comment has been minimized.

Copy link

commented Oct 22, 2015

I'm hitting this also. It's specifically a problem for me as I'm running pip from within Jenkins builds, which have very long paths to the pwd. Interestingly, I have a number of identically-built build slaves (all have BINPRM_BUF_SIZE set to 128, and the same pip and python and virtualenv versions) and I'm experiencing this on some of them but not others, even with varying path lengths.

I like @b-long's suggestion; if virtualenv is going to create a wrapper that doesn't work on the current system, it should at least give the user a warning, if not flat-out fail.

@pfmoore

This comment has been minimized.

Copy link
Member

commented Oct 22, 2015

It's not clear how virtualenv could detect this situation. We can't exactly use a value from a C header, and I don't know how we'd detect we're on a system with this limitation (Windows doesn't have it, does OSX?)

I'm inclined to close this issue as an OS limitation, not a virtualenv problem.

@jantman

This comment has been minimized.

Copy link

commented Oct 22, 2015

For what it's worth, some thoughts...

  1. can ctypes do this?
  2. How about something as simple as:
# on Linux, BINPRM_BUF_SIZE == 128, giving us a maximum shebang length of 127
# 2 bytes for #!, 11 bytes for '/bin/python' leaves us with 114
if sys.platform() == 'Linux' and len(home_dir) > 114:
    log.warn("bin/activate may not work your system, as the length of the shebang line will exceed 128 characters")
@pfmoore

This comment has been minimized.

Copy link
Member

commented Oct 22, 2015

  1. No idea, sorry.
  2. I'm a Windows user so I have no opinion. If you think this is reasonable, I suggest raising a PR and seeing what the Linux people think...
@b-long

This comment has been minimized.

Copy link

commented Oct 22, 2015

I'm sure other users would also appreciate the change I suggested. Can we leave this open for comment and to inspire a PR? Thanks @jantman and @pfmoore 😄

@jantman

This comment has been minimized.

Copy link

commented Oct 22, 2015

BINPRM_BUF_SIZE is defined in a kernel header, and seems to only be used by a small handful of kernel functions. I don't believe there's any way to programmatically detect this (short of reading in the headers, which isn't anywhere near feasible).

That being said, I think it's reasonable to assume that this won't be changing any time soon (there's a quick overview of the maximum shebang length here). The current length is also clearly given in the "Notes" section of the execve(2) man page, where it states, "A maximum line length of 127 characters is allowed for the first line in a #! executable shell script."

I would assume that all POSIX-compliant operating systems have a limit of some sort, though it seems that some of them, specifically the BSD variants, may have limits upwards of 8,000 characters.

While it seems like it would be a reasonable solution to simply hard-code something as naive as my above suggestion (if the platform is Linux and the line is longer than 127 characters), it feels overly crude and specific and inflexible to me.

I'm still thinking about how I'd programmatically test this in a way that wouldn't add too much overhead to virtualenv (granted, we'd only need to test this at venv creation time, so I'd assume some overhead might be acceptable).

I have some simple proof-of-concept code to test whether or not a given path is an acceptable she-bang length, but it's rather ugly as it writes out a file to disk and then executes the file to capture the output. The example and output is here: https://gist.github.com/jantman/ba39f98936643bc948bd

I'm going to have to call it a night, but I'll try to come back to this. I'd certainly appreciate the input of any other Linux users, or users of other OSes who can confirm whether or not they have a similar limit.

@johntyree

This comment has been minimized.

Copy link

commented Feb 16, 2016

Instead of trying to do something tricky like make assumptions about the limit, why not just make a functional test bash script? If python correctly gets invoked then everything is peachy, otherwise it isn't.

@pfmoore

This comment has been minimized.

Copy link
Member

commented Feb 16, 2016

To reiterate:

  1. Please submit PRs to setuptools and distlib if you want this fixed - the code is not in virtualenv or pip.
  2. To be acceptable, code would need to work on all supported platforms, including OSX, BSD, ... and not just Linux (platform-specific conditionals are acceptable, of course, if there's no other way)

Closing this issue, as it isn't a virtualenv problem (you can get the same effect by installing a full Python interpreter to a long directory name)..

@pfmoore pfmoore closed this Feb 16, 2016

openstack-gerrit added a commit to openstack/openstack that referenced this issue Sep 1, 2016

Updated openstack/openstack
Project: openstack-infra/project-config  3ae6ad913b838617593ca67b670b4af1c0708165

Shorten job names for OSA os_keystone repo

Due to the length limit for the shabeng line [1] the jobs
for the OpenStack-Ansible keystone role are failing.

This patch shortens the names appropriately.

[1] pypa/virtualenv#596

Change-Id: I11d39d786a01a5519e58752962e578169f667c7b

openstack-gerrit pushed a commit to openstack/project-config that referenced this issue Sep 1, 2016

Shorten job names for OSA os_keystone repo
Due to the length limit for the shabeng line [1] the jobs
for the OpenStack-Ansible keystone role are failing.

This patch shortens the names appropriately.

[1] pypa/virtualenv#596

Change-Id: I11d39d786a01a5519e58752962e578169f667c7b

openstack-gerrit pushed a commit to openstack/openstack-ansible-os_keystone that referenced this issue Sep 1, 2016

Shorten tox target names
Due to the length limit for the shabeng line [1] the execution of
these tox targets in OpenStack-CI is failing (the full shebang
length is 130 chars).

This patch shortens the names appropriately.

[1] pypa/virtualenv#596

Change-Id: I9011eac714e40d33baff7c1a1fc6eb0fdf47df55

rdoproject pushed a commit to rdo-infra/review.rdoproject.org-config that referenced this issue May 31, 2017

Shorten the name of the integration job for rdo-container-registry
Because we're otherwise hitting the upper limit for amount of characters
and it makes tox unable to find the python things.
pypa/virtualenv#596

Change-Id: Ic1e6b3a55c17f8a2ba41f29e63fc3f0bea5e5c49
@dotchev

This comment has been minimized.

Copy link

commented Dec 22, 2017

We just hit this issue.

Why not use the python from the PATH? virtualenv already prepends it with the bin dir.

#!/usr/bin/env python
@pfmoore

This comment has been minimized.

Copy link
Member

commented Dec 22, 2017

virtualenv already prepends it with the bin dir

Only if you activate the virtualenv. Virtualenv supports being used without activation.

gabsn added a commit to DataDog/dd-trace-py that referenced this issue Dec 22, 2017

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-salt that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I5057111822b6915ed0c7109db64181a2952d0fcf

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-influxdb that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: Icd6ac120246086aa8f9d52600d45ff249f2a9327

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-nginx that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I2f204e9f9e095dc12b0f35c0b5acae7c540ba8b5

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-heka that referenced this issue Feb 8, 2018

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-kubernetes that referenced this issue Feb 8, 2018

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-kubernetes that referenced this issue Feb 8, 2018

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-opencontrail that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: Ic22c391b205d6910bc43c68c3f284f6bcc0de0e8

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-horizon that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: Ie07b8fba95e48363f4e505d008cef96077dfeee8

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-heat that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I1bc85521dada2ed3d05f572e80e936c6f42759e8

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-glance that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I9cb82a4c4d4f5a68aa2a09460a5aae04600ace86

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-nova that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: Ic40ab9a5cff1a6c4dca3b605714c3e115ee3a5b1

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-cinder that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I7c058fe8c0831cc776b05e9a348278196a81e5dc

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-keystone that referenced this issue Feb 8, 2018

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-keystone that referenced this issue Feb 8, 2018

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-neutron that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I7febfdf5573bda6cbd01a96ef4b7c0838f8ba477

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-java that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I5194521dfab2e0be70660224c78b8b1711457ad7

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-mysql that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I7e020e1684fbcd3cf49bfca2ccf6ecddf30d4276

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-roundcube that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: Ib08b3b37ca23379cf7f6c89aa9b0c3f5b497dd8d

cloudlab-autobuild pushed a commit to salt-formulas/salt-formula-apache that referenced this issue Feb 8, 2018

Fixed shebang kernel limitation with virtualenv during test run (pypa…
…/virtualenv#596)

Change-Id: I48f9c7d30906ccdf2a0f8c8bcce4e032cb708df5
@crccheck

This comment has been minimized.

Copy link

commented Aug 8, 2018

If anyone find this confusing:

A work-around for this problem is to not run pip directly, but by run the virtualenv's python interpreter and pass the pip script to python as the source to execute.

That means instead of pip install -r requirements.txt do python -m pip install -r requirements.txt

enkeys added a commit to rh-messaging-qe/iqa_topologies that referenced this issue Oct 11, 2018

enkeys added a commit to rh-messaging-qe/iqa_topologies that referenced this issue Nov 6, 2018

Hyask added a commit to Hyask/websockify that referenced this issue May 28, 2019

Add a __main__.py file for invocation as a module
This file allows an easy invocation of websockify as a module, which is handy when running into shebang issues (pypa/virtualenv#596).
With this patch, `python -m websockify ...` has just the same behavior as `./run ...`.

Hyask added a commit to Hyask/websockify that referenced this issue Jun 13, 2019

Add a __main__.py file for invocation as a module
This file allows an easy invocation of websockify as a module, which is handy when running into shebang issues (pypa/virtualenv#596).
With this patch, `python -m websockify ...` has just the same behavior as `./run ...`.

michaeldallen pushed a commit to michaeldallen/Unit-Testing-and-Test-Driven-Development-in-Python that referenced this issue Jul 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.