New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tox shouldn't call pip directly to avoid shebang limitations #66

Closed
pytoxbot opened this Issue Sep 17, 2016 · 11 comments

Comments

Projects
3 participants
@pytoxbot

pytoxbot commented Sep 17, 2016

When the python interpreter for an environment is in a deep directory hierarchy (in my case due to a jenkins matrix with 4 axis), tox will make a pip call like:

#!sh

../bin/pip

The shebang in this file has the full path for the python interpreter in the virtualenv. In one of my failing tests it was:

#!sh

#!/home/tabo/jenkins/workspace/tbtest3/TOX_DB/postgres/TOX_DJANGO/dj14/TOX_PYTHON/py27/os/linux/.tox/py27-dj14-postgres/bin/python2.7

With a path length of 131 that exceeds linux's limitation of 127. Patching tox to call $PYTHON_BIN_PATH $PIP_BIN_PATH instead of just $PIP_BIN_PATH solves the problem.

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

addresses issue66: add --workdir option to override where tox stores its ".tox" directory
to "pip freeze" to obtain the list of installed packages.
Thanks Ted Shaw, Holger Krekel.
close issue66: add documentation to jenkins page on how to avoid
"too long shebang" lines when calling pip from tox. Note that we
can not use "python -m pip install X" by default because the latter
adds the CWD and pip will think X is installed if it is there.
"pip install X" does not do that.

(experimental) New feature: When a search for a config file fails, tox tries loading

→ <<cset 11c13b4313a4>>

1 similar comment
@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

addresses issue66: add --workdir option to override where tox stores its ".tox" directory
to "pip freeze" to obtain the list of installed packages.
Thanks Ted Shaw, Holger Krekel.
close issue66: add documentation to jenkins page on how to avoid
"too long shebang" lines when calling pip from tox. Note that we
can not use "python -m pip install X" by default because the latter
adds the CWD and pip will think X is installed if it is there.
"pip install X" does not do that.

(experimental) New feature: When a search for a config file fails, tox tries loading

→ <<cset 11c13b4313a4>>

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

So it turns out defaulting to "python -m pip" is a breaking change we cannot do. The problem is that this invocation adds the current working directory and this makes pip think a package is installed if it is in the invocation current working directory. You can change "install_command" to use "python -m pip" if that works for you (e.g. if you don't have packages in your root dir) or, probably a better idea, use the new "--workdir" option to have a tox working directory which is not deeply nested. The latter method has the added benefit that also all other invocations of scripts in a virtualenv will work. Even if we made "python -m pip" work you would still not be able to invoke other installed scripts in venvs that have too long path names.

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

fix issue66 by introducing using default "python -m pip"
and introducing list_dependencies_command. (Ted, Holger)

→ <<cset 73d3b0cabef0>>

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hawkowl

Yeah, I'm hitting this too on Twisted, as we try and move tox-first, including on the Windows builders.

With the install_command workaround, it works better, but then fails right after with:

GLOB sdist-make: /Users/red/code/tw spc/setup.py
py27-tests create: /Users/red/code/tw spc/build/py27-tests
py27-tests installdeps: cryptography >= 0.9, pyopenssl >= 0.13, service_identity, idna >= 0.6, pyserial, python-subunit, pycrypto, appdirs, soappy
py27-tests inst: /Users/red/code/tw spc/build/dist/Twisted-16.1.1.zip
Traceback (most recent call last):
  File "/Users/red/venvs/commands/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 373, in runcommand
    return self.subcommand_test()
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 540, in subcommand_test
    output = py.process.cmdexec("%s freeze" % (pip))
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/py/_process/cmdexec.py", line 28, in cmdexec
    raise ExecutionFailed(status, status, cmd, out, err)
py.process.cmdexec.Error: ExecutionFailed: 127  /Users/red/code/tw spc/build/py27-tests/bin/pip freeze
/bin/sh: /Users/red/code/tw: No such file or directory

I believe https://bitbucket.org/hpk42/tox/pull-requests/172/do-not-call-pip-directly-trying-to-fix/diff should fix this issue.

@pytoxbot pytoxbot closed this Sep 17, 2016

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by gralamin

This is still an issue I hit (running tox 1.7.2). Has this been fixed in any newer versions and missed being closed, or is it still an open defect?

Thanks!

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

thanks for your perseverance!

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @tabo

Plan to do that tonight. I'm still cleaning up the mess I did including prints in the stdlib in that server while debugging this :)

@pytoxbot

This comment has been minimized.

pytoxbot commented Sep 17, 2016

Original comment by @hpk42

could you prepare a patch including adapting tests?

@pawamoy

This comment has been minimized.

pawamoy commented Dec 1, 2016

Just referencing other related issues in virtualenv and pip:

I'm still experiencing this problem when using tox through tox itself with long testenv names!
I'll try --workdir option and give some feedback.

opnfv-github pushed a commit to opnfv/apex that referenced this issue Feb 10, 2017

Fixes networking-bgpvpn pkg install
Also includes minor change for tox working directory to move up one dir
(build dir) for congress.  This is due to tox-dev/tox#66

Removes networking_bgpvpn.conf, and uses neutron service_providers to
set.  This is because we do not have upstream patches to configure
bgpvpn conf file via OOO and having 2 definitions breaks deploy.

Change-Id: I9846bd959c1ed8e2e81f3ae30470ec357da3fc8f
Signed-off-by: Tim Rozet <trozet@redhat.com>
@jsharpe

This comment has been minimized.

jsharpe commented Oct 5, 2017

For reference using install_command is not sufficient for this as there is a later call to pip freeze which is not customisable so for this to work we'd need to be able to also override the pip freeze call too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment