Error - no module named virtualenv

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/360
• Originally reported by: jaraco
• Originally created at: 2016-08-24T02:40:36.844

When using the advertised setuptools integration on a clean operating system that doesn't already have virtualenv installed, the tox run will fail when it attempts to invoke virtualenv:

py35 create: /vagrant/Dropbox/code/public/cherrypy/.tox/py35
ERROR: invocation failed (exit code 1), logfile: /vagrant/Dropbox/code/public/cherrypy/.tox/py35/log/py35-0.log
ERROR: actionid: py35
msg: getenv
cmdargs: ['/usr/bin/python3', '-m', 'virtualenv', '--python', '/usr/bin/python3.5', 'py35']
env: {'TERM': 'xterm-256color', 'HOME': '/home/vagrant', 'SHLVL': '1', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'MAIL': '/var/mail/vagrant', 'LOGNAME': 'vagrant', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'USER': 'vagrant', 'PWD': '/vagrant/p/cherrypy', 'SSH_CLIENT': '10.0.2.2 65337 22', 'LANG': 'en_US', 'SSH_CONNECTION': '10.0.2.2 65337 10.0.2.15 22', 'PYTHONHASHSEED': '2586869941', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_ID': '9', '_': '/usr/bin/python3', 'SSH_TTY': '/dev/pts/0', 'PLAT': 'linux-x86_64', 'VIRTUAL_ENV': '/vagrant/Dropbox/code/public/cherrypy/.tox/py35', 'LANGUAGE': 'en_US:', 'PATH': '/vagrant/Dropbox/code/public/cherrypy/.tox/py35/bin:/home/vagrant/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'OLDPWD': '/home/vagrant', 'XDG_RUNTIME_DIR': '/run/user/900', 'SHELL': '/bin/bash'}

/usr/bin/python3: No module named virtualenv

ERROR: InvocationError: /usr/bin/python3 -m virtualenv --python /usr/bin/python3.5 py35 (see /vagrant/Dropbox/code/public/cherrypy/.tox/py35/log/py35-0.log)

I think the issue is obvious - tox is launching a subprocess to invoke virtualenv, but since virtualenv was loaded dynamically by the setuptools distutils command test, the subprocess doesn't have that library.

I've worked around this issue in the past (when using pytest-runner to test libraries whose test suite would launch subprocesses) by adding all of the items in sys.path to the environment variable PYTHONPATH when launching the subprocess.

I think there are a few options to address this issue.

• Tox could rely on pyvenv on supported Python 3 versions, eliminating the dependency on virtualenv in those environments.
• Tox could launch virtualenv in-process rather than as a subprocess.
• Setuptools could implement the PYTHONPATH technique for all test commands, avoiding these issues in the general case.
• Tox could implement the PYTHONPATH technique when launching virtualenv.
• Tox could explicitly ensure that virtualenv will be on the path in the subprocess (knowing that it could have been resolved by the test command.
• Tox could drop support for the distutils-bootstrapped technique, possibly preferring rwt as an alternate one-line bootstrapper for tox (though rwt itself currently requires bootstrapping).

There may be other options as well. I've listed these approaches in order by my preference.

What are the thoughts and preferences of the tox maintainers?

[pytoxbot]

Original comment by @hpk42

What happens if we just make virtualenv a test requirement for the "setuptools test command integration"?

Right now, tox kind of hard-depends on virtualenv and i think it's cleanest to express this. At the sprint in June we had talks about supporting venv and conda but it's some work obviously.

[jaraco]

What happens if we just make virtualenv a test requirement

It doesn't matter if the requirement comes as a test requirement or a dependency on a test requirement - it will only be available in the immediate Python environment and not any subprocesses. The same thing would happen to tox if somehow tox were to try to invoke tox through a subprocess. I'd like to avoid package maintainers having to communicate to their users that "tox" will install automatically but virtualenv needs to be more permanently installed.

The more I think about this issue, though, the more I think it shouldn't be a tox issue. I'm going to opt for (3) for the near term and (6) in the long term (especially if pypa/pip#3971 lands).

[bittner]

What happens if we just make virtualenv a test requirement for the "setuptools test command integration"?

@hpk42 Note that, as documented in #330, tox doesn't seem to pull virtualenv as a dependency if integrated in setuptools. Not even if you specify it explicitly in setup like so:

setup(
    # ...
    tests_require=['tox', 'virtualenv'])

[jaraco]

Indeed, my report is a duplicate of #330. I've got a fix committed to setuptools and slated for a v27.3.0 release.

[hpk42]

As #330 has been closed with not recommending to use tox within "setup.py test" anymore, we can close this issue here as well.