Ability to share tox environments within a project

[apteryks]

Reproducing Steps

1. Save the following configuration in a "tox.ini" somewhere. Note that the "envdir" is commented out for now.

[tox]
skipsdist = True
envlist = py27,pylint

[testenv]
deps =
    pytest
    pylint
commands =
    pytest --version

[testenv:pylint]
#envdir = {toxworkdir}/py27
commands = pylint --version

2. In the same directory of this newly created tox.ini file, execute tox twice (tox). The output should look like that:

py27 installed: astroid==1.4.8,backports.functools-lru-cache==1.3,configparser==3.5.0,isort==4.2.5,lazy-object-proxy==1.2.2,mccabe==0.5.2,py==1.4.31,pylint==1.6.4,pytest==3.0.5,six==1.10.0,wrapt==1.10.8
py27 runtests: PYTHONHASHSEED='2007619177'
py27 runtests: commands[0] | pytest --version
This is pytest version 3.0.5, imported from /home/mcournoyer/workspace/test_proj/.tox/py27/lib/python2.7/site-packages/pytest.pyc
pylint installed: astroid==1.4.8,backports.functools-lru-cache==1.3,configparser==3.5.0,isort==4.2.5,lazy-object-proxy==1.2.2,mccabe==0.5.2,py==1.4.31,pylint==1.6.4,pytest==3.0.5,six==1.10.0,wrapt==1.10.8
pylint runtests: PYTHONHASHSEED='2007619177'
pylint runtests: commands[0] | pylint --version
No config file found, using default configuration
pylint 1.6.4,
astroid 1.4.8
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4]
_____________________________________________________ summary ______________________________________________________
py27: commands succeeded
pylint: commands succeeded
congratulations :)

real 0m2.066s
user 0m1.804s
sys 0m0.224s

3. Now uncomment the line "#envdir = {toxworkdir}/py27" in the tox.ini file and save it, and rerun tox twice. The output should now look like:

py27 recreate: /home/mcournoyer/workspace/test_proj/.tox/py27
py27 installdeps: pytest, pylint
py27 installed: astroid==1.4.8,backports.functools-lru-cache==1.3,configparser==3.5.0,isort==4.2.5,lazy-object-proxy==1.2.2,mccabe==0.5.2,py==1.4.31,pylint==1.6.4,pytest==3.0.5,six==1.10.0,wrapt==1.10.8
py27 runtests: PYTHONHASHSEED='809406144'
py27 runtests: commands[0] | pytest --version
This is pytest version 3.0.5, imported from /home/mcournoyer/workspace/test_proj/.tox/py27/lib/python2.7/site-packages/pytest.pyc
pylint recreate: /home/mcournoyer/workspace/test_proj/.tox/py27
pylint installdeps: pytest, pylint
pylint installed: astroid==1.4.8,backports.functools-lru-cache==1.3,configparser==3.5.0,isort==4.2.5,lazy-object-proxy==1.2.2,mccabe==0.5.2,py==1.4.31,pylint==1.6.4,pytest==3.0.5,six==1.10.0,wrapt==1.10.8
pylint runtests: PYTHONHASHSEED='809406144'
pylint runtests: commands[0] | pylint --version
No config file found, using default configuration
pylint 1.6.4,
astroid 1.4.8
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4]
_____________________________________________________ summary ______________________________________________________
py27: commands succeeded
pylint: commands succeeded
congratulations :)

real 0m11.402s
user 0m9.856s
sys 0m1.440s

4. Observe that, when reusing the same envdir, the environment always gets recreated (and takes more time to do so) even though the two targets (pytest & pylint) share the exact same requirements.

System Information

OS: Ubuntu 14.04
tox version: 2.5.0
Python version: 2.7.12
pip list: Too many to list here (can provide upon request).

[obestwalter]

Thanks for the detailed report @apteryks.

[apteryks]

No problem! And thanks for the great software!

[speedyleion]

I was able to reproduce this on windows with version 2.5.0

I've traced it to this function in config.py

 def basepython_default(testenv_config, value):
        if value is None:
            for f in testenv_config.factors:
                if f in default_factors:
                    return default_factors[f]
            return sys.executable
        return str(value)

It looks like py27 is a default factor so it grabs the basepython from the default_factors dictionary, this happens to be python2.7.
pylint is not a default factor and thus uses sys.executable. This makes the two environments differ so that each one is being recreated.

You can work around this by setting basepython in the pylint environment to python2.7

[testenv:pylint]
basepython=python2.7
envdir = {toxworkdir}/py27
commands = pylint --version

[speedyleion]

@apteryks if you do discover that the issue can be resolved by specifying basepython, I think that may be by design:

• http://tox.readthedocs.io/en/latest/config.html#confval-basepython=NAME-OR-PATH says that the default is the interpreter used for tox invocation, sys.executable in my example above.
• http://tox.readthedocs.io/en/latest/example/basic.html#a-simple-tox-ini-default-environments though it takes some interpretation it mentions that the default environments will use the appropriate python interpreters.

It would seem pylint is doing what is expected it's not a default so it uses sys.executable. py27 is a default so it finds the systems python2.7 executable.
Again this is only if the basepython fixes the problem you were seeing.

[obestwalter]

@speedyleion thanks for digging into this. I agree that this is a side effect of sharing between a default environment and a non-default environemnt. It is obvious if you share environments that use different interpreters. What is surprising to me is, that the recreation also happens when sharing the same interpreter unless you set basepython as you decribed.

To summarize if you run tox with Python 3.6 the following happens ...

With tox.ini:

[tox]
envlist = py36,nonstandard
[testenv]
envdir = {toxworkdir}/sharedenvdir
commands = python --version

=> against expectation: constant recreation although interpreter is the same for both environments.

With tox.ini:

[tox]
envlist = nonstandard1,nonstandard2
[testenv]
envdir = {toxworkdir}/sharedenvdir
commands = python --version

=> as expected: environment is reused

With tox.ini:

[tox]
envlist = py36,nonstandard
[testenv]
basepython=python3.6
envdir = {toxworkdir}/sharedenvdir
commands = python --version

=> as expected: environment is reused

I guess this is an unintended side effect of the default environments. @hpk42 - what do you think?

BTW: To weird things up one can do:

[tox]
envlist = py36,py27
[testenv]
basepython=python3.6
envdir = {toxworkdir}/sharedenvdir
commands = python --version

... which leads to the py27 environment also be run with python3.6 which is very likely not intended by the user and a mistake in the configuration. This might lead to unnecessary confusion and I wonder if we should catch that and throw an appropriate error, when we encounter configurations like that?

[apteryks]

@speedyleion, @obestwalter: Thanks for looking into this. I can confirm that the environments stop being recreated when I set basepython to python2.7 either in the non-standard environment section [testenv:pylint] or in the shared base environment [testenv].

So I'm grateful I could make it work, but I'm wondering what could be made to make this situation less confusing. Maybe the information that is used to "identify" an interpreter could be taken from its '--version' output rather than its path? It would still be a weak verification, but it would at least allow sharing the envdirs between standard and non-standard environment without setting this not-so-intuitive basepython, provided the sys.executable and the python2.7/python3.6, etc '--version' output match.

[obestwalter]

If you want to look into that a bit more in detail - a PR fixing that will likely be merged, as this is definitely not desirable behaviour. The default environments seem to be a bit special though and I have not looked into their implications in detail though, so there might be some surprises waiting ... it might as well just be though that this is just a case that hasn't been considered in the original implication and is fairly easy to fix in the way that you propose.

BTW My last example should definitely throw an error as I can't imagine any sane use case for that, or am I missing something? I think that is another issue though, so I will open one for that.