Allow tox to skip dependency installation on virtualenvs (--skipdeps)

[ssbarnea]

If you don't mind, I would like to enable tox to ignore the deps sections when is executed. To explain the use case here: we are using tox to test python code on different environment but in our case we need to test the packages using the system installed python libraries (RPMs) and we never use pip as this would break os level dependencies.

Currently what we have to do is something like:

truncate --size 0 requirements.txt;
truncate --size 0 test-requirements.txt;
tox -e py27

Because tox.ini contains deps = -r requir...txt this works but is at least ugly and also is modifying the code.

It would be much better if we could just call tox -e py27 --skipdeps.

[RonnyPfannschmidt]

at first glance your request goes directly against the use case of tox (isolation)

there is a tox extension called tax which does something similar to your stated goal (run the commands and installation in the current instead of a new env

it seems possible/sensible to just do something similar

https://pypi.python.org/pypi/tax
https://github.com/ionelmc/python-tax

[ssbarnea]

@RonnyPfannschmidt Thank for the quick reply. I am afraid that tax extension may not be what need because this extension does not have any option to ignore deps, and this means that when I will run it it will attempt to install new packages from deps.

The entire idea was to bypass dependency installation from tox in order to validate if the current tox targets are working with current system configuration. We do provision the requirements outside but we need to verify that the tests are working.

As you can imagine we do have multiple tox actions to run and something like --skipdeps or --ignoredeps would be really helpfull. Would you accept a PR if I make one?

[obestwalter]

@RonnyPfannschmidt I do not think this goes against the use case of tox. The idea behind tox as I understand it is to make testing posssbile "out of the box". If isolation and installation of dependencies is not needed/wanted to test something "out of the box" I think it would be good to be able to to opt out. Scenarios like wanting to run the tests in the current interpreter (tax) without any installation of dependencies (--skipdeps) could come in handy.

The question is how this should be implemented. Maybe the --skipdeps is possible to implement as a plugin already or with an appropriate new hook added which could be useful in other situations as well. I just started to try to get my head around the code base a bit more, so don't expect me to tackle this any time soon, but I do think that this could be a useful extension of functionality.

[eddie-dunn]

What is the status of this issue? I would really appreciate being able to speed up testing with tox inside a docker container (which already has all required dependencies installed).

[obestwalter]

@eddie-dunn AFAIK nobody is working on it and there is even no agreement yet, if this should be added to tox or not.

[attila123]

+1 for the docker container use case. Due to this feature is not available, I need to remove the commands to run my integration tests from tox.ini and put them to Makefile.

[ssbarnea]

I think that this feature should be added to tox itself as there are lots of use-cases, so it would worth getting some kind of voting regarding the need of lack of need to add it to tox itself.

Fell free to use +1 and -1 directly on the ticket to mark if you agree or not on it. If you opt for another approach use another icon like the confused one.

[gaborbernat]

I tend to disagree with the concept of skipping it. I would prefer to have an extension that in such cases uses the system package manager to install packages (if possible) and fail otherwise. E.g. inside Docker should be list installed python packages, check that my requirements satisfy that -> if it does do nothing, otherwise raise an error.

[janhn]

For completeness, my use case was that I was experiencing rebuilds of virtual environments prepared by tox itself, so this might have been a useful feature (nicely complementing --notest) so that tox can be used in two invocations: 1. build environments (say, in docker/ci context), 2. execute tests.

But 2. was rebuilding the venvs, until I discovered this was caused by the way tox figures out whether it can reuse an environment. All things being same, the only difference was the directory, from which tox was executed, specifically the paths to the files listed in deps, e.g.:

py27 cannot reuse:                                                                                                 
 {('00000000000000000000000000000000', '-r/original/build/path/requirements.txt')}                            
 !=                                                                                                                 
 {('00000000000000000000000000000000', '-r/different/execution/path/requirements.txt')}

Since I'm using {toxinidir} and relative paths, this can be worked around with -c to the original tox.ini.

Perhaps the validation of 'deps equality' could be enhanced by storing dep. file checksums in addition to just deps' paths?

[gaborbernat]

Yes, that's probably what should be changed then.

[attila123]

For the record, I solved this problem with testing in docker containers by "brute force way" of moving all the commands from tox.ini to my Makefile (I am not too familiar with tox anyway) and e.g. for unit testing the 'commands' section is just one 'make' call, e.g.:

commands =
    sh -c "envname={envname} make _unit-test"

(Here I pass envname because it goes into some coverage file names (although I only use py37). Also the '_' in '_unit-test' just signifies that this target is not to be called directly).
So I basically reduced the functionality of tox to just create the virtualenvs with the appropriate dependencies and only for development. For CI testing, I don't use tox at all, I install the python dependencies globally in my docker containers with pip.
I also found the tox-venv (https://gist.github.com/smoser/2d4100a6a5d230ca937f) very useful to run commands inside virtualenvs created by tox, e.g.

/path/to/tox-venv <virtual env name e.g. py37> pip list --outdated

Hope this helps some python devs.

Gist

tox-venv and venv: easily activate tox created environments or create virtualenv envs.

tox-venv and venv: easily activate tox created environments or create virtualenv envs. - README.md

[rpkilby]

Side note - the tox-venv gist is not related to the tox-venv package.

[attila123]

Side note - the tox-venv gist is not related to the tox-venv package.

@rpkilby Thanks for the information. I found this tox-venv ghist first and did not know about the tox-venv pypi package. (As the tox-venv ghist is working for me, I will not try to switch to the tox-venv pypi package (to save time).)

[mike0sv]

I'd like to use tox -e check (pep8 checks and stuff) in my pre-commit hook. Obviously I already have all the requirements in check env, so this option would speed up my commits.

[pabs3]

Another use-case of this feature would be that Debian could use it in autopkgtests on our CI system. These tests aim to test the package as it is installed into a .deb rather than the code in the source package. We could run tox using the tox.ini and tests in the source tree against the code installed in the .deb. This use-case would also require an option to skip the sdist/install stages though.

[ppoilbarbe]

My use case is that tox is used to check sources, doing tests, building package and generating doc with multiple versions of Python (3.5, 3.7, 3.8, 3.9).
For testing and build, I need dependencies. But for checking source (black, flake8...) and building documentation I don't need them.
In a context of CI (Gitlab CI & Docker) we build a lot of packages (our project is at this day 20 repositories) and when checking source or buillding doc of a package, we don't need all dependencies (which may not be available) and it would be a great economy of time not installing all dependencies.
At now, even with nothing in deps (or conda_deps, we build for conda too), tox reads the requirements.txt file.

[gaborbernat]

For testing and build, I need dependencies. But for checking source (black, flake8...) and building documentation I don't need them.

Well, you still need some dependencies (black/flake8), just a different set of dependencies. Sounds to me like you should be using the concept of the extra via setuptools instead of requirements.txt.

[gaborbernat]

Another use-case of this feature would be that Debian could use it in autopkgtests on our CI system.

@pabs3 for your use case I think what you really want is tox to use the Debian installer to provision packages instead of pip. This can be achieved via a plugin.

[pabs3]

Using apt to provision packages instead of pip won't work, because the script that runs tox does not run as root nor have access to sudo. Instead, the test dependencies are installed by autopkgtest using apt run as root before the test is run as non-root, so I think my case definitely needs a way to skip dependency installation too.

…

-- bye, pabs https://bonedaddy.net/pabs3/

[ssbarnea]

There is a trick you can do: create a "system" env in tox and disable all deps install features and enable it to use system site packages. Mainly tox will use a venv that uses only system packages, this should allow you to validate that package works with system level dependencies. You will never be able to translate pip deps to system package deps, is not practically possible.

[pabs3]

That sounds useful, do you have any more info on how to do it?

…

-- bye, pabs https://bonedaddy.net/pabs3/

[gaborbernat]

You will never be able to translate pip deps to system package deps, is not practically possible.

I don't think this is true.

Instead, the test dependencies are installed by autopkgtest using apt run as root before the test is run as non-root

How do you determine what packages need to be installed (and keep them in sync with projects both install and test dependencies)?

For me, it sounds like you want to invoke twice:

• once with root, where it builds the package and its dependencies;
• and once more subsequently without root, and skipping environment manipulation (you can do this by passing https://tox.readthedocs.io/en/latest/config.html#cmdoption-tox-skip-pkg-installt to the second invocation).

[pabs3]

The dh_python3 tool from dh-python translates pip deps to Debian package deps, but only for installed packages and Depends, it cannot translate pip build requirements for packages that aren't installed, which is especially needed to populate Build-Depends. OTOH, for the Perl ecosystem, dh-make-perl can do that translation, so it should be feasible for py2dsc from stdeb to implement this translation. The packages to be installed during autopkgtest are listed in the file debian/tests/control, which is manually written by maintainers and manually kept up to date by them. If stdeb ever implements the translation mentioned above, then they could be automatically updated. You are correct that tox will get invoked twice, once during the .deb build and once during the autopkgtest, which is run often. The one during the .deb build isn't run as root though, pybuild installs to a local directory and runs the tests there. For the autopkgtest the .deb is already installed so the install isn't needed. It does sound like --skip-pkg-install might work for my use-case, so apologies for not knowing about it and the subsequent noise on the bug.

…

-- bye, pabs https://bonedaddy.net/pabs3/

[gaborbernat]

I'll close this now, as none of the use cases raised actually want to skip install deps, just want to alter how to install deps happen, which is a separate topic.