There appears to be no way to install specific development versions, --force-dep is being ignored

[designerzim]

Issue

tox is not recognizing --force-dep to install development builds of packages.

Well, first of all, I hope this is just a documentation issue, but we could not find a solution in the tox documentation, or stack overflow, or anywhere else (although chatgpt gave some convincing and completely wrong solutions).

Since about 2016, there seem to be two options. By default, tox only installs release versions of packages. Or the --pre flag or ini setting pip_pre = true can be used to install pre-release packages. However, this means all packages will use pre-release versions, including external packages. We only want releases of the majority of packages we use. We only need pre-releases for the specific packages we are developing for our testing. In our mono repo environment, versioning is for our product releases, not incremental improvements to our infrastructure.

We are using tox in a test environment. Up until our first release, it worked just fine. Now that there is a release, we've hit the default behavior that tox will not install pre-release versions. It's a huge hassle to increment our versions for every change to our test infrastructure balanced with tracking product releases. But even if we did change that plan, we need a way to test our various packages work before making a new minor release tag.

We only want to allow pre-release versions of our own internal packages under test, but tox is ignoring these. Other internal packages as well as all external packages should only use the latest release versions, unless pinned otherwise.

We have two main packages (names changed for security). mainpackage and supportpackage. --force-dep is passed for mainpackage through devpi, this works just fine. --force-dep for supportpackage is passed to the tox arguments, and this is being ignored.

Version 0.5.1.dev78+geba12981 exists in our devpi. The devpi command uses this. The tox command does not. Instead we get version 0.5.0 installed, the release version.

The end result is:
mainpackage==0.5.1.dev78+geba12981, supportpackage=0.5.0

And they are incompatible.

Environment

• OS: Windows 10
• pip list of the host Python where tox is installed:

There is too much to list fully and I doubt it's helpful as this is from pip, not the tox environment.

-------------------- -----------
...
build                0.8.0
certifi              2022.6.15
cffi                 1.15.1
...
devpi                2.2.0
devpi-client         5.2.3
devpi-common         3.7.0
devpi-server         6.6.0
devpi-web            4.1.0
devpicito            1.1.0
distlib              0.3.6
docutils             0.19
...
pip                  23.0.1
pipenv               2022.9.4
pkginfo              1.8.3
...
py                   1.11.0

...
setuptools           58.1.0
setuptools-scm       7.0.5
...
toml                 0.10.2
tomli                2.0.1
tox                  3.25.1
...

But that's not the tox environment, which the user doesn't have access to from a pip call. The salient package is this:
supportpackage==0.5.0

The output of the tox environment is again too much to be useful, but the important parts are:

py39 installed: ...,mainpackage @ file:///C:/WINDOWS/TEMP/devpi-test630/targz/mainpackage-0.5.1.dev78%2Bgeba12981/.tox/.tmp/package/1/mainpackage-0.5.1.dev78%2Bgeba12981.tar.gz, ..., supportpackage==0.5.0,...

We first issue a devpi command:

From that, the resulting tox command is this:

tox --installpkg C:\WINDOWS\TEMP\devpi-test630\downloads\mainpackage-0.5.1.dev78+geba12981.tar.gz -i ALL=https://gitlab.company.net/project/gitbranch/+simple/ --recreate --result-json C:\WINDOWS\TEMP\devpi-test630\targz\toxreport.json -c C:\WINDOWS\TEMP\devpi-test630\targz\mainpackage-0.5.1.dev78+geba12981\tox.ini -v --force-dep supportpackage==0.5.1.dev78+geba12981 -- -v --dir=C:\GitLab-Runner\builds\mFZyJJD2\0\gitroot\subdir\Demo --instafail -x

Or more succinctly:

tox -v --force-dep supportpackage==0.5.1.dev78+geba12981

git tags look like this:

v0.5.0rc1
v0.5.0

with git describe reporting:
v0.5.0-78-geba12981

0.5.1.dev78 is derived by setuptools-scm based on the git describe. The specific dev version changes with every check in.

Output of running tox

Too much private information to post it all but the salient bit is here:

This bug existed in 0.5.0 in an unused method. The method is now being used with the attribute fixed in our current development package. But as previously described, tox is not installing with the forced dependency to use the current development package.

Minimal example

Have a release version tag, like v1.0

Upload two packages with this version to a devpi server. The primary package for testing, and a support package needed for testing. Write a basic unit test that calls one function, class, or method of the support package from the main package. Consider putting an invalid attribute in the support package in a function, but do not call upon the function yet.

Either make a pre-release version tag, like v1.0.dev1, or use setuptools-scm to automatically determine versions, and make any changes after the release tag. For clearer effect, change something fundamental between the two packages that will break compatibility with older versions. e.g. change the name of a method called by the main package into the supporting package. Upload both packages to the same devpi server.

Try to run tox --force-dep package==1.1.dev4 (or whatever the derived or specified dev version is) for the development version. Check to see what packages got installed and if they differ. If only using tox, you will only see 1.0 for both packages. If using devpi to install the main package, they should differ. If they were made incompatible, tests should also fail.

[designerzim]

We found a way around this, by adding our dependency to tox.ini. By adding supportpackage there, --force-dep is finally being acknowledged. When it is in setup.cfg, it seems to follow pip's rules and either be all or nothing with the --pre flag.

This seems like a documentation solution. It would be great if this was clarified in the --force-dep description.

[gaborbernat]

PR welcome to fix this 😊