Single cross-version travis envlist

[ryanhiebert]

From @jayvdb on #16:

Ultimately, every part of your build matrix must be specified in the .travis.yml

Worth thinking about whether this is avoidable, at least for simple cases, and it looks like it is.

The suggestion that he brings up is to have a key in the [tox:travis] section to declare the envs that tox-travis should run:

[tox:travis]
envlist = py{27,34,py,py3}

And then use that to populate the list of environments if no Travis python version build matrix is specified. That makes some sense but:

• It would require parsing the .travis.yml file, which I currently avoid. I don't want to duplicate their parsing logic.
• It's unclear to me what it should do if there are multiple python versions declared in the .travis.yml, but not enough to cover the envs listed in the tox-travis envlist. Should it try to assign them? Warn? Related to Runtime check for tox envs that won't run #18 .

So far I'm not sure if an attempt at this would be a feature or a misfeature. If someone wants to come up with a design of the work that would need to be done, I wouldn't stand in the way, but I'd encourage thinking through the design well before writing the code. I think there are some challenges that need to be addressed first, and discussed to make sure that we can live with the consequences of the complexity they will add.

[jayvdb]

IMO this issue should be titled "Single cross-version travis envlist" or similar, as the single envlist is what I was thinking might be valuable.

I'm appreciating that this could be a mis-feature. There is not much benefit at present for using tox-travis with a single envlist. The only benefit I can see is the simplicity it brings for the simple case, as .travis.yml then doesnt need to include the list of supported/tested versions, and some people are more comfortable modifying tox.ini , rather than .travis.yml. That it runs all envs in a single job is also a small positive if the test envs are very quick, as the time to spin up multiple images can be unnecessary delays, but the single job with one log becomes a negative very quickly as the test suite grows and the failures are more complicated.

As I mentioned in the original comment, this mode is already possible by

[tox:travis]
2.7 = py{27,34,py,py3}

Aliasing envlist to be the same as whatever TRAVIS_PYTHON_VERSION is (default 2.7) is all that is required for this 'feature' to work. This would work well for the very simple case where .travis.yml has only a single job. For multiple jobs, dont use envlist ;-)

However it would re-run the envs if multiple jobs invoke tox with tox-travis in effect (no -e or TOXENV). How much of a problem is that? The user should notice that this has occurred, and rectify their build config.

We could add a warning or error by checking that TRAVIS_JOB_NUMBER isnt a second job in the build , with the warning or error message explaining that [tox:travis] envlist mode is only supported on the first job in the build.

I cant think of another approach which could accurately warn the user without very accurate parsing of .travis.yml , which would be problematic.

[ryanhiebert]

OK, that's a more reasonable thought, then, if it only attempts to work with a single job. Let me share another thought with you, and see if it affects your thought on the design of it.

Currently the keys in tox:travis are the version specifiers. This seemed good to start, but I've come to realize that it was short-sighted if we want to be able to understand other factors as well. In another ticket, I proposed having the keys be the environment variable that they were associated with, but that never quite felt right.

What I like the idea of most ATM for specifying per-version dependencies is this:

[tox:travis]
python = 
    2.7: py27
    pypy: pypy
    nightly: py36

Given that idea, perhaps the single-runner version would just be:

[tox:travis]
python = py{27,34,py,py3}

Which would necessarily be telling tox-travis that only one Python should be running. It also leaves the possibility for envlist to be used later, as an explicit declaration of all the envs that tox-travis should know about.

[jayvdb]

Yea, I've been thinking about #4 ; not sure I have a good answer for that one yet, but I've got some ideas floating around.

What is interesting for this issue is the problem of supporting python on linux and osx and eventually win (travis-ci/travis-ci#2104), if we were using a single cross-version travis envlist, as that would require multiple jobs , which would break the TRAVIS_JOB_NUMBER check.

Here is a osx travis job that has multiple python interpreters available to tox
https://travis-ci.org/jayvdb/my-ci-test/builds/141555469
https://github.com/jayvdb/my-ci-test/blob/python-osx/.travis.yml

The benefit of a single job is even greater for osx, as they are much slower starting up.

This can be easily supported in tox-travis by allowing a single envlist and using tox's skip-missing-interpreters. This is again assuming that there is only one job for each platform, and Python version is the only factor. But that means we cant assert that TRAVIS_JOB_NUMBER is the first job in the build.

When TRAVIS_JOB_NUMBER shows that the build has multiple jobs, rather than parsing .travis.yml, we could instead fetch a parsed version from the Travis API builds.

e.g. https://api.travis-ci.org/builds/140967962 includes the config for each job, with an 'os' key for each job. We could assert that there is only one job per os.

Here is the JSON for two individual jobs, which makes it easier to see what the API provides for each job:
from the python: list : https://api.travis-ci.org/jobs/140967967
from matrix: include: : https://api.travis-ci.org/jobs/140967981

Then we run into the very likely problem that a multi-os tox.ini very likely doesnt support all testenv's for all OS in the matrix, possibly because the dev team doesnt have resources to set it up, or there are unsolved (and sometimes impossible) problems on a specific platform. The dev team would want to disable some testenv on some platforms, but tox only has skip support for Python version, so tox-travis needs to fill this void. (and #4 is one part of the solution, but Travis also has unequal support for each platform, so django factors on Linux can be mapped to Django support in Travis, but it doesnt exist in OSX and the dev team needs to manually set it up.)

[ryanhiebert]

You've convinced me that attempting to warn using TRAVIS_JOB_NUMBER is a bad idea. I'll go even further, though and say that given the complexity, attempting to warn at all when stuff is run too much is a bad idea. It's too complicated to get correctly, and I don't think it's worth it. Folks that care will see that too much is running on their Travis jobs, and they'll figure out why.

Still, we'd like to support a simple case where a single job can run all the desired tox envs. First, though, I think we've got to have some idea where we want to go with the other factors.

There are 4 factors that I think would be helpful to support in tox-travis. There may be other factors, but I think all of these would be helpful. They are: os, language, python, and env. env is going to be a bit more complicated, so I'll leave it out of this discussion for the time being.

I have a pattern of how I'd like to support these that I think will be quite nice, and make a simple paradigm. First, I'd like to see the keys in the tox:travis config section match the keys in the .travis.yml file. People shouldn't need to know what environment variables Travis uses. Second, I want to use the format I specified in a previous comment to associate the values with the envs that should be run. Third, I want the default of any of those to simply mean "run everything".

The one exception to those rules is that the python versions should be an exception to that third rule, and the default should be to only match envs that are associated with that python version identifier (py27, py34, etc), since that is the most common case. This leads me to the desire to provide configuration to allow that default to be reverted.

Also, I note that I have no way to tell the difference between the version being absent and the version being defaulted (to 2.7), and between the os being absent and the os being defaulted (to linux). Thus, basing an "everything" variable on the absence of a configuration item isn't going to work.

All in all, a cross-platform configuration might look something like this in the tox.ini. The name of the default-affecting flag is up for debate.

[tox:travis]
python_all_envs = True
language =
    python: py{27,32,33,34,py}-{spam,eggs}
os =
    osx: py{27,34}-{spam}

It seems a bit line noisy when you're just overriding one thing each, but I don't see any reason to not enable overriding for the non-default os and language if desired. The python_all_envs configuration sticks out as totally different from the rest in this case, but it seems best to me to keep the defaults for the Python versions, and this seems like the most straightforward way to let the language and os settings trickle down.

[ryanhiebert]

Alternative for the default-affecting flag: autoenv, defaulting to True (so the override would be False). The "autoenv" feature would be telling tox-travis to use the appropriate envs for the given python versions.

[ryanhiebert]

It occurs to me that it would still be possible to use the python (or os or language) keys bare, and just have it apply that line to all variations:

python =
    py{27,34}, docs  # Only these will run for _any_ Python version
    3.5: docs        # Run the docs on py35.
    3.3: spam        # Won't be run, because it's not in the list for any Python version
    # 2.7 and 3.4 are defaulted, automatically, to py27 and py34
    # autoenv = False could be added to make it run all the versions

This would make sense if you think about it like tox's deps, which may be universal or conditional based on the presence of a prefix.

[ryanhiebert]

@jayvdb: Do you still see enough benefit to this for it to be worth doing?

[ryanhiebert]

I'm open to re-evaluating if anyone comes up with a need for it, but I think the common cases are pretty much handled with the variants that we look at. Feel free to make your case for re-opening if you disagree.

[ryanhiebert]

With the discussion on #97, I'm re-opening this issue to add the envlist configuration for Tox-Travis. I'm not including in this issue the idea of allowing all Python versions to be run on a single Travis job, because I don't think it should be spelled using envlist. I am open to the possibility of it being a feature with a different setting, perhaps the autoenv = False that I mentioned above.

[ryanhiebert]

Also, it should default to the Travis envlist, plus any manually specified testenv sections, plus the pyXX that maches the version of Python that is currently running (as approximately specified on #97).