Non-listed environment not used even if configured in gh-actions block of tox.ini

[imphil]

I have a tox.ini that looks like this (a bit stripped down):

[tox]
envlist = py3

[testenv]
# Dependencies prefixed with "ci: " are only installed when running in CI,
# which calls effectively "tox -e py3-ci" (see the gh-actions section below).
deps =
    pytest
    ci: pytest-github-actions-annotate-failures

[gh-actions]
# Mapping between the Python version used in GitHub Actions matrix builds, and
# the used Tox environment.
python =
    3.5: py3-ci
    3.6: py3-ci
    3.7: py3-ci
    3.8: py3-ci
    3.9: py3-ci

What I want to achieve is the following: run the py3-ci environment when running tox in GitHub Actions, but run the py3 environment otherwise. If I'd list the py3-ci environment in envlist, normal developers running tox would also get this environment if they don't override it on the command line.

Manually running tox py3-ci works fine and tox picks the right factor/environment, but when I run tox (without any argument) in GitHub actions, tox doesn't run anything.

Is there a bug somewhere that tries to match the environments in in the gh-actions key against the ones in envlist?

[marlonjames]

I just ran into this as well.
Looks like it's here:

tox-gh-actions/src/tox_gh_actions/plugin.py

Line 28 in a063112

envlist = get_envlist_from_factors(config.envlist, factors)

[ymyzk]

Hi @imphil, thanks for filing a ticket. This is actually an expected behavior of the current tox-gh-actions. In short, how tox-gh-actions works is

1. Get list of environment candidates from envlist
2. Pick environments from based on factors in the [gh-actions] section.
3. Run selected environments.

In this case, py3-ci is not in the envlist so tox-gh-actions won't try to run.

A workaround will be running tox -e py3-ci on GitHub Actions.

[imphil]

Thanks for your answer @ymyzk. I understand the current behavior, but I'm wondering if it couldn't be changed? Is there a reason why only listed environments can be used in gh-actions?

Besides, the current behavior is confusing: I have an environment specified in gh-actions, and I can run it locally, however, when running GH Actions, tox simply does nothing (since the environment is silently ignored). At least a warning would be very helpful.

A workaround will be running tox -e py3-ci on GitHub Actions.

That's what I'm doing now, but it prevents me from using the (otherwise nice) environment selection in the gh-actions section.

[ymyzk]

I understand the current behavior, but I'm wondering if it couldn't be changed? Is there a reason why only listed environments can be used in gh-actions?

Sure, I can elaborate a bit more on why we're adopting this behavior using a concrete example.

Let's consider the following simple example.

[tox]
envlist = py39-a, py39-b

[gh-actions]
python =
    3.9: py39

The current implementation tries to find environments which has a factor py39. It means tox-gh-actions runs py39-a and py39-b on GitHub Actions as explained in the README as well.

If I understand your expectation correctly and implement it naively, tox-gh-actions should run py39 environment for the example above because tox-gh-actions should treat [gh-actions] as a list of environments to run.

So, this is a design choice and I feel the first behavior is convenient for more users. That's why it works like this. I'm open to ideas if there's a way to satisfy both use cases while keeping tox-gh-actions behavior consistent.

I have an environment specified in gh-actions

This may be a source of confusion. What we specify in [gh-actions] is not a list of environments but a list of factors to match.

That's what I'm doing now, but it prevents me from using the (otherwise nice) environment selection in the gh-actions section.

Maybe I can have a better workaround and/or an idea by having a look at actual tox.ini rather than simplified version. For the simplified one above, I felt tox -e py3-ci is a reasonable but it may not be the case for the actual one.

Thanks.

[marlonjames]

My problem is related.
If @imphil's tox.ini file is structured slightly differently:

[tox]
envlist = py3

[testenv]
deps =
    pytest

[testenv:py3-ci]
# Dependencies only when running in CI,
# which calls effectively "tox -e py3-ci" (see the gh-actions section below).
deps =
    {[testenv]deps}
    pytest-github-actions-annotate-failures

[gh-actions]
# Mapping between the Python version used in GitHub Actions matrix builds, and
# the used Tox environment.
python =
    3.5: py3-ci
    3.6: py3-ci
    3.7: py3-ci
    3.8: py3-ci
    3.9: py3-ci

Now, py3-ci is in envconfigs, but not in envlist or envlist_default. It can be seen with tox -a:

$ tox -l
py3
$ tox -a
py3
py3-ci

I think that in this instance py3-ci should be runnable by tox-gh-actions.

[brechtm]

I now finally understand that tox-gh-actions only considers environments that are listed in envlist in tox.ini. Note that the README doesn't explicitly mention that.

I would like to argue that this doesn't make much sense. envlist is only the default set of environments to run, when TOXENV isn't set or the -e option is not passed to tox on the command line. Also, I think it is normal for the default list of environments to run is only a subset of those run on CI, since many people will not have the complete set of interpreters required installed.

The full list of possible environments could be retrieved from tox internals instead. However, if that doesn't work with the envisaged design, perhaps the [gh-actions] section could accept it's own envlist property which would replace the function of [tox]envlist in the current tox-gh-actions release?

[brechtm]

I just learned about tox's skip_missing_interpreters configuration option. This can provide a solution in some cases, but not for the "py3-ci" case described in this issue's description.

[merwok]

I was coming here to report the same problem 🙂

First, thanks for making this plugin — a small tox config section is much nicer than having to translate 3.9 to py39 in yaml/shell!

I am also used to using envlist to define the default behaviour of tox when run by developers on their machine without any -e param, and without having to configure skip_missing_interpreters (don’t want that for CI). For CI, I expected that defining the workflow jobs and the mapping (3.9: coverage, check) would be enough.

tox is capable of determining the list of valid envs (see tox -a), and it is useful that tox and tox -e ALL can do different things. Could this plugin be modified to use the same code that tox -a does?