Not all environments shown when 'TOXENV' set

[stephenfin]

Steps to reproduce

1. Export TOXENV

   export TOXENV=py27
   

2. List all environments

   tox -av
   

This was seen with the latest, master version of Sphinx.

Expected result

All environments are still listed.

$ tox -a
docs
flake8
mypy
coverage
py27
py34
py35
py36
pypy
du11
du12
du13
du14
pylint

Actual result

Only a limited number of tox envs are shown. These correspond to the envs that are explicitly defined rather than the implicit (?) ones, as seen here

$ tox -a
py27
docs
flake8
mypy
pylint

[martinbroadhurst]

This seems to be the documented behaviour, according to the documentation for envlist:

envlist=CSV

Determining the environment list that tox is to operate on happens in this order
(if any is found, no further lookups are made):

    command line option -eENVLIST
    environment variable TOXENV
    tox.ini file’s envlist

[obestwalter]

Closing this as intended and documented behaviour as pointed out by @martinbroadhurst. I have no idea how to make the docs clearer either, but if somebody has an idea: please open a PR.

[obestwalter]

Sorry ... was a bit too fast here. The behaviour described in envlist docs is not describing what what should be shown, but what should be run if tox is called.

The mismatch of what is shown when TOXENV is set has to do with the way default envs are determined and I am not sure yet if and how this listing behaviour should be fixed, but reopening this for now for further clarification.

[gaborbernat]

the list existing environments (aka tox -al) has nothing to do with the current to run target list (aka toxenv/envlist).

[obestwalter]

Correct - that is what I was trying to explain after reopening :)

[rpkilby]

What's happening is config.envlist is parsed/set here:

tox/src/tox/config/__init__.py

Line 1037 in f8fe74c

config.envlist, all_envs = self._getenvdata(reader, config)

But -a/-l assumes that config.envlist consists of the defaults, which isn't true given the above.

tox/src/tox/session/commands/show_env.py

Line 8 in 5b68897

default = config.envlist # this only the defaults

---

I'm not entirely sure what the correct behavior is. Currently plugins like tox-travis rely on being able to alter the envlist (e.g., only match/run the py37 envs). And it's useful that I can check that restricted, matching list. e.g., with tox-travis, I can run $ TRAVIS=y TRAVIS_PYTHON_VERSION=3.7 tox -l and verify the list of matched envs. On the other hand, this result is also unintuitive per the OP, and it doesn't really make sense outside of tox-travis usage.

My thoughts:

• Setting config.envlist must remain a method for altering the list of executed test envs.
• Another config variable could be added for providing the full/reference list that contains all envs that tox is aware of. tox -l would then reference this new variable instead of config.envlist.
• Possibly add another CLI option to display only the selected envs in the envlist (which would preserve the current functionality). Maybe this could just be a verbose --listenvs-selected option, since this is a somewhat niche use case.
• Above changes wouldn't seem too complex. The envlist is only referenced in a few places.