Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not all environments shown when 'TOXENV' set #720

Closed
stephenfin opened this issue Dec 19, 2017 · 6 comments · Fixed by #1284
Closed

Not all environments shown when 'TOXENV' set #720

stephenfin opened this issue Dec 19, 2017 · 6 comments · Fixed by #1284

Comments

@stephenfin
Copy link
Contributor

@stephenfin stephenfin commented Dec 19, 2017

Steps to reproduce

  1. Export TOXENV

    export TOXENV=py27
    
  2. List all environments

    tox -av
    

This was seen with the latest, master version of Sphinx.

Expected result

All environments are still listed.

$ tox -a
docs
flake8
mypy
coverage
py27
py34
py35
py36
pypy
du11
du12
du13
du14
pylint

Actual result

Only a limited number of tox envs are shown. These correspond to the envs that are explicitly defined rather than the implicit (?) ones, as seen here

$ tox -a
py27
docs
flake8
mypy
pylint
@martinbroadhurst
Copy link

@martinbroadhurst martinbroadhurst commented Dec 31, 2017

This seems to be the documented behaviour, according to the documentation for envlist:

envlist=CSV

Determining the environment list that tox is to operate on happens in this order
(if any is found, no further lookups are made):

    command line option -eENVLIST
    environment variable TOXENV
    tox.ini file’s envlist

@gaborbernat gaborbernat added this to the 3.5 milestone Sep 18, 2018
@gaborbernat gaborbernat removed this from the 3.5 milestone Oct 8, 2018
@gaborbernat gaborbernat added this to the 3.6 milestone Oct 8, 2018
@gaborbernat gaborbernat removed this from the 3.6 milestone Oct 8, 2018
@gaborbernat gaborbernat removed this from the 3.6 milestone Oct 8, 2018
@gaborbernat gaborbernat added this to the 3.6 milestone Oct 8, 2018
@obestwalter
Copy link
Member

@obestwalter obestwalter commented Nov 2, 2018

Closing this as intended and documented behaviour as pointed out by @martinbroadhurst. I have no idea how to make the docs clearer either, but if somebody has an idea: please open a PR.

@obestwalter
Copy link
Member

@obestwalter obestwalter commented Nov 2, 2018

Sorry ... was a bit too fast here. The behaviour described in envlist docs is not describing what what should be shown, but what should be run if tox is called.

The mismatch of what is shown when TOXENV is set has to do with the way default envs are determined and I am not sure yet if and how this listing behaviour should be fixed, but reopening this for now for further clarification.

@obestwalter obestwalter reopened this Nov 2, 2018
@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Nov 2, 2018

the list existing environments (aka tox -al) has nothing to do with the current to run target list (aka toxenv/envlist).

@obestwalter
Copy link
Member

@obestwalter obestwalter commented Nov 2, 2018

Correct - that is what I was trying to explain after reopening :)

@gaborbernat gaborbernat removed this from the 3.6 milestone Dec 16, 2018
@gaborbernat gaborbernat added this to the 3.7 milestone Dec 16, 2018
@rpkilby
Copy link
Member

@rpkilby rpkilby commented Mar 13, 2019

What's happening is config.envlist is parsed/set here:

config.envlist, all_envs = self._getenvdata(reader, config)

But -a/-l assumes that config.envlist consists of the defaults, which isn't true given the above.

default = config.envlist # this only the defaults


I'm not entirely sure what the correct behavior is. Currently plugins like tox-travis rely on being able to alter the envlist (e.g., only match/run the py37 envs). And it's useful that I can check that restricted, matching list. e.g., with tox-travis, I can run $ TRAVIS=y TRAVIS_PYTHON_VERSION=3.7 tox -l and verify the list of matched envs. On the other hand, this result is also unintuitive per the OP, and it doesn't really make sense outside of tox-travis usage.

My thoughts:

  • Setting config.envlist must remain a method for altering the list of executed test envs.
  • Another config variable could be added for providing the full/reference list that contains all envs that tox is aware of. tox -l would then reference this new variable instead of config.envlist.
  • Possibly add another CLI option to display only the selected envs in the envlist (which would preserve the current functionality). Maybe this could just be a verbose --listenvs-selected option, since this is a somewhat niche use case.
  • Above changes wouldn't seem too complex. The envlist is only referenced in a few places.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants