Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting environments not found error when running a partial set of factors #3096

Closed
atugushev opened this issue Aug 17, 2023 · 12 comments · Fixed by #3099
Closed

Getting environments not found error when running a partial set of factors #3096

atugushev opened this issue Aug 17, 2023 · 12 comments · Fixed by #3099

Comments

@atugushev
Copy link

atugushev commented Aug 17, 2023

Issue

Since 4.9.0 having env_list=a-b in tox.ini and running top -e a raises a error:

HandledError| provided environments not found in configuration file: ['a']

This error was caught today on on pip-tools' CI (jazzband/pip-tools#1963), see the log. It's likely a regression introduced in #3089.

Environment

Provide at least:

  • OS: Ubuntu
Output of pip list of the host Python, where tox is installed
Package       Version
------------- -------
cachetools    5.3.1
chardet       5.2.0
colorama      0.4.6
distlib       0.3.7
filelock      3.12.2
packaging     23.1
pip           22.3.1
platformdirs  3.10.0
pluggy        1.2.0
pyproject-api 1.5.4
setuptools    65.5.1
tox           4.9.0
virtualenv    20.24.3
wheel         0.40.0

Output of running tox

Output of tox -rvv
a-b: 266 W remove tox env folder /.tox/a-b [tox/tox_env/api.py:322]
a-b: 307 I find interpreter for spec PythonSpec(path=/usr/local/bin/python) [virtualenv/discovery/builtin.py:58]
a-b: 307 I proposed PythonInfo(spec=CPython3.11.3.final.0-64, exe=/usr/local/bin/python, platform=linux, version='3.11.3 (main, May  3 2023, 08:13:29) [GCC 10.2.1 20210110]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
a-b: 308 D accepted PythonInfo(spec=CPython3.11.3.final.0-64, exe=/usr/local/bin/python, platform=linux, version='3.11.3 (main, May  3 2023, 08:13:29) [GCC 10.2.1 20210110]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:67]
a-b: 310 D filesystem is case-sensitive [virtualenv/info.py:26]
a-b: 349 I create virtual environment via CPython3Posix(dest=/.tox/a-b, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:50]
a-b: 350 D create folder /.tox/a-b/bin [virtualenv/util/path/_sync.py:12]
a-b: 350 D create folder /.tox/a-b/lib/python3.11/site-packages [virtualenv/util/path/_sync.py:12]
a-b: 351 D write /.tox/a-b/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:32]
a-b: 351 D 	home = /usr/local/bin [virtualenv/create/pyenv_cfg.py:36]
a-b: 351 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:36]
a-b: 352 D 	version_info = 3.11.3.final.0 [virtualenv/create/pyenv_cfg.py:36]
a-b: 352 D 	virtualenv = 20.24.3 [virtualenv/create/pyenv_cfg.py:36]
a-b: 352 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:36]
a-b: 353 D 	base-prefix = /usr/local [virtualenv/create/pyenv_cfg.py:36]
a-b: 353 D 	base-exec-prefix = /usr/local [virtualenv/create/pyenv_cfg.py:36]
a-b: 353 D 	base-executable = /usr/local/bin/python [virtualenv/create/pyenv_cfg.py:36]
a-b: 354 D symlink /usr/local/bin/python to /.tox/a-b/bin/python [virtualenv/util/path/_sync.py:32]
a-b: 355 D create virtualenv import hook file /.tox/a-b/lib/python3.11/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:91]
a-b: 355 D create /.tox/a-b/lib/python3.11/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:94]
a-b: 356 D ============================== target debug ============================== [virtualenv/run/session.py:52]
a-b: 356 D debug via /.tox/a-b/bin/python /usr/local/lib/python3.11/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:200]
a-b: 356 D {
  "sys": {
    "executable": "/.tox/a-b/bin/python",
    "_base_executable": "/usr/local/bin/python3.11",
    "prefix": "/.tox/a-b",
    "base_prefix": "/usr/local",
    "real_prefix": null,
    "exec_prefix": "/.tox/a-b",
    "base_exec_prefix": "/usr/local",
    "path": [
      "/usr/local/lib/python311.zip",
      "/usr/local/lib/python3.11",
      "/usr/local/lib/python3.11/lib-dynload",
      "/.tox/a-b/lib/python3.11/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.11.3 (main, May  3 2023, 08:13:29) [GCC 10.2.1 20210110]",
  "makefile_filename": "/usr/local/lib/python3.11/config-3.11-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' (frozen)>",
  "site": "<module 'site' (frozen)>",
  "datetime": "<module 'datetime' from '/usr/local/lib/python3.11/datetime.py'>",
  "math": "<module 'math' from '/usr/local/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/usr/local/lib/python3.11/json/__init__.py'>"
} [virtualenv/run/session.py:53]
a-b: 400 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv) [virtualenv/run/session.py:57]
a-b: 406 D install pip from wheel /usr/local/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
a-b: 406 D install wheel from wheel /usr/local/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/wheel-0.41.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
a-b: 408 D install setuptools from wheel /usr/local/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/setuptools-68.0.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
a-b: 414 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip to /.tox/a-b/lib/python3.11/site-packages/pip [virtualenv/util/path/_sync.py:40]
a-b: 416 D copy /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools-68.0.0.virtualenv to /.tox/a-b/lib/python3.11/site-packages/setuptools-68.0.0.virtualenv [virtualenv/util/path/_sync.py:40]
a-b: 417 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.1-py3-none-any/wheel-0.41.1.dist-info to /.tox/a-b/lib/python3.11/site-packages/wheel-0.41.1.dist-info [virtualenv/util/path/_sync.py:40]
a-b: 420 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/pkg_resources to /.tox/a-b/lib/python3.11/site-packages/pkg_resources [virtualenv/util/path/_sync.py:40]
a-b: 441 D copy /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.1-py3-none-any/wheel-0.41.1.virtualenv to /.tox/a-b/lib/python3.11/site-packages/wheel-0.41.1.virtualenv [virtualenv/util/path/_sync.py:40]
a-b: 443 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.1-py3-none-any/wheel to /.tox/a-b/lib/python3.11/site-packages/wheel [virtualenv/util/path/_sync.py:40]
a-b: 549 D generated console scripts wheel3 wheel3.11 wheel-3.11 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
a-b: 559 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools-68.0.0.dist-info to /.tox/a-b/lib/python3.11/site-packages/setuptools-68.0.0.dist-info [virtualenv/util/path/_sync.py:40]
a-b: 567 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools to /.tox/a-b/lib/python3.11/site-packages/setuptools [virtualenv/util/path/_sync.py:40]
a-b: 789 D copy /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/distutils-precedence.pth to /.tox/a-b/lib/python3.11/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:40]
a-b: 791 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/_distutils_hack to /.tox/a-b/lib/python3.11/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:40]
a-b: 796 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
a-b: 868 D copy directory /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip-23.2.1.dist-info to /.tox/a-b/lib/python3.11/site-packages/pip-23.2.1.dist-info [virtualenv/util/path/_sync.py:40]
a-b: 871 D copy /root/.local/share/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip-23.2.1.virtualenv to /.tox/a-b/lib/python3.11/site-packages/pip-23.2.1.virtualenv [virtualenv/util/path/_sync.py:40]
a-b: 873 D generated console scripts pip-3.11 pip3 pip pip3.11 [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
a-b: 873 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:63]
a-b: 878 D write /.tox/a-b/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:32]
a-b: 878 D 	home = /usr/local/bin [virtualenv/create/pyenv_cfg.py:36]
a-b: 878 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:36]
a-b: 878 D 	version_info = 3.11.3.final.0 [virtualenv/create/pyenv_cfg.py:36]
a-b: 879 D 	virtualenv = 20.24.3 [virtualenv/create/pyenv_cfg.py:36]
a-b: 879 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:36]
a-b: 879 D 	base-prefix = /usr/local [virtualenv/create/pyenv_cfg.py:36]
a-b: 879 D 	base-exec-prefix = /usr/local [virtualenv/create/pyenv_cfg.py:36]
a-b: 880 D 	base-executable = /usr/local/bin/python [virtualenv/create/pyenv_cfg.py:36]
a-b: 882 W commands_pre[0]> python -c 'print("a")' [tox/tox_env/api.py:427]
a
a-b: 916 I exit 0 (0.03 seconds) /> python -c 'print("a")' pid=556 [tox/execute/api.py:279]
a-b: 917 W commands_pre[1]> python -c 'print("b")' [tox/tox_env/api.py:427]
b
a-b: 946 I exit 0 (0.03 seconds) /> python -c 'print("b")' pid=562 [tox/execute/api.py:279]
a-b: 947 W commands[0]> python -c 'print("ok")' [tox/tox_env/api.py:427]
ok
a-b: 976 I exit 0 (0.03 seconds) /> python -c 'print("ok")' pid=568 [tox/execute/api.py:279]
  a-b: OK (0.71=setup[0.62]+cmd[0.03,0.03,0.03] seconds)
  congratulations :) (0.80 seconds)

Minimal example

  1. tox.ini:
# tox.ini
[tox]
env_list=a-b

[testenv]
package=skip
commands_pre =
  a: python -c 'print("a")'
  b: python -c 'print("b")'
commands=python -c 'print("ok")'
  1. run tox:
$ tox -e a
ROOT: HandledError| provided environments not found in configuration file: ['a']

Expected result

$ pip install tox==4.8.0
Successfully installed tox-4.8.0

$ tox -e a
a: commands_pre[0]> python -c 'print("a")'
a
a: commands[0]> python -c 'print("ok")'
ok
  a: OK (0.11=setup[0.05]+cmd[0.03,0.03] seconds)
  congratulations :) (0.19 seconds)
@tjsmart
Copy link
Contributor

tjsmart commented Aug 18, 2023

In the current version you would have to specify a as an environment in your env_list:

Contents of tox.ini:

[tox]
env_list=a-b
    a-b
    a
    b

[testenv]
package=skip
commands_pre =
  a: python -c 'print("a")'
  b: python -c 'print("b")'
commands=python -c 'print("ok")'

And then a is an allowed environment name:

$ tox -e a
a: commands_pre[0]> python -c 'print("a")'
a
a: commands[0]> python -c 'print("ok")'
ok
  a: OK (0.10=setup[0.04]+cmd[0.03,0.03] seconds)
  congratulations :) (0.15 seconds)

@tjsmart
Copy link
Contributor

tjsmart commented Aug 18, 2023

But to be fair, you're use case seems perfectly valid to me. Perhaps we can improve it to be smarter about these situations.

QRemy added a commit to Astro-Kirsty/gammapy that referenced this issue Aug 18, 2023
see tox-dev/tox#3096
Signed-off-by:  <quentin.remy@live.fr>
@gaborbernat
Copy link
Member

PR welcome 👍

@BeyondEvil
Copy link
Contributor

I'll highjack this issue since I believe while my issue is not exactly the same, it's too closely related to justify a separate issue.

This, I believe is valid Tox config, but now fails:

[tox]
env_list={3.9}

[testenv]
package = skip
commands =
  cov: python -c 'print("with cov")'
  !cov: python -c 'print("without cov")'

Output:

❯ tox -e 3.9
3.9: commands[0]> python -c 'print("without cov")'
without cov
  3.9: OK (0.23=setup[0.21]+cmd[0.02] seconds)
  congratulations :) (0.27 seconds)

❯ tox -e 3.9-cov
ROOT: HandledError| provided environments not found in configuration file: ['3.9-cov']

@gaborbernat
Copy link
Member

@BeyondEvil PR welcome 👍

@BeyondEvil
Copy link
Contributor

@BeyondEvil PR welcome 👍

What was the reason for the PR that broke this? It doesn't seem to be related to an issue.

@gaborbernat
Copy link
Member

Because we no longer allow arbitrary tox env creation, and 3.9-cov is not defined. Note it would work as py3.9-cov... but I agree we should accept both pyx.y and x.y as valid on-demand tox envs.

@BeyondEvil
Copy link
Contributor

Because we no longer allow arbitrary tox env creation,

Just out of curiosity, why not?

@gaborbernat
Copy link
Member

Because people were complaining that typos are also accepted... tox r -e release vs tox r -e releas 👍

@BeyondEvil
Copy link
Contributor

Because people were complaining that typos are also accepted... tox r -e release vs tox r -e releas 👍

Ah, fair enough. 😊

@atugushev
Copy link
Author

atugushev commented Aug 20, 2023

In the current version you would have to specify a as an environment in your env_list:
...
But to be fair, you're use case seems perfectly valid to me. Perhaps we can improve it to be smarter about these situations.

@tjsmart, thank you! I am now leaning toward the idea that the current behavior is more accurate than the previous one. It seems more sensible to have explicitly declared environments, as opposed to running partial factors which could lead to unexpected issues.

Take pip-tools' environments for example:

[tox]
envlist = py{37,38,39,310,311,312,py3}-pip{previous,latest,main}-coverage

Previously, we could simply run tox -e coverage without knowing which Python and pip versions it would use. That's not very precise, is it?

But, consider this example where we know exactly what environments to run and what to expect from tox:

[tox]
envlist = 
     py{37,38,39,310,311,312,py3}-pip{previous,latest,main}-coverage
     py{37,38,39,310,311,312,py3}-pip{previous,latest,main}
     pip{previous,latest,main}

I'm interested to hear what @gaborbernat thinks about this.

@gaborbernat
Copy link
Member

#3099 will fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants