tox 3.14.2 uses wrong Python interpreters #1484

hynek · 2019-12-14T06:24:36Z

When trying to run a tox env using Python 3.8, it actually uses 3.7 (concretely the system interpreter).

I only noticed it because I used hashes in my lockfiles and my builds started failing due to a lack of a hash to dependency that is need on 3.7 but not on 3.8.

Downgrading to homebrew's 3.14.1 fixes it.

tox -rvv:

using tox.ini: /Users/hynek/Work/rdap-svc/tox.ini (pid 49542)
  removing /Users/hynek/Work/rdap-svc/.tox/log
using tox-3.14.2 from /Users/hynek/.local/pipx/venvs/tox/lib/python3.7/site-packages/tox/__init__.py (pid 49542)
GLOB start: packaging
GLOB sdist-make: /Users/hynek/Work/rdap-svc/setup.py
  removing /Users/hynek/Work/rdap-svc/.tox/dist
[49543] /Users/hynek/Work/rdap-svc$ /Users/hynek/.local/pipx/venvs/tox/bin/python setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/GLOB-0.log
running sdist
running egg_info
writing src/rdap_svc.egg-info/PKG-INFO
writing dependency_links to src/rdap_svc.egg-info/dependency_links.txt
writing top-level names to src/rdap_svc.egg-info/top_level.txt
reading manifest file 'src/rdap_svc.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching '*.txt'
warning: no previously-included files found matching '*.cfg'
warning: no previously-included files found matching '*.rst'
warning: no previously-included files found matching '*.ini'
warning: no previously-included files found matching '*.yml'
warning: no previously-included files found matching '*.yaml'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching 'Makefile'
warning: no previously-included files found matching 'conftest.py'
warning: no previously-included files found matching '.envrc'
warning: no previously-included files found matching 'Dockerfile'
warning: no previously-included files found matching 'test-queries.http'
warning: no previously-included files found matching '.dockerignore'
warning: no previously-included files found matching 'Pipfile*'
warning: no previously-included files found matching '*.toml'
warning: no previously-included files found matching '*.paw'
warning: no previously-included files found matching '*.pot'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'deploy'
no previously-included directories found matching 'requirements'
warning: no files found matching '*.md' under directory 'src'
writing manifest file 'src/rdap_svc.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating rdap-svc-0
creating rdap-svc-0/src
creating rdap-svc-0/src/rdap_svc
creating rdap-svc-0/src/rdap_svc.egg-info
creating rdap-svc-0/src/rdap_svc/templates
copying files to rdap-svc-0...
copying MANIFEST.in -> rdap-svc-0
copying setup.py -> rdap-svc-0
copying src/rdap_svc/__init__.py -> rdap-svc-0/src/rdap_svc
[... MANY LINES... ]
copying src/rdap_svc.egg-info/PKG-INFO -> rdap-svc-0/src/rdap_svc.egg-info
copying src/rdap_svc.egg-info/SOURCES.txt -> rdap-svc-0/src/rdap_svc.egg-info
copying src/rdap_svc.egg-info/dependency_links.txt -> rdap-svc-0/src/rdap_svc.egg-info
copying src/rdap_svc.egg-info/not-zip-safe -> rdap-svc-0/src/rdap_svc.egg-info
copying src/rdap_svc.egg-info/top_level.txt -> rdap-svc-0/src/rdap_svc.egg-info
copying src/rdap_svc/templates/help.html -> rdap-svc-0/src/rdap_svc/templates
Writing rdap-svc-0/setup.cfg
creating '.tox/dist/rdap-svc-0.zip' and adding 'rdap-svc-0' to it
adding 'rdap-svc-0'
adding 'rdap-svc-0/src'
adding 'rdap-svc-0/PKG-INFO'
adding 'rdap-svc-0/MANIFEST.in'
adding 'rdap-svc-0/setup.py'
adding 'rdap-svc-0/setup.cfg'
adding 'rdap-svc-0/src/rdap_svc.egg-info'
adding 'rdap-svc-0/src/rdap_svc'
adding 'rdap-svc-0/src/rdap_svc.egg-info/PKG-INFO'
adding 'rdap-svc-0/src/rdap_svc.egg-info/not-zip-safe'
adding 'rdap-svc-0/src/rdap_svc.egg-info/SOURCES.txt'
adding 'rdap-svc-0/src/rdap_svc.egg-info/top_level.txt'
adding 'rdap-svc-0/src/rdap_svc.egg-info/dependency_links.txt'
[... MANY LINES ...]
removing 'rdap-svc-0' (and everything under it)

GLOB finish: packaging  after 0.43 seconds
copying new sdistfile to '/Users/hynek/.tox/distshare/rdap-svc-0.zip'
package .tmp/package/1/rdap-svc-0.zip links to dist/rdap-svc-0.zip (/Users/hynek/Work/rdap-svc/.tox)
mypy start: getenv /Users/hynek/Work/rdap-svc/.tox/mypy
mypy cannot reuse: -r flag
mypy recreate: /Users/hynek/Work/rdap-svc/.tox/mypy
python3.8 (/Users/hynek/.local/venvs/rdap-svc/bin/python3.8) is {'executable': '/Users/hynek/.local/pipx/venvs/tox/bin/python', 'name': 'python', 'version_info': [3, 8, 0, 'final', 0], 'version': '3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) \n[Clang 6.0 (clang-600.0.57)]', 'is_64': True, 'sysplatform': 'darwin'}
mypy uses /Users/hynek/.local/pipx/venvs/tox/bin/python
  removing /Users/hynek/Work/rdap-svc/.tox/mypy
setting PATH=/Users/hynek/Work/rdap-svc/.tox/mypy/bin:/Users/hynek/.local/venvs/rdap-svc/bin:/Users/hynek/.pyenv/shims:/Users/hynek/.local/bin:/Users/hynek/Library/Python/3.7/bin:/Users/hynek/.poetry/bin:/Users/hynek/.gem/ruby/2.5.0/bin/:/Users/hynek/.gem/ruby/2.6.0/bin/:/usr/local/opt/openssl@1.1/bin:/Users/hynek/Work/go/bin:/Applications/SQLAnywhere17/System/bin64:/usr/local/opt/perl/bin:/Users/hynek/.cargo/bin:/usr/local/opt/gettext/bin:/opt/X11/bin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/MacGPG2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
[49551] /Users/hynek/Work/rdap-svc/.tox$ /Users/hynek/.local/pipx/venvs/tox/bin/python -m virtualenv --no-download --python /Users/hynek/.local/pipx/venvs/tox/bin/python mypy
Already using interpreter /Users/hynek/.local/pipx/venvs/tox/bin/python
Using base prefix '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7'
New python executable in /Users/hynek/Work/rdap-svc/.tox/mypy/bin/python
Installing setuptools, pip, wheel...
done.
mypy installdeps: -rrequirements/main.txt, -rrequirements/dev.txt
setting PATH=/Users/hynek/Work/rdap-svc/.tox/mypy/bin:/Users/hynek/.local/venvs/rdap-svc/bin:/Users/hynek/.pyenv/shims:/Users/hynek/.local/bin:/Users/hynek/Library/Python/3.7/bin:/Users/hynek/.poetry/bin:/Users/hynek/.gem/ruby/2.5.0/bin/:/Users/hynek/.gem/ruby/2.6.0/bin/:/usr/local/opt/openssl@1.1/bin:/Users/hynek/Work/go/bin:/Applications/SQLAnywhere17/System/bin64:/usr/local/opt/perl/bin:/Users/hynek/.cargo/bin:/usr/local/opt/gettext/bin:/opt/X11/bin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/MacGPG2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
[49560] /Users/hynek/Work/rdap-svc$ /Users/hynek/Work/rdap-svc/.tox/mypy/bin/python -m pip install -rrequirements/main.txt -rrequirements/dev.txt
Looking in indexes: https://pypi.vm.ag/root/vrmd/+simple/
Collecting attrs==19.3.0
  Using cached https://pypi.vm.ag/root/pypi/%2Bf/08a/96c641c3a74e4/attrs-19.3.0-py2.py3-none-any.whl
[... MANY LINES ...]
Requirement already satisfied: setuptools>=3.0 in ./.tox/mypy/lib/python3.7/site-packages (from gunicorn[setproctitle]==20.0.4->-r requirements/main.txt (line 41)) (41.6.0)
Collecting importlib-metadata>=0.12; python_version < "3.8"
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    importlib-metadata>=0.12; python_version < "3.8" from https://pypi.vm.ag/root/pypi/%2Bf/d95/141fbfa7ef2ec/importlib_metadata-1.3.0-py2.py3-none-any.whl#sha256=d95141fbfa7ef2ec65cfd945e2af7e5a6ddbd7c8d9a25e66ff3be8e3daf9f60f (from pluggy==0.13.1->-r requirements/dev.txt (line 135))
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [-rrequirements/main.txt, -rrequirements/dev.txt]; v = InvocationError('/Users/hynek/Work/rdap-svc/.tox/mypy/bin/python -m pip install -rrequirements/main.txt -rrequirements/dev.txt', 1)
mypy finish: getenv /Users/hynek/Work/rdap-svc/.tox/mypy after 32.84 seconds
__________________________________________________________________________ summary ___________________________________________________________________________
ERROR:   mypy: could not install deps [-rrequirements/main.txt, -rrequirements/dev.txt]; v = InvocationError('/Users/hynek/Work/rdap-svc/.tox/mypy/bin/python -m pip install -rrequirements/main.txt -rrequirements/dev.txt', 1)
cleanup /Users/hynek/Work/rdap-svc/.tox/.tmp/package/1/rdap-svc-0.zip

Please note:

Using base prefix '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7'

which leads to:

$ /Users/hynek/.local/pipx/venvs/tox/bin/python
Python 3.7.3 (default, Nov 15 2019, 04:04:52)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Please also note that Python 3.8 is the Python interpreter in the active virtualenv (but tox is not ran out of that virtualenv. It uses the Catalina system 3.7 installed using pipx). IOW it uses the tox interpreter, so I suspect a connection here (homebrew's tox also uses 3.7, just not the system's).

Could this be related to #1462?

The text was updated successfully, but these errors were encountered:

gaborbernat · 2019-12-15T11:16:44Z

@hynek can you validate this by using PyPi tox 3.14.1?

Julian · 2019-12-15T12:31:39Z

I'm not sure if this is the same issue, a related one or a different one, but I've had on my list to look into that in the last few months tox has started to run pypy toxenvs on the wrong Python too -- specifically, now if you run tox -e pypy, you get pypy3 rather than pypy2.

asottile · 2019-12-20T06:29:01Z

is there a sample tox.ini that's triggering this, I suspect it's related to an ordering/caching problem and most likely #1462

hynek · 2019-12-20T10:46:16Z

I don't think the tox.ini is the problem. I can only reproduce it if I use Catalina's system Python it seems.

gaborbernat · 2019-12-20T10:56:07Z

I don't have access to Catalin yet (our company doesn't allow upgrading to it yet, deah stability) so can't validate fix. When eventually I'll be able to I can try to check out if you have a project that replicates this in a stable fashion.

gaborbernat · 2020-04-19T09:22:27Z

@hynek I'm 99% this was pypa/virtualenv#1704

asottile · 2020-04-19T16:20:51Z

@gaborbernat I don't think so, conda isn't involved here and it regressed in a tox changeset, pretty sure the culprit is #1462

asottile · 2020-04-19T16:28:36Z

yes I can still reproduce:

/usr/local/bin/pip3 install --user tox
virtualenv venv3 -ppython3.8
. venv3/bin/activate

this tox.ini

[tox]
skipsdist = true
envlist = py37,py38

[testenv]
commands =
    python --version

/path/to/tox -rvv

(venv3) asottile-mac:y asottile$ /Users/asottile/Library/Python/3.7/bin/tox -rvv
using tox.ini: /private/tmp/y/tox.ini (pid 90128)
  removing /private/tmp/y/.tox/log
using tox-3.14.6 from /Users/asottile/Library/Python/3.7/lib/python/site-packages/tox/__init__.py (pid 90128)
skipping sdist step
py37 start: getenv /private/tmp/y/.tox/py37
py37 cannot reuse: -r flag
py37 recreate: /private/tmp/y/.tox/py37
py37 uses /usr/local/opt/python/bin/python3.7
  removing /private/tmp/y/.tox/py37
setting PATH=/private/tmp/y/.tox/py37/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90130] /private/tmp/y/.tox$ /usr/local/opt/python/bin/python3.7 -m virtualenv --no-download --python /usr/local/opt/python/bin/python3.7 py37
created virtual environment CPython3.7.7.final.0-64 in 298ms
  creator CPython3Posix(dest=/private/tmp/y/.tox/py37, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/asottile/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
py37 finish: getenv /private/tmp/y/.tox/py37 after 0.72 seconds
py37 start: finishvenv 
write config to /private/tmp/y/.tox/py37/.tox-config1 as '72f04ae20581db472afbb800b418eeabc5a0b2df5b89e9fb162567cb361519c9 /usr/local/opt/python/bin/python3.7\n3.14.6 0 0 0'
py37 finish: finishvenv  after 0.00 seconds
py37 start: envreport 
setting PATH=/private/tmp/y/.tox/py37/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90132] /private/tmp/y$ /private/tmp/y/.tox/py37/bin/python -m pip freeze >.tox/py37/log/py37-0.log
py37 finish: envreport  after 0.60 seconds
py37 installed: 
py37 start: run-test-pre 
py37 run-test-pre: PYTHONHASHSEED='3851750361'
py37 finish: run-test-pre  after 0.00 seconds
py37 start: run-test 
py37 run-test: commands[0] | python --version
setting PATH=/private/tmp/y/.tox/py37/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90133] /private/tmp/y$ /private/tmp/y/.tox/py37/bin/python --version
Python 3.7.7
py37 finish: run-test  after 0.01 seconds
py37 start: run-test-post 
py37 finish: run-test-post  after 0.00 seconds
py38 start: getenv /private/tmp/y/.tox/py38
py38 cannot reuse: -r flag
py38 recreate: /private/tmp/y/.tox/py38
python3.8 (/private/tmp/y/venv3/bin/python3.8) is {'executable': '/usr/local/Cellar/python/3.7.7/bin/python3.7', 'name': 'python', 'version_info': [3, 8, 0, 'final', 0], 'version': '3.8.0 (default, Jan 13 2020, 10:45:48) \n[Clang 10.0.1 (clang-1001.0.46.4)]', 'is_64': True, 'sysplatform': 'darwin'}
py38 uses /usr/local/Cellar/python/3.7.7/bin/python3.7
  removing /private/tmp/y/.tox/py38
setting PATH=/private/tmp/y/.tox/py38/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90136] /private/tmp/y/.tox$ /usr/local/opt/python/bin/python3.7 -m virtualenv --no-download --python /usr/local/Cellar/python/3.7.7/bin/python3.7 py38
created virtual environment CPython3.7.7.final.0-64 in 295ms
  creator CPython3Posix(dest=/private/tmp/y/.tox/py38, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/asottile/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
py38 finish: getenv /private/tmp/y/.tox/py38 after 0.69 seconds
py38 start: finishvenv 
write config to /private/tmp/y/.tox/py38/.tox-config1 as '72f04ae20581db472afbb800b418eeabc5a0b2df5b89e9fb162567cb361519c9 /usr/local/Cellar/python/3.7.7/bin/python3.7\n3.14.6 0 0 0'
py38 finish: finishvenv  after 0.00 seconds
py38 start: envreport 
setting PATH=/private/tmp/y/.tox/py38/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90138] /private/tmp/y$ /private/tmp/y/.tox/py38/bin/python -m pip freeze >.tox/py38/log/py38-0.log
py38 finish: envreport  after 0.59 seconds
py38 installed: 
py38 start: run-test-pre 
py38 run-test-pre: PYTHONHASHSEED='3851750361'
py38 finish: run-test-pre  after 0.00 seconds
py38 start: run-test 
py38 run-test: commands[0] | python --version
setting PATH=/private/tmp/y/.tox/py38/bin:/private/tmp/y/venv3/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[90139] /private/tmp/y$ /private/tmp/y/.tox/py38/bin/python --version
Python 3.7.7
py38 finish: run-test  after 0.01 seconds
py38 start: run-test-post 
py38 finish: run-test-post  after 0.00 seconds
___________________________________ summary ____________________________________
  py37: commands succeeded
  py38: commands succeeded
  congratulations :)

This also updates the tox minversion to a very recent version in hopes of triggering tox provisioning, which seems to avoid tox-dev/tox#1484

This is intended to remedy these two github issues: tox-dev#882 tox-dev#1484

(Do NOT use basepython as a fallback) This is intended to remedy these two github issues: tox-dev#882 tox-dev#1484 by :user: jackdesert

jackdesert · 2021-04-26T19:25:34Z

If tox keeps using the wrong python interpreter, make sure you are specifying interpreters WITHOUT PERIODS.

See this comment:
#2047 (comment)

holdenk · 2021-06-30T18:08:47Z

I ran into this as well, tox picks up my default system python if it can't find the right Python. This is confusing I would prefer an error from tox saying it can't make a venv with the right Python instead of making a venv with my systems Python 2.7 cause that's just hella confusing.

jugmac00 · 2022-01-05T10:38:44Z

Looks like I am also affected for the first time, though it is a little bit more complicated as containers are involved:
https://code.launchpad.net/~cjwatson/lpcraft/+git/lpcraft/+merge/413628

gaborbernat · 2022-01-14T19:13:35Z

Can someone throw together a docker image to replicate the issue? Then we can move ahead.

gaborbernat · 2022-01-26T11:34:58Z

Closing this until we have a replicate of this via a Docker image, thanks!

GeeWee · 2022-05-25T13:34:49Z

I just wanted to say I came here and had the same issue, but it was because my python versions were specified as e.g 3.7 and not 37

hynek added the bug:normal affects many people or has quite an impact label Dec 14, 2019

webknjaz mentioned this issue Jan 18, 2020

Env recreation during tox --devenv is broken for me #1501

Closed

gaborbernat closed this as completed Apr 19, 2020

asottile reopened this Apr 19, 2020

dmtucker added a commit to dmtucker/pytest-mypy that referenced this issue Oct 11, 2020

Add support for Python 3.9

f04923c

This also updates the tox minversion to a very recent version in hopes of triggering tox provisioning, which seems to avoid tox-dev/tox#1484

dmtucker added a commit to dmtucker/pytest-mypy that referenced this issue Oct 11, 2020

Add support for Python 3.9

9b2154a

This also updates the tox minversion to a very recent version in hopes of triggering tox provisioning, which seems to avoid tox-dev/tox#1484

jackdesert pushed a commit to jackdesert/tox that referenced this issue Apr 26, 2021

Fail if Specified Interpreter Not Found

a24e3ff

This is intended to remedy these two github issues: tox-dev#882 tox-dev#1484

jackdesert mentioned this issue Apr 26, 2021

Fail if Specified Interpreter Not Found #2047

Closed

jackdesert pushed a commit to jackdesert/tox that referenced this issue Apr 26, 2021

Fail if Specified Interpreter Not Found

99c2e54

(Do NOT use basepython as a fallback) This is intended to remedy these two github issues: tox-dev#882 tox-dev#1484 by :user: jackdesert

jackdesert pushed a commit to jackdesert/tox that referenced this issue Apr 26, 2021

Fail if Specified Interpreter Not Found

30642f6

(Do NOT use basepython as a fallback) This is intended to remedy these two github issues: tox-dev#882 tox-dev#1484 by :user: jackdesert

gaborbernat added this to the 4.0 milestone Jan 13, 2022

gaborbernat added the needs:more-info label Jan 14, 2022

gaborbernat removed the bug:normal affects many people or has quite an impact label Jan 14, 2022

gaborbernat closed this as completed Jan 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tox 3.14.2 uses wrong Python interpreters #1484

tox 3.14.2 uses wrong Python interpreters #1484

hynek commented Dec 14, 2019

gaborbernat commented Dec 15, 2019

Julian commented Dec 15, 2019 •

edited

Loading

asottile commented Dec 20, 2019

hynek commented Dec 20, 2019

gaborbernat commented Dec 20, 2019

gaborbernat commented Apr 19, 2020

asottile commented Apr 19, 2020

asottile commented Apr 19, 2020

jackdesert commented Apr 26, 2021

holdenk commented Jun 30, 2021

jugmac00 commented Jan 5, 2022

gaborbernat commented Jan 14, 2022

gaborbernat commented Jan 26, 2022

GeeWee commented May 25, 2022

tox 3.14.2 uses wrong Python interpreters #1484

tox 3.14.2 uses wrong Python interpreters #1484

Comments

hynek commented Dec 14, 2019

gaborbernat commented Dec 15, 2019

Julian commented Dec 15, 2019 • edited Loading

asottile commented Dec 20, 2019

hynek commented Dec 20, 2019

gaborbernat commented Dec 20, 2019

gaborbernat commented Apr 19, 2020

asottile commented Apr 19, 2020

asottile commented Apr 19, 2020

jackdesert commented Apr 26, 2021

holdenk commented Jun 30, 2021

jugmac00 commented Jan 5, 2022

gaborbernat commented Jan 14, 2022

gaborbernat commented Jan 26, 2022

GeeWee commented May 25, 2022

Julian commented Dec 15, 2019 •

edited

Loading