Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`.dist-info.virtualenv` marker files break importlib.metadata in app-data seeded venvs #1589

Closed
cjolowicz opened this issue Feb 12, 2020 · 8 comments · Fixed by #1591
Closed

`.dist-info.virtualenv` marker files break importlib.metadata in app-data seeded venvs #1589

cjolowicz opened this issue Feb 12, 2020 · 8 comments · Fixed by #1591

Comments

@cjolowicz
Copy link

@cjolowicz cjolowicz commented Feb 12, 2020

Since virtualenv 20.0.0, dist-info directories in virtual environments seeded using the app-data method are accompanied by empty marker files with the same name as the directory plus a .virtualenv suffix:

# inject a no-op root element, as workaround for bug added
# by https://github.com/pypa/pip/commit/c7ae06c79#r35523722
marker = self._image_dir / "{}.virtualenv".format(self._dist_info.name)
marker.write_text("")

This breaks importlib.metadata, which picks these files up and attempts to create importlib.metadata.Distribution instances for them. As a result, something like importlib.metadata.version("pip") will return None.

The following session was run in a virtual environment created by virtualenv 20.0.2 inside a python:3.8.1 Docker image:

Python 3.8.1 (default, Feb  2 2020, 08:37:37)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.metadata
>>> for d in importlib.metadata.distributions():
...     name = d.metadata["name"]
...     version = d.metadata["version"]
...     path = d._path
...     print(f"{path=} {name=} {version=}")
...
path=PosixPath('/venv/lib/python3.8/site-packages/pip-20.0.2.dist-info.virtualenv') name=None version=None
path=PosixPath('/venv/lib/python3.8/site-packages/wheel-0.34.2.dist-info.virtualenv') name=None version=None
path=PosixPath('/venv/lib/python3.8/site-packages/pip-20.0.2.dist-info') name='pip' version='20.0.2'
path=PosixPath('/venv/lib/python3.8/site-packages/wheel-0.34.2.dist-info') name='wheel' version='0.34.2'
path=PosixPath('/venv/lib/python3.8/site-packages/setuptools-45.2.0.dist-info') name='setuptools' version='45.2.0'
path=PosixPath('/venv/lib/python3.8/site-packages/setuptools-45.2.0.dist-info.virtualenv') name=None version=None
>>> d = importlib.metadata.distribution("pip")
>>> d.metadata["name"]
>>> importlib.metadata.version("pip")
>>>

This may in fact be an issue in importlib.metadata. I'm reporting it here in the assumption that virtualenv may be able to fix or work around this issue faster, and that it would need to work around the issue anyway for Python 3.8.0 and 3.8.1.

The issue also causes Poetry to break when invoked inside such virtual environments, for example in Nox and tox: python-poetry/poetry#1953. This is currently affecting projects that use Poetry in their CI pipelines.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

This indeed is a really bad choice of name on our side; will create a PR to fix this (which will land today).

@cjolowicz

This comment has been minimized.

Copy link
Author

@cjolowicz cjolowicz commented Feb 12, 2020

If I'm not mistaken, the issue only affects the following environments:

  • Python 3.8.0 and 3.8.1
  • importlib_metadata < 1.4

Python >= 3.8.2rc1 and importlib_metadata >= 1.4 use stricter matching and will not pick up the marker files.

See python/cpython@98b1c0c

Need to check that...

Update:

As expected:

  • Python 3.8.2rc1 is not affected
  • importlib_metadata 1.4.0 and 1.5.0 are not affected
  • importlib_metadata 1.3.0 is affected
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.3/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-3-2020-02-12) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

@cjolowicz

This comment has been minimized.

Copy link
Author

@cjolowicz cjolowicz commented Feb 12, 2020

That fixes it, many thanks :)

@cjolowicz

This comment has been minimized.

Copy link
Author

@cjolowicz cjolowicz commented Feb 12, 2020

If this issue still affects you after upgrading to virtualenv 20.0.3, the reason may be that you have app data generated by an earlier version of 20.0. virtualenv will continue to use this data to generate new environments.

A simple way to resolve this is by deleting the app data directory, for example:

  • /Users/jdoe/Library/Application Support/virtualenv (macOS)
  • /home/jdoe/.local/share/virtualenv (Linux)
  • C:\Users\jdoe\AppData\Local\pypa\virtualenv (Windows 7)

Find out the location of the app data by running the following (preferably inside a virtualenv):

pip install appdirs
python -c 'import appdirs; print(appdirs.user_data_dir(appname="virtualenv", appauthor="pypa"))'
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

Hmm every version should have it's own app data 🤔it's a bug if not 😊

@cjolowicz

This comment has been minimized.

Copy link
Author

@cjolowicz cjolowicz commented Feb 12, 2020

I still had this after upgrading:

$ ls ~/Library/Application\ Support/virtualenv/seed-v1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/
pip/  pip-20.0.2.dist-info/  pip-20.0.2.dist-info.virtualenv
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 12, 2020

It's a bug then 😟

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.