.dist-info.virtualenv marker files break importlib.metadata in app-data seeded venvs

[cjolowicz]

Since virtualenv 20.0.0, dist-info directories in virtual environments seeded using the app-data method are accompanied by empty marker files with the same name as the directory plus a .virtualenv suffix:

virtualenv/src/virtualenv/seed/via_app_data/pip_install/base.py

Lines 73 to 76 in c8ac632

	# inject a no-op root element, as workaround for bug added
	# by https://github.com/pypa/pip/commit/c7ae06c79#r35523722
	marker = self._image_dir / "{}.virtualenv".format(self._dist_info.name)
	marker.write_text("")

This breaks importlib.metadata, which picks these files up and attempts to create importlib.metadata.Distribution instances for them. As a result, something like importlib.metadata.version("pip") will return None.

The following session was run in a virtual environment created by virtualenv 20.0.2 inside a python:3.8.1 Docker image:

Python 3.8.1 (default, Feb  2 2020, 08:37:37)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.metadata
>>> for d in importlib.metadata.distributions():
...     name = d.metadata["name"]
...     version = d.metadata["version"]
...     path = d._path
...     print(f"{path=} {name=} {version=}")
...
path=PosixPath('/venv/lib/python3.8/site-packages/pip-20.0.2.dist-info.virtualenv') name=None version=None
path=PosixPath('/venv/lib/python3.8/site-packages/wheel-0.34.2.dist-info.virtualenv') name=None version=None
path=PosixPath('/venv/lib/python3.8/site-packages/pip-20.0.2.dist-info') name='pip' version='20.0.2'
path=PosixPath('/venv/lib/python3.8/site-packages/wheel-0.34.2.dist-info') name='wheel' version='0.34.2'
path=PosixPath('/venv/lib/python3.8/site-packages/setuptools-45.2.0.dist-info') name='setuptools' version='45.2.0'
path=PosixPath('/venv/lib/python3.8/site-packages/setuptools-45.2.0.dist-info.virtualenv') name=None version=None
>>> d = importlib.metadata.distribution("pip")
>>> d.metadata["name"]
>>> importlib.metadata.version("pip")
>>>

This may in fact be an issue in importlib.metadata. I'm reporting it here in the assumption that virtualenv may be able to fix or work around this issue faster, and that it would need to work around the issue anyway for Python 3.8.0 and 3.8.1.

The issue also causes Poetry to break when invoked inside such virtual environments, for example in Nox and tox: python-poetry/poetry#1953. This is currently affecting projects that use Poetry in their CI pipelines.

[gaborbernat]

This indeed is a really bad choice of name on our side; will create a PR to fix this (which will land today).

[cjolowicz]

If I'm not mistaken, the issue only affects the following environments:

• Python 3.8.0 and 3.8.1
• importlib_metadata < 1.4

Python >= 3.8.2rc1 and importlib_metadata >= 1.4 use stricter matching and will not pick up the marker files.

See python/cpython@98b1c0c

Need to check that...

Update:

As expected:

• Python 3.8.2rc1 is not affected
• importlib_metadata 1.4.0 and 1.5.0 are not affected
• importlib_metadata 1.3.0 is affected

[gaborbernat]

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.3/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-3-2020-02-12) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

[cjolowicz]

That fixes it, many thanks :)

[cjolowicz]

If this issue still affects you after upgrading to virtualenv 20.0.3, the reason may be that you have app data generated by an earlier version of 20.0. virtualenv will continue to use this data to generate new environments.

A simple way to resolve this is by deleting the app data directory, for example:

• /Users/jdoe/Library/Application Support/virtualenv (macOS)
• /home/jdoe/.local/share/virtualenv (Linux)
• C:\Users\jdoe\AppData\Local\pypa\virtualenv (Windows 7)

Find out the location of the app data by running the following (preferably inside a virtualenv):

pip install appdirs
python -c 'import appdirs; print(appdirs.user_data_dir(appname="virtualenv", appauthor="pypa"))'

[gaborbernat]

Hmm every version should have it's own app data 🤔it's a bug if not 😊

[cjolowicz]

I still had this after upgrading:

$ ls ~/Library/Application\ Support/virtualenv/seed-v1/3.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/
pip/  pip-20.0.2.dist-info/  pip-20.0.2.dist-info.virtualenv

[gaborbernat]

It's a bug then 😟