Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Windows] Unable to inject jaraco.clipboard #528

Closed
jaraco opened this issue Oct 20, 2020 · 21 comments · Fixed by #593
Closed

[Windows] Unable to inject jaraco.clipboard #528

jaraco opened this issue Oct 20, 2020 · 21 comments · Fixed by #593

Comments

@jaraco
Copy link
Member

jaraco commented Oct 20, 2020

Describe the bug
Today I'm upgrading my Python 3.8 envs with Python 3.9. reinstall-all failed (found executables missing), so I uninstalled manually and attempted to reinstall. After pipx install 'xonsh[ptk]', I attempted to inject two deps:

PS C:\WINDOWS\system32> pipx inject xonsh jaraco.clipboard
Internal error with venv metadata inspection.
Unable to install jaraco-clipboard.
Check the name or spec for errors, and verify that it can be installed with pip.
←[?25h
PS C:\WINDOWS\system32> pipx inject xonsh keyring
  injected package keyring into venv xonsh
done!
←[?25h

With --verbose:

PS C:\WINDOWS\system32> pipx inject --verbose xonsh jaraco.clipboard
←[?25lpipx > (setup:567): pipx version is 0.15.6.0
pipx > (setup:568): Default python interpreter is 'c:\\program files\\python39\\python.exe'
pipx > (run_pipx_command:135): Virtual Environment location is C:\Users\jaraco\.local\pipx\venvs\xonsh
pipx > (needs_upgrade:54): Time since last upgrade of shared libs, in seconds: 238598.75953483582. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (package_name_from_spec:247): Determined package name: jaraco-clipboard
pipx > (package_name_from_spec:248): Package name determined in 0.0s
pipx > (_parsed_package_to_package_or_url:127): cleaned package spec: jaraco-clipboard
installing jaraco-clipboard...
pipx > (run_subprocess:113): running C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts\python.exe -m pip install jaraco-clipboard
Requirement already satisfied: jaraco-clipboard in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (2.0.1)
Requirement already satisfied: jaraco.windows>=3.4; sys_platform == "win32" in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco-clipboard) (5.0.0)
Requirement already satisfied: six in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (1.15.0)
Requirement already satisfied: jaraco.text in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.2.0)
Requirement already satisfied: jaraco.ui in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (2.0.1)
Requirement already satisfied: jaraco.collections in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.0.0)
Requirement already satisfied: path.py>=6.2 in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (12.5.0)
Requirement already satisfied: jaraco.structures>=1.1.1 in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (2.0)
Requirement already satisfied: more-itertools in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (8.5.0)
Requirement already satisfied: jaraco.functools in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.text->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.0.1)
Requirement already satisfied: jaraco.classes in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.ui->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.1.0)
Requirement already satisfied: path in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from path.py>=6.2->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (15.0.0)
pipx > (run_subprocess:113): running C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts\python.exe -c <contents of venv_metadata_inspector.py> jaraco-clipboard C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts
pipx > (get_venv_metadata_for_package:282): Internal error with venv metadata inspection.
pipx > (get_venv_metadata_for_package:283): venv_metadata_inspector.py Traceback:
Traceback (most recent call last):
  File "<string>", line 159, in <module>
  File "<string>", line 120, in main
  File "<string>", line 37, in get_apps
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 465, in get_distribution
    dist = get_provider(dist)
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 341, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 770, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jaraco-clipboard' distribution was not found and is required by the application
pipx > (_parsed_package_to_package_or_url:127): cleaned package spec: jaraco-clipboard
Unable to install jaraco-clipboard.
Check the name or spec for errors, and verify that it can be installed with pip.
←[?25h

Expected behavior
The package should have installed correctly.

It looks like the problem is the manglingnormalization of the name. The package name is jaraco.clipboard not jaraco-clipboard:

PS C:\WINDOWS\system32> pip-run jaraco-clipboard
Collecting jaraco-clipboard
  Using cached jaraco.clipboard-2.0.1-py2.py3-none-any.whl (4.6 kB)
Collecting jaraco.windows>=3.4; sys_platform == "win32"
  Using cached jaraco.windows-5.0.0-py3-none-any.whl (54 kB)
Collecting jaraco.text
  Using cached jaraco.text-3.2.0-py2.py3-none-any.whl (8.1 kB)
Collecting more-itertools
  Using cached more_itertools-8.5.0-py3-none-any.whl (44 kB)
Collecting jaraco.collections
  Using cached jaraco.collections-3.0.0-py2.py3-none-any.whl (9.4 kB)
Collecting jaraco.ui
  Using cached jaraco.ui-2.0.1-py2.py3-none-any.whl (7.5 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting path.py>=6.2
  Using cached path.py-12.5.0-py3-none-any.whl (2.3 kB)
Collecting jaraco.structures>=1.1.1
  Using cached jaraco.structures-2.0-py2.py3-none-any.whl (4.3 kB)
Collecting jaraco.functools
  Using cached jaraco.functools-3.0.1-py3-none-any.whl (6.7 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.1.0-py2.py3-none-any.whl (5.7 kB)
Collecting path
  Using cached path-15.0.0-py3-none-any.whl (21 kB)
Installing collected packages: six, more-itertools, jaraco.functools, jaraco.text, jaraco.classes, jaraco.collections, jaraco.ui, path, path.py, jaraco.structures, jaraco.windows, jaraco-clipboard
Successfully installed jaraco-clipboard jaraco.classes-3.1.0 jaraco.collections-3.0.0 jaraco.functools-3.0.1 jaraco.structures-2.0 jaraco.text-3.2.0 jaraco.ui-2.0.1 jaraco.windows-5.0.0 more-itertools-8.5.0 path-15.0.0 path.py-12.5.0 six-1.15.0
WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available.
You should consider upgrading via the 'c:\program files\python39\python.exe -m pip install --upgrade pip' command.
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.get_distribution('jaraco-clipboard')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 480, in get_distribution
    dist = get_provider(dist)
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 356, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 899, in require
    needed = self.resolve(parse_requirements(requirements))
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 785, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jaraco-clipboard' distribution was not found and is required by the application
>>> pkg_resources.get_distribution('jaraco.clipboard')
jaraco.clipboard 2.0.1 (c:\users\jaraco\appdata\local\temp\pip-run-ir9ktr_4)
>>>
@jaraco
Copy link
Member Author

jaraco commented Oct 20, 2020

I worked around the issue by running ~/.local/pipx/venvs/xonsh/Scripts/python -m pip install jaraco.clipboard.

@uranusjr
Copy link
Member

uranusjr commented Oct 20, 2020

I recall dealing with this bug in pip. setuptools normalises package names differently from PEP 503, specifically it does not replace dots with dashes, which likely is the cause.

We should probably switch to importlib-metadata instead.

@itsayellow
Copy link
Contributor

This is due to packaging.utils.canonicalize_name, which ironically I was inspired to use packaging because of past @jaraco PR's.

It's really unfortunate that there are multiple ways to normalize package names. ☹️

@itsayellow
Copy link
Contributor

So this is confusing me.

PEP503 says

This PEP references the concept of a "normalized" project name. As per PEP 426 the only valid characters in a name are the ASCII alphabet, ASCII numbers, ., -, and _. The name should be lowercased with all runs of the characters ., -, or _ replaced with a single - character.

But PEP426 which it "references" only talks about dashes and underscores being equivalent, not periods:

All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent.

So does this mean that package names can be different by replacing a . with a -? But then using the Simple API they would be considered the same name?

???

@uranusjr
Copy link
Member

uranusjr commented Oct 20, 2020

The reference of PEP 426 in PEP 503 is about valid characters in a package name; PEP 503 is the only active PEP that defines package name normalisation. Also note that PEP 426 was withdrawn, so any definition in it is not enforcing anyway.

It's really unfortunate that there are multiple ways to normalize package names. ☹️

There is really only one way to do it: PEP 503. setuptools’s implementation predates the specification, and they did not change the behaviour to match the PEP (likely for legacy reasons).

@itsayellow
Copy link
Contributor

So are you suggesting that we should match setuptool's implementation of normalized names, despite the fact that it is "wrong", in the sense that it is not in compliance with PEP503?

@itsayellow
Copy link
Contributor

BTW, I hope these comments don't sound irate, I am just trying to make sense of all of this with a healthy dose of humor. 🙂

@uranusjr
Copy link
Member

No, the other way around, we should either not use, or work around setuptools’s implementation to match the standard.

@uranusjr
Copy link
Member

uranusjr commented Oct 21, 2020

The problem right now is we’re using setuptools’s get_distribution to find a distribution matching the name, and that function does not correctly find a package from a normalised name. So we should not use it.

pip uses a lower-level construct, pkg_resources.WorkingSet, to build the distribution mapping with correct name normalisation: https://github.com/pypa/pip/blob/08c99b6e00135ca8df2e98db58aa0b701b971c64/src/pip/_internal/utils/misc.py#L421-L537

Alternatively, Python 3.8+ includes importlib.metadata.distribution(), which correctly implements PEP 503 logic. The same implementation is also available as importlib-metadata on PyPI for earlier Python 3 versions.

@itsayellow
Copy link
Contributor

Ah! OK. I see the problem is in venv_metadata_inspector.py, not during the pip install. I was assuming before that pip install was not finding the normalized package name.

@itsayellow
Copy link
Contributor

itsayellow commented Oct 21, 2020

So for python < 3.8 we'd have to install the backport importlib-metadata to make this work I suppose. Which would probably have to be in the shared library venv I'm guessing.

@itsayellow
Copy link
Contributor

For reference
#344
#361

@itsayellow
Copy link
Contributor

I think I have a venv_metadata_inspector.py version that doesn't use pkg_resources that seems to work across all tests on all platforms.

I just started with @jaraco 's #344 and made a few changes. I think it works really well. I'm doing a lot of metadata debugging to make sure it's giving proper metadata, and it looks good.

I'll probably wait to make it the only change in a pipx version since it's such a core part of pipx.

@itsayellow
Copy link
Contributor

Sigh, well while the new venv_metadata_inspector.py using packaging and importlib.metadata seems to be working well, it does not fix this particular issue with a name with a "." in it:

importlib.metadata.PackageNotFoundError: jaraco-clipboard

@itsayellow
Copy link
Contributor

Python 3.8.6 (default, Oct  8 2020, 14:06:32) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import metadata
>>> metadata.distribution("jaraco-clipboard")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 504, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: jaraco-clipboard
>>> metadata.distribution("jaraco.clipboard")
<importlib.metadata.PathDistribution object at 0x1014a54c0>
>>> 

@itsayellow
Copy link
Contributor

itsayellow commented Oct 24, 2020

@uranusjr, interestingly, if I am reading the pip code correctly _search_distribution canoncializes BOTH the distribution to search for AND every distribution in the list of distributions it gets, before attempting to match:
https://github.com/pypa/pip/blob/08c99b6e00135ca8df2e98db58aa0b701b971c64/src/pip/_internal/utils/misc.py#L489

I guess we can do that too, perhaps as a backup if we get a PackageNotFoundError

@uranusjr
Copy link
Member

uranusjr commented Oct 24, 2020

Huh. I read the implementation, and it seems like importlib-metadata also only normalises _, not .. The normalisation method it uses also does not account for repeating separators (e.g. foo--bar). 1

Some more context. PEP 376 says the .dist-info directory should be named {name}-{version}.dist-info, but does not require either the name and version to be normalised in any way. The PyPA specification says the name should be normalised per PEP 503,2 but none of the installers (including pip, conda, and wheel-building tools) actually do that. In practice, that both the name and version parts have their dashes replaced by underscores (to avoid ambiguity), but that’s the end of it. Maybe we should formalise this behaviour as well. We should fix the specification to match reality, and amend the implementation in importlib.metadata.

Edit: I raised this issue to the PyPA.

cc @jaraco

Footnotes

  1. https://github.com/python/importlib_metadata/blob/5b73f25b7b895354d92a6ec8073a373ff225b13c/importlib_metadata/__init__.py#L497

  2. https://packaging.python.org/specifications/recording-installed-packages/#the-dist-info-directory

@itsayellow
Copy link
Contributor

@jaraco, I just noticed that jaraco.clipboard has no entry points (I believe). Do you have a package with a dot in the name that has some entry points so I can try a pipx install with it?

@jaraco
Copy link
Member Author

jaraco commented Oct 25, 2020

When you say entry points, I assume you mean console_scripts entry points. The jaraco.financial project has a few.

But you should also note that the reported failure is when attempting to inject jaraco.clipboard (so that library is available to xonsh). So you should be able to replicate the issue by injecting jaraco.clipboard into any pipx environment.

@jaraco jaraco changed the title [Windows] Unable to install jaraco.clipboard [Windows] Unable to inject jaraco.clipboard Oct 25, 2020
@itsayellow
Copy link
Contributor

Right, I understand about the injection. I was just looking for a testcase to make sure the metadata inspector is both not failing with an error AND getting all the pipx-pertinent metadata.

Thanks for the package name.

@itsayellow
Copy link
Contributor

An update on this, it's not forgotten! Still finalizing my new venv_metadata_inspector.py code that both fixes this Issue and other issues we didn't even realize we had. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants