[Windows] Unable to inject jaraco.clipboard #528

jaraco · 2020-10-20T15:27:51Z

Describe the bug
Today I'm upgrading my Python 3.8 envs with Python 3.9. reinstall-all failed (found executables missing), so I uninstalled manually and attempted to reinstall. After pipx install 'xonsh[ptk]', I attempted to inject two deps:

PS C:\WINDOWS\system32> pipx inject xonsh jaraco.clipboard
Internal error with venv metadata inspection.
Unable to install jaraco-clipboard.
Check the name or spec for errors, and verify that it can be installed with pip.
←[?25h
PS C:\WINDOWS\system32> pipx inject xonsh keyring
  injected package keyring into venv xonsh
done!
←[?25h

With --verbose:

PS C:\WINDOWS\system32> pipx inject --verbose xonsh jaraco.clipboard
←[?25lpipx > (setup:567): pipx version is 0.15.6.0
pipx > (setup:568): Default python interpreter is 'c:\\program files\\python39\\python.exe'
pipx > (run_pipx_command:135): Virtual Environment location is C:\Users\jaraco\.local\pipx\venvs\xonsh
pipx > (needs_upgrade:54): Time since last upgrade of shared libs, in seconds: 238598.75953483582. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (package_name_from_spec:247): Determined package name: jaraco-clipboard
pipx > (package_name_from_spec:248): Package name determined in 0.0s
pipx > (_parsed_package_to_package_or_url:127): cleaned package spec: jaraco-clipboard
installing jaraco-clipboard...
pipx > (run_subprocess:113): running C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts\python.exe -m pip install jaraco-clipboard
Requirement already satisfied: jaraco-clipboard in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (2.0.1)
Requirement already satisfied: jaraco.windows>=3.4; sys_platform == "win32" in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco-clipboard) (5.0.0)
Requirement already satisfied: six in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (1.15.0)
Requirement already satisfied: jaraco.text in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.2.0)
Requirement already satisfied: jaraco.ui in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (2.0.1)
Requirement already satisfied: jaraco.collections in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.0.0)
Requirement already satisfied: path.py>=6.2 in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (12.5.0)
Requirement already satisfied: jaraco.structures>=1.1.1 in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (2.0)
Requirement already satisfied: more-itertools in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (8.5.0)
Requirement already satisfied: jaraco.functools in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.text->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.0.1)
Requirement already satisfied: jaraco.classes in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from jaraco.ui->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (3.1.0)
Requirement already satisfied: path in c:\users\jaraco\.local\pipx\venvs\xonsh\lib\site-packages (from path.py>=6.2->jaraco.windows>=3.4; sys_platform == "win32"->jaraco-clipboard) (15.0.0)
pipx > (run_subprocess:113): running C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts\python.exe -c <contents of venv_metadata_inspector.py> jaraco-clipboard C:\Users\jaraco\.local\pipx\venvs\xonsh\Scripts
pipx > (get_venv_metadata_for_package:282): Internal error with venv metadata inspection.
pipx > (get_venv_metadata_for_package:283): venv_metadata_inspector.py Traceback:
Traceback (most recent call last):
  File "<string>", line 159, in <module>
  File "<string>", line 120, in main
  File "<string>", line 37, in get_apps
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 465, in get_distribution
    dist = get_provider(dist)
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 341, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "C:\Users\jaraco\.local\pipx\shared\Lib\site-packages\pkg_resources\__init__.py", line 770, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jaraco-clipboard' distribution was not found and is required by the application
pipx > (_parsed_package_to_package_or_url:127): cleaned package spec: jaraco-clipboard
Unable to install jaraco-clipboard.
Check the name or spec for errors, and verify that it can be installed with pip.
←[?25h

Expected behavior
The package should have installed correctly.

It looks like the problem is the ~~mangling~~normalization of the name. The package name is jaraco.clipboard not jaraco-clipboard:

PS C:\WINDOWS\system32> pip-run jaraco-clipboard
Collecting jaraco-clipboard
  Using cached jaraco.clipboard-2.0.1-py2.py3-none-any.whl (4.6 kB)
Collecting jaraco.windows>=3.4; sys_platform == "win32"
  Using cached jaraco.windows-5.0.0-py3-none-any.whl (54 kB)
Collecting jaraco.text
  Using cached jaraco.text-3.2.0-py2.py3-none-any.whl (8.1 kB)
Collecting more-itertools
  Using cached more_itertools-8.5.0-py3-none-any.whl (44 kB)
Collecting jaraco.collections
  Using cached jaraco.collections-3.0.0-py2.py3-none-any.whl (9.4 kB)
Collecting jaraco.ui
  Using cached jaraco.ui-2.0.1-py2.py3-none-any.whl (7.5 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting path.py>=6.2
  Using cached path.py-12.5.0-py3-none-any.whl (2.3 kB)
Collecting jaraco.structures>=1.1.1
  Using cached jaraco.structures-2.0-py2.py3-none-any.whl (4.3 kB)
Collecting jaraco.functools
  Using cached jaraco.functools-3.0.1-py3-none-any.whl (6.7 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.1.0-py2.py3-none-any.whl (5.7 kB)
Collecting path
  Using cached path-15.0.0-py3-none-any.whl (21 kB)
Installing collected packages: six, more-itertools, jaraco.functools, jaraco.text, jaraco.classes, jaraco.collections, jaraco.ui, path, path.py, jaraco.structures, jaraco.windows, jaraco-clipboard
Successfully installed jaraco-clipboard jaraco.classes-3.1.0 jaraco.collections-3.0.0 jaraco.functools-3.0.1 jaraco.structures-2.0 jaraco.text-3.2.0 jaraco.ui-2.0.1 jaraco.windows-5.0.0 more-itertools-8.5.0 path-15.0.0 path.py-12.5.0 six-1.15.0
WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available.
You should consider upgrading via the 'c:\program files\python39\python.exe -m pip install --upgrade pip' command.
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.get_distribution('jaraco-clipboard')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 480, in get_distribution
    dist = get_provider(dist)
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 356, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 899, in require
    needed = self.resolve(parse_requirements(requirements))
  File "c:\program files\python39\lib\site-packages\pkg_resources\__init__.py", line 785, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jaraco-clipboard' distribution was not found and is required by the application
>>> pkg_resources.get_distribution('jaraco.clipboard')
jaraco.clipboard 2.0.1 (c:\users\jaraco\appdata\local\temp\pip-run-ir9ktr_4)
>>>

The text was updated successfully, but these errors were encountered:

jaraco · 2020-10-20T15:35:30Z

I worked around the issue by running ~/.local/pipx/venvs/xonsh/Scripts/python -m pip install jaraco.clipboard.

uranusjr · 2020-10-20T17:45:30Z

I recall dealing with this bug in pip. setuptools normalises package names differently from PEP 503, specifically it does not replace dots with dashes, which likely is the cause.

We should probably switch to importlib-metadata instead.

itsayellow · 2020-10-20T19:52:20Z

This is due to packaging.utils.canonicalize_name, which ironically I was inspired to use packaging because of past @jaraco PR's.

It's really unfortunate that there are multiple ways to normalize package names. ☹️

itsayellow · 2020-10-20T20:10:32Z

So this is confusing me.

PEP503 says

This PEP references the concept of a "normalized" project name. As per PEP 426 the only valid characters in a name are the ASCII alphabet, ASCII numbers, ., -, and _. The name should be lowercased with all runs of the characters ., -, or _ replaced with a single - character.

But PEP426 which it "references" only talks about dashes and underscores being equivalent, not periods:

All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent.

So does this mean that package names can be different by replacing a . with a -? But then using the Simple API they would be considered the same name?

???

uranusjr · 2020-10-20T20:38:10Z

The reference of PEP 426 in PEP 503 is about valid characters in a package name; PEP 503 is the only active PEP that defines package name normalisation. Also note that PEP 426 was withdrawn, so any definition in it is not enforcing anyway.

It's really unfortunate that there are multiple ways to normalize package names. ☹️

There is really only one way to do it: PEP 503. setuptools’s implementation predates the specification, and they did not change the behaviour to match the PEP (likely for legacy reasons).

itsayellow · 2020-10-21T05:15:05Z

So are you suggesting that we should match setuptool's implementation of normalized names, despite the fact that it is "wrong", in the sense that it is not in compliance with PEP503?

itsayellow · 2020-10-21T06:37:53Z

BTW, I hope these comments don't sound irate, I am just trying to make sense of all of this with a healthy dose of humor. 🙂

uranusjr · 2020-10-21T11:31:43Z

No, the other way around, we should either not use, or work around setuptools’s implementation to match the standard.

uranusjr · 2020-10-21T11:40:56Z

The problem right now is we’re using setuptools’s get_distribution to find a distribution matching the name, and that function does not correctly find a package from a normalised name. So we should not use it.

pip uses a lower-level construct, pkg_resources.WorkingSet, to build the distribution mapping with correct name normalisation: https://github.com/pypa/pip/blob/08c99b6e00135ca8df2e98db58aa0b701b971c64/src/pip/_internal/utils/misc.py#L421-L537

Alternatively, Python 3.8+ includes importlib.metadata.distribution(), which correctly implements PEP 503 logic. The same implementation is also available as importlib-metadata on PyPI for earlier Python 3 versions.

itsayellow · 2020-10-21T17:57:30Z

Ah! OK. I see the problem is in venv_metadata_inspector.py, not during the pip install. I was assuming before that pip install was not finding the normalized package name.

itsayellow · 2020-10-21T18:00:36Z

So for python < 3.8 we'd have to install the backport importlib-metadata to make this work I suppose. Which would probably have to be in the shared library venv I'm guessing.

itsayellow · 2020-10-22T06:17:36Z

For reference
#344
#361

itsayellow · 2020-10-24T05:39:39Z

I think I have a venv_metadata_inspector.py version that doesn't use pkg_resources that seems to work across all tests on all platforms.

I just started with @jaraco 's #344 and made a few changes. I think it works really well. I'm doing a lot of metadata debugging to make sure it's giving proper metadata, and it looks good.

I'll probably wait to make it the only change in a pipx version since it's such a core part of pipx.

itsayellow · 2020-10-24T19:20:00Z

Sigh, well while the new venv_metadata_inspector.py using packaging and importlib.metadata seems to be working well, it does not fix this particular issue with a name with a "." in it:

importlib.metadata.PackageNotFoundError: jaraco-clipboard

itsayellow · 2020-10-24T19:25:20Z

Python 3.8.6 (default, Oct  8 2020, 14:06:32) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import metadata
>>> metadata.distribution("jaraco-clipboard")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 504, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: jaraco-clipboard
>>> metadata.distribution("jaraco.clipboard")
<importlib.metadata.PathDistribution object at 0x1014a54c0>
>>>

itsayellow · 2020-10-24T19:31:57Z

@uranusjr, interestingly, if I am reading the pip code correctly _search_distribution canoncializes BOTH the distribution to search for AND every distribution in the list of distributions it gets, before attempting to match:
https://github.com/pypa/pip/blob/08c99b6e00135ca8df2e98db58aa0b701b971c64/src/pip/_internal/utils/misc.py#L489

I guess we can do that too, perhaps as a backup if we get a PackageNotFoundError

uranusjr · 2020-10-24T22:41:53Z

Huh. I read the implementation, and it seems like importlib-metadata also only normalises _, not .. The normalisation method it uses also does not account for repeating separators (e.g. foo--bar). ¹

Some more context. PEP 376 says the .dist-info directory should be named {name}-{version}.dist-info, but does not require either the name and version to be normalised in any way. The PyPA specification says the name should be normalised per PEP 503,² but none of the installers (including pip, conda, and wheel-building tools) actually do that. In practice, that both the name and version parts have their dashes replaced by underscores (to avoid ambiguity), but that’s the end of it. Maybe we should formalise this behaviour as well. We should fix the specification to match reality, and amend the implementation in importlib.metadata.

Edit: I raised this issue to the PyPA.

cc @jaraco

itsayellow · 2020-10-25T18:49:03Z

@jaraco, I just noticed that jaraco.clipboard has no entry points (I believe). Do you have a package with a dot in the name that has some entry points so I can try a pipx install with it?

jaraco · 2020-10-25T19:08:36Z

When you say entry points, I assume you mean console_scripts entry points. The jaraco.financial project has a few.

But you should also note that the reported failure is when attempting to inject jaraco.clipboard (so that library is available to xonsh). So you should be able to replicate the issue by injecting jaraco.clipboard into any pipx environment.

itsayellow · 2020-10-25T19:23:29Z

Right, I understand about the injection. I was just looking for a testcase to make sure the metadata inspector is both not failing with an error AND getting all the pipx-pertinent metadata.

Thanks for the package name.

itsayellow · 2020-11-09T07:21:29Z

An update on this, it's not forgotten! Still finalizing my new venv_metadata_inspector.py code that both fixes this Issue and other issues we didn't even realize we had. 😉

uranusjr mentioned this issue Oct 25, 2020

Clarify .dist-info naming to match reality pypa/packaging.python.org#781

Merged

jaraco changed the title ~~[Windows] Unable to install jaraco.clipboard~~ [Windows] Unable to inject jaraco.clipboard Oct 25, 2020

uranusjr mentioned this issue Oct 25, 2020

Fix dot handling in dist-info directory name python/importlib_metadata#253

Merged

This was referenced Dec 17, 2020

Packages with . in their name are no longer install-able - Internal Error #587

Closed

Overhaul of venv_metadata_inspector.py, now using modern python libraries #588

Closed

itsayellow mentioned this issue Dec 27, 2020

Improved venv inspection #593

Merged

1 task

diazona mentioned this issue Jan 14, 2021

Installing some packages using --index-url fails on pipx 0.15.6.0 #598

Closed

itsayellow closed this as completed in #593 Jan 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Windows] Unable to inject jaraco.clipboard #528

[Windows] Unable to inject jaraco.clipboard #528

jaraco commented Oct 20, 2020

jaraco commented Oct 20, 2020

uranusjr commented Oct 20, 2020 •

edited

Loading

itsayellow commented Oct 20, 2020

itsayellow commented Oct 20, 2020

uranusjr commented Oct 20, 2020 •

edited

Loading

itsayellow commented Oct 21, 2020

itsayellow commented Oct 21, 2020

uranusjr commented Oct 21, 2020

uranusjr commented Oct 21, 2020 •

edited

Loading

itsayellow commented Oct 21, 2020

itsayellow commented Oct 21, 2020 •

edited

Loading

itsayellow commented Oct 22, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020 •

edited

Loading

uranusjr commented Oct 24, 2020 •

edited

Loading

itsayellow commented Oct 25, 2020

jaraco commented Oct 25, 2020 •

edited

Loading

itsayellow commented Oct 25, 2020

itsayellow commented Nov 9, 2020

[Windows] Unable to inject jaraco.clipboard #528

[Windows] Unable to inject jaraco.clipboard #528

Comments

jaraco commented Oct 20, 2020

jaraco commented Oct 20, 2020

uranusjr commented Oct 20, 2020 • edited Loading

itsayellow commented Oct 20, 2020

itsayellow commented Oct 20, 2020

uranusjr commented Oct 20, 2020 • edited Loading

itsayellow commented Oct 21, 2020

itsayellow commented Oct 21, 2020

uranusjr commented Oct 21, 2020

uranusjr commented Oct 21, 2020 • edited Loading

itsayellow commented Oct 21, 2020

itsayellow commented Oct 21, 2020 • edited Loading

itsayellow commented Oct 22, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020

itsayellow commented Oct 24, 2020 • edited Loading

uranusjr commented Oct 24, 2020 • edited Loading

Footnotes

itsayellow commented Oct 25, 2020

jaraco commented Oct 25, 2020 • edited Loading

itsayellow commented Oct 25, 2020

itsayellow commented Nov 9, 2020

uranusjr commented Oct 20, 2020 •

edited

Loading

uranusjr commented Oct 20, 2020 •

edited

Loading

uranusjr commented Oct 21, 2020 •

edited

Loading

itsayellow commented Oct 21, 2020 •

edited

Loading

itsayellow commented Oct 24, 2020 •

edited

Loading

uranusjr commented Oct 24, 2020 •

edited

Loading

jaraco commented Oct 25, 2020 •

edited

Loading