Arch with python-pip: TypeError: expected string or bytes-like object

[GrzegorzKozub]

As soon as I've upgraded Python from 3.8 to 3.9 on Arch Linux I noticed a strange behaviour with all packages that depend on setuptools. What I'll decribe below does NOT happen with Python 3.8 and these packages nor with Python 3.9 and packages that do not depend on setuptools. This is shy I'm reporting this issue here.

1. Have a fresh Python 3.9 installation with no --user packages, meaning ~/.local/bin, ~/.local/lib and ~/.local/include are all empty
2. Install a package that does not depend on setuptools, for example pip install --user vim-vint - installs OK
3. Install the same or any other package that that does not depend on setuptools - installs OK
4. Install any package that depends on setuptools, for example pip install --user locust - installs OK
5. Try installing any package now - always fails with the following error

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 324, in run
    requirement_set = resolver.resolve(
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "/usr/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 437, in check_if_exists
    if not self.req.specifier.contains(existing_version, prereleases=True):
  File "/usr/lib/python3.9/site-packages/packaging/specifiers.py", line 790, in contains
    item = parse(item)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 296, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

At this point you are unable to use pip install because it will always give the above error.

Observation: even though setuptools was originally installed in /usr/lib/python3.9/site-packages/, after we've installed a package that depends on setuptools it was also put in ~/.local/lib/python3.9/site-packages/.

[jaraco]

I attempted to replicate the issue you described with this Dockerfile but was unsuccessful:

FROM archlinux
RUN pacman -Sy python-pip --noconfirm
RUN pip install --user zope.interface
RUN pip install pipx

The install zope.interface line shows this output:

Collecting zope.interface
  Downloading zope.interface-5.2.0-cp39-cp39-manylinux2010_x86_64.whl (241 kB)
Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from zope.interface) (51.0.0)
Installing collected packages: zope.interface
Successfully installed zope.interface-5.2.0

Note, locust doesn't depend on setuptools directly, but does indirectly through zope.interface (and others), so I've used zope.interface in place of locust for faster resolution.

Can you help produce a Dockerfile that replicates the undesirable behavior?

[GrzegorzKozub]

Thanks for looking into it! In fact, and I previously did not think this was important, I'm using --force-reinstall param to perform an update of already installd packages. This param seems to be the culprit here.

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user zope.interface
RUN pip install --user zope.interface
RUN pip install --user --force-reinstall zope.interface
RUN pip install --user zope.interface

Although setuptools is installed globally in /usr/lib/python3.9/site-packages/, as soon as I install anything using --force-reinstall, setuptools gets installed locally into ~/.local/lib/python3.9/site-packages/ and the whole pip install is not usable anymore with any package.

This brings me to a conclusion that I have setuptools 51.0 globally but when I --upgrade it then 51.1 is installed locally and that seems to be breaking pip install:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user --upgrade setuptools
RUN pip install --user lastversion

[jaraco]

Thanks Grzegorz for the repro. With that, I think I've refined the problem a bit further. If you don't install python-pip but instead install pip from source, then also install setuptools using pip (system, then user), the problem doesn't occur:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S python wget --noconfirm
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python
RUN pip install setuptools
RUN pip install --user --upgrade setuptools
RUN pip install lastversion

That leads me to believe that the Arch packaging of pip (or its version) may also be implicated.

[GrzegorzKozub]

Based on that, I feel like it's more Arch package than setuptools itself. I will consider reporting this issue in Arch bug tracker and I think you can close this issue here at your convenience. Thanks for helping out and mery xmas!

[rlaphoenix]

I'm also on arch linux and have been getting this over the past couple of days. Surely it's no coincidence that mainly arch linux users are having the problem?

All that I needed to do was install pip from source rather than from the arch repo: sudo wget https://bootstrap.pypa.io/get-pip.py -O - | python

[jaraco]

To be sure, the issue isn't strictly due to arch's python-pip. Changing the repro to install setuptools<51.1 also bypasses the error, confirming the "Version 51.1" indication in the report.

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user --upgrade 'setuptools<51.1'
RUN pip install --user lastversion

I looked at the changes for 51.1 and there's nothing there that strikes me as particularly implicated. Here's the diff: https://github.com/pypa/setuptools/compare/v51.0.0..v51.1.0

[jaraco]

I wonder if maybe the issue is this empty install_requires directive. I've used that same syntax in other projects with no declared dependencies and never had an issue with it, but it seems plausible this declaration has revealed a defect in setuptools or pip.

[jaraco]

I think I've disconfirmed that theory with this Dockerfile, which still fails:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pacman -S --noconfirm git
RUN git clone --branch workaround/2495-no-install-requires --depth 10 https://github.com/pypa/setuptools
WORKDIR setuptools
RUN python bootstrap.py
RUN pip install --user .
WORKDIR /
CMD pip install --user lastversion

[jaraco]

I also tried cleaning the indentation on the setup.cfg, but that had no effect on the failure either (6ab9f473c87058ffdc88f357d167d7f7b2750154).

[jaraco]

I tried using pdb to debug pip, but unfortunately, pip is unfriendly to debuggers:

[root@61533a76fa1c /]# python -m pdb -m pip install --user lastversion
> /usr/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> from __future__ import absolute_import
(Pdb) c
Collecting lastversion
  Downloading lastversion-1.2.4-py3-none-any.whl (32 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "/usr/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 438, in check_if_exists
    if not self.req.specifier.contains(existing_version, prereleases=True):
  File "/usr/lib/python3.9/site-packages/packaging/specifiers.py", line 790, in contains
    item = parse(item)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 296, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object
The program exited via sys.exit(). Exit status: 2
> /usr/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> from __future__ import absolute_import
(Pdb) q

[jaraco]

I tried installing the exact same versions of pip and setuptools to the system site-packages, and that succeeds without failure:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python wget
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python - pip==20.2.3 setuptools==51.0.0
RUN pip install --user --upgrade setuptools
CMD pip install --user lastversion

So that indicates to me there are two factors at play here:

• Arch packaging of python-pip, and
• Setuptools 51.1 in user-site-packages.

Both are required to trigger the failure.

[jaraco]

I'm able to replicate the issue using a local mounted copy of setuptools for the upgrade install step

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm wget git
RUN pacman -S --noconfirm python python-pip

# pip install --user .
# pip install --user lastversion

I've tried reverting the whole setup.cfg file to the code from v51.0.0 and the problem persists, suggesting that the changes to that file aren't implicated in the issue.

[rlaphoenix]

I will say, what's odd is my system pip location doesn't have setuptools installed (checked via sudo pip freeze | grep setuptools), and can't install it either (due to the issue occurring). But pip install setuptools will, and I'm on the latest pypi version of it too. Quite odd.

[jaraco]

Hmm. I tried pip install --user . against a local checkout of setuptools 51.0.0, and the error remained.

I'm now beginning to think that Setuptools 51.1 isn't implicated... and the only reason I've previously been unable to replicate the issue with 51.0 was because 51.0 was the version installed by python-pip. Indeed, forcing a reinstall in the user directory replicates the issue without involving Setuptools 51.1:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm wget git
RUN pacman -S --noconfirm python python-pip

RUN pip install --user --force-reinstall setuptools==51
RUN pip install --user lastversion

[jaraco]

checked via sudo pip freeze | grep setuptools

Some (all?) versions of pip will hide certain "special" packages, including setuptools. Run pip freeze --all to see those packages.

[rlaphoenix]

checked via sudo pip freeze | grep setuptools

Some (all?) versions of pip will hide certain "special" packages, including setuptools. Run pip freeze --all to see those packages.

You're right, it shows up now. 51.1.0 on both system and user pip locations, with only user pip working.

[jaraco]

I put in some troubleshooting code in req_install.py:

diff --git a/src/pip/_internal/req/req_install.py b/src/pip/_internal/req/req_install.py
index f25cec96a..59d479fd6 100644
--- a/src/pip/_internal/req/req_install.py
+++ b/src/pip/_internal/req/req_install.py
@@ -435,6 +435,12 @@ class InstallRequirement(object):
             return
 
         existing_version = existing_dist.parsed_version
+        
+        try:
+            self.req.specifier.contains(existing_version, prereleases=True)
+        except TypeError:
+            breakpoint()
+
         if not self.req.specifier.contains(existing_version, prereleases=True):
             self.satisfied_by = None
             if use_user_site:

And found these are the values present when the issue occurs:

pip install --user lastversion
Collecting lastversion
  Using cached lastversion-1.2.4-py3-none-any.whl (32 kB)
Collecting PyYAML
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
     |████████████████████████████████| 269 kB 2.2 MB/s 
> /usr/lib/python3.9/site-packages/pip/_internal/req/req_install.py(444)check_if_exists()
-> if not self.req.specifier.contains(existing_version, prereleases=True):
(Pdb) self.req
<Requirement('packaging')>
(Pdb) self.req.specifier
<SpecifierSet('')>
(Pdb) existing_version
<Version('20.8')>

Here's a user reporting a very similar traceback in a very different environment. They never root-caused the issue.

[jaraco]

The same error doesn't occur if I pass the same inputs to the same function:

[root@bb62d5796f95 packaging-20.8-py3.9.egg-info]# python
Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import packaging.requirements
>>> import packaging.version
>>> ver = packaging.version.Version('20.8')
>>> req = packaging.requirements.Requirement('packaging')
>>> req.specifier
<SpecifierSet('')>
>>> req.specifier.contains(ver, prereleases=True)
True

So there's another factor present that I can't yet see.

[jaraco]

Okay, I'm pretty close to understanding the root cause.

Looking at the packaging.specifiers code, I see that the existing_version, even though it's a "Version", is not the same "Version" as found in packaging.specifiers, so fails this check. Back to the Pdb:

(Pdb) import packaging.specifiers
(Pdb) isinstance(existing_version, packaging.specifiers.Version)
False

[jaraco]

Inspecting the class for that existing_version, here's what I see.

(Pdb) existing_version.__class__
<class 'pkg_resources.extern.packaging.version.Version'>

Here's loosely what's happening:

• Arch de-vendors the vendored dependencies in pip and setuptools, one of which is 'packaging'.
• Later, a version of setuptools is installed that includes the vendored dependencies.
• Pip relies on pkg_resources, part of Setuptools, to load the installed package versions, which now defers to the user-installed setuptools with the vendored dependencies.
• Packaging trips over itself when it gets a version parsed from a vendored copy of itself.

I'm not sure why this problem doesn't exhibit when using pip and setuptools installed with their vendored dependencies (why do the two versions of packaging not conflict between pkg_resources._vendor and pip._vendor).

Regardless, what it boils down to is that you can use one form of package management or another, but mixing arch-managed packages and pip-managed packages may encounter this issue.

This issue will likely be addressed when pip moves to relying on importlib_metadata for loading package info (#7413). This issue will also be alleviated if Setuptools could avoid vendoring dependencies, although that effort was tried and failed.

There are probably lots of related issues. I found pypa/setuptools#1383.

I'm not sure there's a lot Setuptools can do here, and it feels like there are a few options pip could take to address it, so I'm transferring this issue over to pip, although potentially it should be handled more broadly at pypa/packaging-problems.

I'll let the pip maintainers decide.

[uranusjr]

There are actually multiple places where pip “string-type” the version to prevent this. I’m not very motivated to work on this, however, since this is very difficult to trigger, and arguably a downstream issue that Arch packagers should be responsible for. Things might be different if this happens in other parts of pip, but the two parts that cause this issue—the legacy resolver, and pkg_resources—are both on their way to retirement, and a fix would become irrelevant soon.

[pradyunsg]

pip is unfriendly to debugger

Wait, what? This is news to me!

[jaraco]

pip is unfriendly to debugger

Wait, what? This is news to me!

I suspect the issue is actually PEBCAK. I wanted to use python -m pdb -m pip to invoke pip under a debugger, but because pip traps exceptions, the debugger doesn't get a chance to intervene. If there were a better entry point that bypassed the exception trapping, or better, if pip could detect that it was running under a debugger and not trap global exceptions, that would make use of pdb much easier.

[pradyunsg]

No worries @jaraco! I don't know if that workflow works. I have personally used only used GUI debuggers with the debugger() callback. :)

---

This is 100% because Arch is debundling pip, despite our advice to not do so in our vendoring policy. Please take this up with the Arch Linux maintainers.

Also, if someone could let them know that I'm requesting them actually vendor stuff in pip, in exchange for not breaking their users in weird ways, that'd be great! :)

[pradyunsg]

Alternatively, users can avoid using their distro-provided pip, and use pip from get-pip.py, which won't be breakable on this manner. :)

[rlaphoenix]

I used the get-pip via sudo, and it still fails to work via sudo pip, but regular pip now works.

[pradyunsg]

Please don't sudo install pip. That's a recipe for breaking your Linux distro. :(

See https://stackoverflow.com/questions/15028648/is-it-acceptable-and-safe-to-run-pip-install-under-sudo for more details.

[YodaEmbedding]

Solution for Arch users

The way I solved this was to uninstall the offending package (setuptools):

pip uninstall setuptools