Arch with python-pip: TypeError: expected string or bytes-like object

[GrzegorzKozub]

As soon as I've upgraded Python from 3.8 to 3.9 on Arch Linux I noticed a strange behaviour with all packages that depend on setuptools. What I'll decribe below does NOT happen with Python 3.8 and these packages nor with Python 3.9 and packages that do not depend on setuptools. This is shy I'm reporting this issue here.

1. Have a fresh Python 3.9 installation with no --user packages, meaning ~/.local/bin, ~/.local/lib and ~/.local/include are all empty
2. Install a package that does not depend on setuptools, for example pip install --user vim-vint - installs OK
3. Install the same or any other package that that does not depend on setuptools - installs OK
4. Install any package that depends on setuptools, for example pip install --user locust - installs OK
5. Try installing any package now - always fails with the following error

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 324, in run
    requirement_set = resolver.resolve(
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "/usr/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 437, in check_if_exists
    if not self.req.specifier.contains(existing_version, prereleases=True):
  File "/usr/lib/python3.9/site-packages/packaging/specifiers.py", line 790, in contains
    item = parse(item)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 296, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

At this point you are unable to use pip install because it will always give the above error.

Observation: even though setuptools was originally installed in /usr/lib/python3.9/site-packages/, after we've installed a package that depends on setuptools it was also put in ~/.local/lib/python3.9/site-packages/.

[jaraco]

I attempted to replicate the issue you described with this Dockerfile but was unsuccessful:

FROM archlinux
RUN pacman -Sy python-pip --noconfirm
RUN pip install --user zope.interface
RUN pip install pipx

The install zope.interface line shows this output:

Collecting zope.interface
  Downloading zope.interface-5.2.0-cp39-cp39-manylinux2010_x86_64.whl (241 kB)
Requirement already satisfied: setuptools in /usr/lib/python3.9/site-packages (from zope.interface) (51.0.0)
Installing collected packages: zope.interface
Successfully installed zope.interface-5.2.0

Note, locust doesn't depend on setuptools directly, but does indirectly through zope.interface (and others), so I've used zope.interface in place of locust for faster resolution.

Can you help produce a Dockerfile that replicates the undesirable behavior?

[GrzegorzKozub]

Thanks for looking into it! In fact, and I previously did not think this was important, I'm using --force-reinstall param to perform an update of already installd packages. This param seems to be the culprit here.

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user zope.interface
RUN pip install --user zope.interface
RUN pip install --user --force-reinstall zope.interface
RUN pip install --user zope.interface

Although setuptools is installed globally in /usr/lib/python3.9/site-packages/, as soon as I install anything using --force-reinstall, setuptools gets installed locally into ~/.local/lib/python3.9/site-packages/ and the whole pip install is not usable anymore with any package.

This brings me to a conclusion that I have setuptools 51.0 globally but when I --upgrade it then 51.1 is installed locally and that seems to be breaking pip install:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user --upgrade setuptools
RUN pip install --user lastversion

[jaraco]

Thanks Grzegorz for the repro. With that, I think I've refined the problem a bit further. If you don't install python-pip but instead install pip from source, then also install setuptools using pip (system, then user), the problem doesn't occur:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S python wget --noconfirm
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python
RUN pip install setuptools
RUN pip install --user --upgrade setuptools
RUN pip install lastversion

That leads me to believe that the Arch packaging of pip (or its version) may also be implicated.

[GrzegorzKozub]

Based on that, I feel like it's more Arch package than setuptools itself. I will consider reporting this issue in Arch bug tracker and I think you can close this issue here at your convenience. Thanks for helping out and mery xmas!

[rlaphoenix]

I'm also on arch linux and have been getting this over the past couple of days. Surely it's no coincidence that mainly arch linux users are having the problem?

All that I needed to do was install pip from source rather than from the arch repo: sudo wget https://bootstrap.pypa.io/get-pip.py -O - | python

[jaraco]

To be sure, the issue isn't strictly due to arch's python-pip. Changing the repro to install setuptools<51.1 also bypasses the error, confirming the "Version 51.1" indication in the report.

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pip install --user --upgrade 'setuptools<51.1'
RUN pip install --user lastversion

I looked at the changes for 51.1 and there's nothing there that strikes me as particularly implicated. Here's the diff: https://github.com/pypa/setuptools/compare/v51.0.0..v51.1.0

[jaraco]

I wonder if maybe the issue is this empty install_requires directive. I've used that same syntax in other projects with no declared dependencies and never had an issue with it, but it seems plausible this declaration has revealed a defect in setuptools or pip.

[jaraco]

I think I've disconfirmed that theory with this Dockerfile, which still fails:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python python-pip
RUN pacman -S --noconfirm git
RUN git clone --branch workaround/2495-no-install-requires --depth 10 https://github.com/pypa/setuptools
WORKDIR setuptools
RUN python bootstrap.py
RUN pip install --user .
WORKDIR /
CMD pip install --user lastversion

[jaraco]

I also tried cleaning the indentation on the setup.cfg, but that had no effect on the failure either (6ab9f473c87058ffdc88f357d167d7f7b2750154).

[jaraco]

I tried using pdb to debug pip, but unfortunately, pip is unfriendly to debuggers:

[root@61533a76fa1c /]# python -m pdb -m pip install --user lastversion
> /usr/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> from __future__ import absolute_import
(Pdb) c
Collecting lastversion
  Downloading lastversion-1.2.4-py3-none-any.whl (32 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "/usr/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "/usr/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 438, in check_if_exists
    if not self.req.specifier.contains(existing_version, prereleases=True):
  File "/usr/lib/python3.9/site-packages/packaging/specifiers.py", line 790, in contains
    item = parse(item)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 296, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object
The program exited via sys.exit(). Exit status: 2
> /usr/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> from __future__ import absolute_import
(Pdb) q

[jaraco]

I tried installing the exact same versions of pip and setuptools to the system site-packages, and that succeeds without failure:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm python wget
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python - pip==20.2.3 setuptools==51.0.0
RUN pip install --user --upgrade setuptools
CMD pip install --user lastversion

So that indicates to me there are two factors at play here:

• Arch packaging of python-pip, and
• Setuptools 51.1 in user-site-packages.

Both are required to trigger the failure.

[jaraco]

I'm able to replicate the issue using a local mounted copy of setuptools for the upgrade install step

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm wget git
RUN pacman -S --noconfirm python python-pip

# pip install --user .
# pip install --user lastversion

I've tried reverting the whole setup.cfg file to the code from v51.0.0 and the problem persists, suggesting that the changes to that file aren't implicated in the issue.

[rlaphoenix]

I will say, what's odd is my system pip location doesn't have setuptools installed (checked via sudo pip freeze | grep setuptools), and can't install it either (due to the issue occurring). But pip install setuptools will, and I'm on the latest pypi version of it too. Quite odd.

[jaraco]

Hmm. I tried pip install --user . against a local checkout of setuptools 51.0.0, and the error remained.

I'm now beginning to think that Setuptools 51.1 isn't implicated... and the only reason I've previously been unable to replicate the issue with 51.0 was because 51.0 was the version installed by python-pip. Indeed, forcing a reinstall in the user directory replicates the issue without involving Setuptools 51.1:

FROM archlinux
RUN pacman -Sy --noconfirm
RUN pacman -S --noconfirm wget git
RUN pacman -S --noconfirm python python-pip

RUN pip install --user --force-reinstall setuptools==51
RUN pip install --user lastversion