Non-ASCII character '\xe2' in file

[bhundven]

I got the following while running python2.7 -m pip install -U --user pipenv on CentOS7:

File "/home/users/bhundven/.local/lib/python2.7/site-packages/zipp.py", line 153
SyntaxError: Non-ASCII character '\xe2' in file /home/users/bhundven/.local/lib/python2.7/site-packages/zipp.py on line 154, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

I found that the ascii characters were added in dff39a6
I was able to pin zipp==1.1.1, and was able to move forward: python2.7 -m pip install -U --user "zipp==1.1.1" pipenv

[jaraco]

What version of zipp is being installed on Python 2? If it’s zipp 2 or later, you are getting an incompatible version of zipp probably because you have an old pip installed. Your best bet is to install pip 9 or later before installing pipenv. Or just pin to zipp<2 in your environment when installing pipenv (or anything else that might require zipp).

[bhundven]

1.2.0

[jaraco]

When I check out maint/1.x and run tox -e py27, the tests pass. Also, tests pass on CI, which makes it difficult for me to reproduce the problem (and validate a fix and consider writing a test). Do you know why it's passing in my environments but failing in yours? Do you know what settings I would need to set in CentOS 7 to replicate the issue?

[jaraco]

I tried replicating the issue no CentOS 7 without any luck:

zipp maint/1.x $ cat Dockerfile                                                                                                                                           
FROM centos:7
RUN yum install -y wget
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python
RUN pip install zipp
ENV LANG=C
CMD python -c "import zipp" && echo "no problem"
zipp maint/1.x $ docker run -it @($(docker build -q .).strip())                                                                                                           
no problem

[bhundven]

It seems to be an environment issue.

In our environment, we are using python2-pip from epel. Which I'm finding, might be the issue (it's pretty old).

I did not know about the pypa install, and after trying that, things are better.

It also seems to trigger mostly when using pipenv as non-root.
After switching to the pypa pip install, I can't reproduce it.

This is the way I can reproduce the issue.
Dockerfile

FROM centos:7
RUN useradd -m -U test
RUN yum install -y epel-release
RUN yum install -y python-devel python2-pip python-virtualenv which

USER test
WORKDIR /home/test
ENV LANG=C
ENV PATH "/home/test/.local/bin:$PATH"
RUN python -m pip install -U --user wheel pipenv
RUN pipenv --python=/usr/bin/python2.7 --site-packages install --skip-lock ansible
CMD pipenv run python -c "import zipp" && echo "no problem"

(this is not the way I actually got to this issue, but this works (breaks) the way I think)
So it looks like I need to retool some of our developer workstations to not use the epel pip. and that won't work... because I can't always assume that other devs I work with will do "the right thing".

[bhundven]

For example, this is working:

FROM centos:7
RUN useradd -m -U test
RUN yum install -y which

USER test
WORKDIR /home/test
ENV LANG=C
ENV PATH "/home/test/.local/bin:$PATH"
# No access to the system's site-packages, so it installs to the users site-packages
RUN curl https://bootstrap.pypa.io/get-pip.py | python
RUN python -m pip install -U --user wheel pipenv
RUN pipenv --python=/usr/bin/python2.7 --site-packages install --skip-lock ansible
CMD pipenv run python -c "import zipp" && echo "no problem"

[jaraco]

Thanks so much for the replication in a Dockerfile. From that, I was able to distill the issue down to this Dockerfile:

FROM centos:7
RUN useradd -m -U test
RUN yum install -y epel-release
RUN yum install -y python-devel python2-pip python-virtualenv which

USER test
WORKDIR /home/test
RUN python -m pip install --user -U wheel setuptools virtualenv
RUN python -m virtualenv --help

It took a while to eliminate pipenv from the script. Turns out, if you don't upgrade setuptools, which pipenv does, you get a completely invalid zipp. But with this installation, you get a mostly valid zipp but with the syntax error.

If you look at the build log though, you'll see a few warnings:

...
Step 7/9 : RUN python -m pip install --user -U wheel setuptools virtualenv
 ---> Running in ad08cdd53f27
...
Collecting configparser>=3.5; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/e5/7c/d4ccbcde76b4eea8cbd73b67b88c72578e8b4944d1270021596e80b13deb/configparser-5.0.0.tar.gz (75kB)
  Running setup.py (path:/tmp/pip-build-DYKwb2/configparser/setup.py) egg_info for package configparser produced metadata for project name unknown. Fix your #egg=configparser fragments.
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/ce/8c/2c5f7dc1b418f659d36c04dec9446612fc7b45c8095cc7369dd772513055/zipp-3.1.0.tar.gz
  Running setup.py (path:/tmp/pip-build-DYKwb2/zipp/setup.py) egg_info for package zipp produced metadata for project name unknown. Fix your #egg=zipp fragments.
...
You are using pip version 8.1.2, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

As you can see, the problem is that pip as bundled with EPEL is <9, and as a result attempts to download zipp 3.x, which is unsupported on Python 2.

The proper workaround is to upgrade pip before installing virtualenv or pipenv:

FROM centos:7
RUN useradd -m -U test
RUN yum install -y epel-release
RUN yum install -y python-devel python2-pip python-virtualenv which

USER test
WORKDIR /home/test
RUN python -m pip install --user -U pip
RUN python -m pip install --user -U wheel setuptools virtualenv
RUN python -m virtualenv --help

or pin to zipp<2 when installing virtualenv or pipenv:

FROM centos:7
RUN useradd -m -U test
RUN yum install -y epel-release
RUN yum install -y python-devel python2-pip python-virtualenv which

USER test
WORKDIR /home/test
RUN python -m pip install --user -U wheel setuptools virtualenv 'zipp<2'
RUN python -m virtualenv --help

Furthermore, zipp 1.x has the encoding header, which would be the proper fix for the problem if it were missing.

Unfortunately, there's not much zipp can do here short of restoring Python 2 support in its latest releases (along with its transitive dependencies), and Centos 7 users are going to continue to run into issues as Python 2 support is sunset in the ecosystem and packages drop support for it (and other older releases).

I hope this analysis provides some clarity and viable workarounds. Good luck, and feel free to follow up if you have more questions.

[bhundven]

@jaraco Thank you so much for explaining that! I'm fixing my automation scripts to upgrade pip, setuptools, and wheel!