Pip 20.2.1 causes UnicodeDecodeError #8717

ha-nso-li · 2020-08-06T01:51:26Z

Environment

pip version: 20.2.1
Python version: Python 3.7.3
OS: Windows 10 Pro 1909

I using venv environment.

My venv path has multi byte character, hangul, i think it may cause error.

Description

Pip 20.2.1 causes UnicodeDecodeError when install some packages. In my case, rope.

But I could downgrade pip to 20.1 using pip. 20.1 has no issue.

There are similar issue. (#8658) But my enviroment is different OS (Win10 vs CentOS) and Python. (3.7 vs 2.7) And traceback is different, too. I think both issues are not same.

Expected behavior

How to Reproduce

Install PIP 20.2.1
Try to install rope

Output

PS C:\Users\정한솔\source\repos\retinanet> & c:/Users/정한솔/source/repos/retinanet/venv/Scripts/Activate.ps1
(venv) PS C:\Users\정한솔\source\repos\retinanet> pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed 
    req_to_install.check_if_exists(self.use_user_site)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
    return [d for d in working_set
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
    return _no_global_under_venv()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install pip==20.1 --ignore-installed
Collecting pip==20.1
  Using cached pip-20.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Successfully installed pip-20.2.1
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
Collecting rope
  Using cached rope-0.17.0.tar.gz (248 kB)
Building wheels for collected packages: rope
  Building wheel for rope (setup.py) ... done
  Created wheel for rope: filename=rope-0.17.0-py3-none-any.whl size=180521 sha256=bfb5eb13225f3cd93ec36b87982e799b8ee1b50c0d7a00cfde2864fff2160805
  Stored in directory: c:\users\정한솔\appdata\local\pip\cache\wheels\fc\68\52\627ca0d67f266c203ff5ef7e441036cf2049cdbb3e030c9e0a
Successfully built rope
Installing collected packages: rope
Successfully installed rope-0.17.0
(venv) PS C:\Users\정한솔\source\repos\retinanet> python --version
Python 3.7.3

The text was updated successfully, but these errors were encountered:

uranusjr · 2020-08-06T02:09:22Z

The ticket you linked is a different issue. ~~Is this specific to 20.2.1 (and presumably 20.2.0)? Or does the same happen on 20.1 and earlier?~~ Edit: I missed the line you said 20.1 does not have this issue, sorry.

uranusjr · 2020-08-06T03:26:25Z

OK, I got this one sorted out. The root cause to the error is that pyvenv.cfg is opened with a different encoding from what it is written by venv.

pyvenv.cfg exists before 20.2, however, and is not the reason this pops up now. The trigger is #8695; this code path is not run prior to 20.2, but the redundant dist_is_local check causes pyvenv.cfg to be read, triggering this error when the file has incompatible encoding.

So this exact error should no longer be a problem once #8702 is merged. But the root exception cause remains, and could still be hit in the future in other situations. @hansoli68 would you be able to check what encoding your pyvenv.cfg is written with? PEP 405 does not seem to specify what encoding pyvenv.cfg should be written in, and/or upload the file somewhere?

We’ll need to have a conversation with CPython maintainers (and perhaps also virtualenv maintainers) to clarify this.

uranusjr · 2020-08-06T03:41:52Z

It seems like venv has always used UTF-8 without explicitly specifying it anywhere. I’ll file a PR to fix the encoding.

https://github.com/python/cpython/blame/b5789a7419655f66692fab463320048bd2290e81/Lib/venv/__init__.py#L147-L157

ha-nso-li · 2020-08-06T04:35:55Z

@uranusjr My pyvenv.cfg is encoded with UTF-8, too. Thanks for your fix.

162: Update pip to 20.2.2 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.1** to **20.2.2**. <details> <summary>Changelog</summary> ### 20.2.2 ``` =================== Bug Fixes --------- - Only attempt to use the keyring once and if it fails, don't try again. This prevents spamming users with several keyring unlock prompts when they cannot unlock or don't want to do so. (`8090 <https://github.com/pypa/pip/issues/8090>`_) - Fix regression that distributions in system site-packages are not correctly found when a virtual environment is configured with ``system-site-packages`` on. (`8695 <https://github.com/pypa/pip/issues/8695>`_) - Disable caching for range requests, which causes corrupted wheels when pip tries to obtain metadata using the feature ``fast-deps``. (`8701 <https://github.com/pypa/pip/issues/8701>`_, `8716 <https://github.com/pypa/pip/issues/8716>`_) - Always use UTF-8 to read ``pyvenv.cfg`` to match the built-in ``venv``. (`8717 <https://github.com/pypa/pip/issues/8717>`_) - 2020 Resolver: Correctly handle marker evaluation in constraints and exclude them if their markers do not match the current environment. (`8724 <https://github.com/pypa/pip/issues/8724>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Aug 6, 2020

ha-nso-li changed the title ~~Pip 20.2.1 cause UnicodeDecodeError~~ Pip 20.2.1 causes UnicodeDecodeError Aug 6, 2020

pradyunsg added C: encoding Related to text encoding and likely, UnicodeErrors kind: crash For situations where pip crashes labels Aug 6, 2020

triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Aug 6, 2020

uranusjr mentioned this issue Aug 6, 2020

Always use UTF-8 to read pyvenv.cfg #8718

Merged

pradyunsg closed this as completed in #8718 Aug 7, 2020

fireattack mentioned this issue Sep 28, 2020

PIP doesn't read setup.cfg in UTF-8, which causes UnicodeDecodeError #8931

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip 20.2.1 causes UnicodeDecodeError #8717

Pip 20.2.1 causes UnicodeDecodeError #8717

ha-nso-li commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading

ha-nso-li commented Aug 6, 2020

Pip 20.2.1 causes UnicodeDecodeError #8717

Pip 20.2.1 causes UnicodeDecodeError #8717

Comments

ha-nso-li commented Aug 6, 2020 • edited Loading

uranusjr commented Aug 6, 2020 • edited Loading

uranusjr commented Aug 6, 2020 • edited Loading

uranusjr commented Aug 6, 2020 • edited Loading

ha-nso-li commented Aug 6, 2020

ha-nso-li commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading

uranusjr commented Aug 6, 2020 •

edited

Loading