Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pip 20.2.1 causes UnicodeDecodeError #8717

Closed
ha-nso-li opened this issue Aug 6, 2020 · 4 comments · Fixed by #8718
Closed

Pip 20.2.1 causes UnicodeDecodeError #8717

ha-nso-li opened this issue Aug 6, 2020 · 4 comments · Fixed by #8718
Labels
C: encoding Related to text encoding and likely, UnicodeErrors kind: crash For situations where pip crashes

Comments

@ha-nso-li
Copy link

ha-nso-li commented Aug 6, 2020

Environment

  • pip version: 20.2.1
  • Python version: Python 3.7.3
  • OS: Windows 10 Pro 1909

I using venv environment.

My venv path has multi byte character, hangul, i think it may cause error.

Description

Pip 20.2.1 causes UnicodeDecodeError when install some packages. In my case, rope.

But I could downgrade pip to 20.1 using pip. 20.1 has no issue.

There are similar issue. (#8658) But my enviroment is different OS (Win10 vs CentOS) and Python. (3.7 vs 2.7) And traceback is different, too. I think both issues are not same.

Expected behavior

How to Reproduce

  1. Install PIP 20.2.1
  2. Try to install rope

Output

PS C:\Users\정한솔\source\repos\retinanet> & c:/Users/정한솔/source/repos/retinanet/venv/Scripts/Activate.ps1
(venv) PS C:\Users\정한솔\source\repos\retinanet> pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed 
    req_to_install.check_if_exists(self.use_user_site)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
    return [d for d in working_set
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
    return _no_global_under_venv()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install pip==20.1 --ignore-installed
Collecting pip==20.1
  Using cached pip-20.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Successfully installed pip-20.2.1
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
Collecting rope
  Using cached rope-0.17.0.tar.gz (248 kB)
Building wheels for collected packages: rope
  Building wheel for rope (setup.py) ... done
  Created wheel for rope: filename=rope-0.17.0-py3-none-any.whl size=180521 sha256=bfb5eb13225f3cd93ec36b87982e799b8ee1b50c0d7a00cfde2864fff2160805
  Stored in directory: c:\users\정한솔\appdata\local\pip\cache\wheels\fc\68\52\627ca0d67f266c203ff5ef7e441036cf2049cdbb3e030c9e0a
Successfully built rope
Installing collected packages: rope
Successfully installed rope-0.17.0
(venv) PS C:\Users\정한솔\source\repos\retinanet> python --version
Python 3.7.3
@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Aug 6, 2020
@ha-nso-li ha-nso-li changed the title Pip 20.2.1 cause UnicodeDecodeError Pip 20.2.1 causes UnicodeDecodeError Aug 6, 2020
@uranusjr
Copy link
Member

uranusjr commented Aug 6, 2020

The ticket you linked is a different issue. Is this specific to 20.2.1 (and presumably 20.2.0)? Or does the same happen on 20.1 and earlier? Edit: I missed the line you said 20.1 does not have this issue, sorry.

@pradyunsg pradyunsg added C: encoding Related to text encoding and likely, UnicodeErrors kind: crash For situations where pip crashes labels Aug 6, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Aug 6, 2020
@uranusjr
Copy link
Member

uranusjr commented Aug 6, 2020

OK, I got this one sorted out. The root cause to the error is that pyvenv.cfg is opened with a different encoding from what it is written by venv.

pyvenv.cfg exists before 20.2, however, and is not the reason this pops up now. The trigger is #8695; this code path is not run prior to 20.2, but the redundant dist_is_local check causes pyvenv.cfg to be read, triggering this error when the file has incompatible encoding.

So this exact error should no longer be a problem once #8702 is merged. But the root exception cause remains, and could still be hit in the future in other situations. @hansoli68 would you be able to check what encoding your pyvenv.cfg is written with? PEP 405 does not seem to specify what encoding pyvenv.cfg should be written in, and/or upload the file somewhere?

We’ll need to have a conversation with CPython maintainers (and perhaps also virtualenv maintainers) to clarify this.

@uranusjr
Copy link
Member

uranusjr commented Aug 6, 2020

It seems like venv has always used UTF-8 without explicitly specifying it anywhere. I’ll file a PR to fix the encoding.

https://github.com/python/cpython/blame/b5789a7419655f66692fab463320048bd2290e81/Lib/venv/__init__.py#L147-L157

@ha-nso-li
Copy link
Author

ha-nso-li commented Aug 6, 2020

@uranusjr My pyvenv.cfg is encoded with UTF-8, too. Thanks for your fix.

bors bot added a commit to duckinator/emanate that referenced this issue Aug 11, 2020
162: Update pip to 20.2.2 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.1** to **20.2.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.2.2
   ```
   ===================

Bug Fixes
---------

- Only attempt to use the keyring once and if it fails, don&#39;t try again.
  This prevents spamming users with several keyring unlock prompts when they
  cannot unlock or don&#39;t want to do so. (`8090 &lt;https://github.com/pypa/pip/issues/8090&gt;`_)
- Fix regression that distributions in system site-packages are not correctly
  found when a virtual environment is configured with ``system-site-packages``
  on. (`8695 &lt;https://github.com/pypa/pip/issues/8695&gt;`_)
- Disable caching for range requests, which causes corrupted wheels
  when pip tries to obtain metadata using the feature ``fast-deps``. (`8701 &lt;https://github.com/pypa/pip/issues/8701&gt;`_, `8716 &lt;https://github.com/pypa/pip/issues/8716&gt;`_)
- Always use UTF-8 to read ``pyvenv.cfg`` to match the built-in ``venv``. (`8717 &lt;https://github.com/pypa/pip/issues/8717&gt;`_)
- 2020 Resolver: Correctly handle marker evaluation in constraints and exclude
  them if their markers do not match the current environment. (`8724 &lt;https://github.com/pypa/pip/issues/8724&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: encoding Related to text encoding and likely, UnicodeErrors kind: crash For situations where pip crashes
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants