Installing packages fails if Python 3 installed into path with non-ASCII characters #9054

reflechant · 2020-10-26T13:07:33Z

Environment

pip version: 20.2.3 - 20.2.4
Python version: 3.9
OS: Windows 10 Home

Description
A fresh installation of Python 3.9 can't install any libraries with pip if username contains non-ASCII characters.
Probably it affects any paths with non-ASCII characters.

This is not new and happened before multiple times across recent years. One of the latest examples is #4984. I kindly suggest adding a unittest for that error.

This is especially bad because Python is number one language for education now. And such errors can confuse children a lot - they are not developers and are likely to have both prerequisites for this bug: 1) username in native language 2) installing Python with default settings

Expected behavior
pip is expected to work

How to Reproduce

Create a user with non-ASCII name
Install Python 3.9 with default settings
Try to install any library with pip, for example pip install pylint
An error occurs

Output

C:\Users\ФШЪЙAUC>C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Users\ФШЪЙAUC>C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe -m pip install pylint
Collecting pylint
  Downloading pylint-2.6.0-py3-none-any.whl (325 kB)
     |--------------------------------| 325 kB 1.6 MB/s
Collecting mccabe<0.7,>=0.6
  Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting isort<6,>=4.2.5
  Downloading isort-5.6.4-py3-none-any.whl (98 kB)
     |--------------------------------| 98 kB 6.4 MB/s
Collecting toml>=0.7.1
  Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting colorama; sys_platform == "win32"
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting astroid<=2.5,>=2.4.0
  Downloading astroid-2.4.2-py3-none-any.whl (213 kB)
     |--------------------------------| 213 kB 6.4 MB/s
Collecting six~=1.12
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting wrapt~=1.11
  Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting lazy-object-proxy==1.4.*
  Downloading lazy-object-proxy-1.4.3.tar.gz (34 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\cli\base_command.py", line 228, in _main
    status = self.run(options, args)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\commands\install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 340, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\operations\prepare.py", line 482, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\operations\prepare.py", line 91, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 65, in _setup_isolation
    self.req.build_env = BuildEnvironment()
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\build_env.py", line 82, in __init__
    fp.write(textwrap.dedent(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 152-154: character maps to <undefined>
WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available.
You should consider upgrading via the 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe -m pip install --upgrade pip' command.

C:\Users\ФШЪЙAUC>C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe -m pip install --upgrade pip
Collecting pip
  Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
     |--------------------------------| 1.5 MB 2.2 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
  WARNING: The scripts pip.exe, pip3.9.exe and pip3.exe are installed in 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.2.4

C:\Users\ФШЪЙAUC>C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe -m pip install pylint
Collecting pylint
  Using cached pylint-2.6.0-py3-none-any.whl (325 kB)
Collecting astroid<=2.5,>=2.4.0
  Using cached astroid-2.4.2-py3-none-any.whl (213 kB)
Collecting isort<6,>=4.2.5
  Using cached isort-5.6.4-py3-none-any.whl (98 kB)
Collecting colorama; sys_platform == "win32"
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting toml>=0.7.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting mccabe<0.7,>=0.6
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting lazy-object-proxy==1.4.*
  Using cached lazy-object-proxy-1.4.3.tar.gz (34 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\cli\base_command.py", line 228, in _main
    status = self.run(options, args)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\commands\install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 340, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\operations\prepare.py", line 482, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\operations\prepare.py", line 91, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\distributions\sdist.py", line 65, in _setup_isolation
    self.req.build_env = BuildEnvironment()
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_internal\build_env.py", line 82, in __init__
    fp.write(textwrap.dedent(
  File "C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 152-154: character maps to <undefined>

C:\Users\ФШЪЙAUC>

The text was updated successfully, but these errors were encountered:

uranusjr · 2020-10-26T13:46:20Z

Can you try the same command with --no-build-isolation? This would help a lot to pinpoint the cause.

reflechant · 2020-10-26T13:53:08Z

C:\Users\ФШЪЙAUC>C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe -m pip install --no-build-isolation pylint
Collecting pylint
  Using cached pylint-2.6.0-py3-none-any.whl (325 kB)
Collecting isort<6,>=4.2.5
  Using cached isort-5.6.4-py3-none-any.whl (98 kB)
Collecting toml>=0.7.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting colorama; sys_platform == "win32"
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting mccabe<0.7,>=0.6
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting astroid<=2.5,>=2.4.0
  Using cached astroid-2.4.2-py3-none-any.whl (213 kB)
Collecting six~=1.12
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting lazy-object-proxy==1.4.*
  Using cached lazy-object-proxy-1.4.3.tar.gz (34 kB)
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe' 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_vendor\pep517\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\977E~1\AppData\Local\Temp\tmp5w_47zqn'
         cwd: C:\Users\ФШЪЙAUC\AppData\Local\Temp\pip-install-0jqgl7aw\lazy-object-proxy
    Complete output (11 lines):
    WARNING: The wheel package is not available.
    running dist_info
    creating C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info
    writing C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\PKG-INFO
    writing dependency_links to C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\dependency_links.txt
    writing top-level names to C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\top_level.txt
    writing manifest file 'C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\SOURCES.txt'
    reading manifest file 'C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\SOURCES.txt'
    writing manifest file 'C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.egg-info\SOURCES.txt'
    creating 'C:\Users\ФШЪЙ\xc3\u0168\xc7\AppData\Local\Temp\pip-modern-metadata-fiik746p\lazy_object_proxy.dist-info'
    error: invalid command 'bdist_wheel'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\python.exe' 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\lib\site-packages\pip\_vendor\pep517\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\977E~1\AppData\Local\Temp\tmp5w_47zqn' Check the logs for full command output.

C:\Users\ФШЪЙAUC>

pfmoore · 2020-10-26T14:31:47Z

When writing site_customize.py in build_env (line 82) we need to open the file in UTF-8 mode.

The --no-build-isolation test failed because the user didn't have wheel installed in system sit epackages, and is an unrelated problem.

uranusjr · 2020-10-26T14:45:42Z

Could you try pip install wheel first (or paste the traceback if that also fails) and try the --no-build-isolation command again? I am trying to make sure there are no other encoding errors down the pipeline, once we fix the problem in build_env.

reflechant · 2020-10-26T15:02:58Z

after installing wheel with pip it works.

C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39>python.exe -m pip install --no-build-isolation pylint
Collecting pylint
  Using cached pylint-2.6.0-py3-none-any.whl (325 kB)
Collecting isort<6,>=4.2.5
  Using cached isort-5.6.4-py3-none-any.whl (98 kB)
Collecting mccabe<0.7,>=0.6
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting colorama; sys_platform == "win32"
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting toml>=0.7.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting astroid<=2.5,>=2.4.0
  Using cached astroid-2.4.2-py3-none-any.whl (213 kB)
Collecting six~=1.12
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting lazy-object-proxy==1.4.*
  Using cached lazy-object-proxy-1.4.3.tar.gz (34 kB)
    Preparing wheel metadata ... done
Collecting wrapt~=1.11
  Using cached wrapt-1.12.1.tar.gz (27 kB)
Building wheels for collected packages: lazy-object-proxy, wrapt
  Building wheel for lazy-object-proxy (PEP 517) ... done
  Created wheel for lazy-object-proxy: filename=lazy_object_proxy-1.4.3-cp39-cp39-win_amd64.whl size=20254 sha256=a82cf6b3435f97f55e3d02965a4b3cc607d2492ced3b3e41e4910855f71d913e
  Stored in directory: c:\users\фшъйauc\appdata\local\pip\cache\wheels\41\20\07\2a3e02cdfc8b442404202f6ef99ff1b1c16b73910968a46f2f
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.12.1-cp39-cp39-win_amd64.whl size=33689 sha256=837611dd277d87a9e75a5873114b14adfa4980057506d2a8b4c9642d9d5c02e8
  Stored in directory: c:\users\фшъйauc\appdata\local\pip\cache\wheels\98\23\68\efe259aaca055e93b08e74fbe512819c69a2155c11ba3c0f10
Successfully built lazy-object-proxy wrapt
Installing collected packages: isort, mccabe, colorama, toml, six, lazy-object-proxy, wrapt, astroid, pylint
  WARNING: The script isort.exe is installed in 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts epylint.exe, pylint.exe, pyreverse.exe and symilar.exe are installed in 'C:\Users\ФШЪЙAUC\AppData\Local\Programs\Python\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed astroid-2.4.2 colorama-0.4.4 isort-5.6.4 lazy-object-proxy-1.4.3 mccabe-0.6.1 pylint-2.6.0 six-1.15.0 toml-0.10.1 wrapt-1.12.1

P.S. It's on pip 20.2.4

uranusjr · 2020-10-26T15:07:57Z

Oh, awesome! So build_env is indeed the only part that’s failing. (Makes sense I guess, this is the most un-trotted path for newcomers, since most beginner-friendly packages have wheels, which wouldn’t hit this code path.)

pradyunsg · 2020-10-26T16:20:46Z

Yea, this is probably because of Python 3.9 being used, which won't have wheels available for most packages.

uranusjr · 2020-10-28T07:45:27Z

I was working on a PR for this and got curious. The path we assumed caused the problem is actually encodable in the user’s code page, and I can write it without a problem:

>>> p = r'C:\Users\ФШЪЙAUC'
>>> p.encode('cp1251')
b'C:\\Users\\\xd4\xd8\xda\xc9AUC'
>>> with open('test', 'w', encoding='cp1251') as f:
...  f.write(r'c:\users\фшъйauc')
... 
16

which makes me want to second guess whether we’re actually identifying the bug correctly.

reflechant · 2020-11-17T18:51:51Z

Looks like it affects Python 3.8.x (both classical installer and Microsoft Store one) too: black library fails to install. Doing pip install wheel and then installing black with --no-build-isolation helps.

Masynchin · 2021-07-16T11:28:50Z

Can this error closed by #9136 ? PR was closed because of py2 support, as mentioned in conversation, but now python 2 doesn't need support

pfmoore · 2021-07-16T12:40:30Z

As noted on #9136 "feel free to submit a new PR". I don't honestly know if fixing that issue would fix this one, but the easiest way to find out would probably be to try (after all, any fix should include a test that demonstrates the problem and that the new code fixes it, so even just starting with a test case in test-driven style would be useful...)

Masynchin · 2021-07-16T12:56:12Z

I set PYTHONUTF8=1 env variable as mentioned in #9573 and my problem (black hook in pre-commit raises UnicodeEncodeError) was solved. As I know, this is equivalent to open(..., encoding="utf-8"). I think this (set encoding in file open) can be solution but I don't know how to test this

methane · 2022-05-05T04:11:59Z

This issue should be fixed by #10593

uranusjr · 2022-05-05T12:47:24Z

Good spot!

pradyunsg added C: build logic Stuff related to metadata generation / wheel generation C: encoding Related to text encoding and likely, UnicodeErrors OS: windows Windows specific kind: crash For situations where pip crashes labels Oct 26, 2020

uranusjr mentioned this issue Nov 15, 2020

Use UTF-8 to write sitecustomize.py #9136

Closed

Harmon758 mentioned this issue Nov 24, 2020

installation error Rapptz/discord.py#6077

Closed

uranusjr mentioned this issue Feb 9, 2021

Pip opening file with cp1252 encoding raises exception when temp folder path contains non-latin char #9573

Closed

1 task

pekkaklarck mentioned this issue Apr 30, 2022

Installing packages fails if Python 3 installed into path with non-ASCII characters #4984

Closed

uranusjr closed this as completed May 5, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jun 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing packages fails if Python 3 installed into path with non-ASCII characters #9054

Installing packages fails if Python 3 installed into path with non-ASCII characters #9054

reflechant commented Oct 26, 2020

uranusjr commented Oct 26, 2020

reflechant commented Oct 26, 2020 •

edited

Loading

pfmoore commented Oct 26, 2020

uranusjr commented Oct 26, 2020 •

edited

Loading

reflechant commented Oct 26, 2020

uranusjr commented Oct 26, 2020 •

edited by pradyunsg

Loading

pradyunsg commented Oct 26, 2020

uranusjr commented Oct 28, 2020 •

edited

Loading

reflechant commented Nov 17, 2020

Masynchin commented Jul 16, 2021

pfmoore commented Jul 16, 2021

Masynchin commented Jul 16, 2021 •

edited

Loading

methane commented May 5, 2022

uranusjr commented May 5, 2022

Installing packages fails if Python 3 installed into path with non-ASCII characters #9054

Installing packages fails if Python 3 installed into path with non-ASCII characters #9054

Comments

reflechant commented Oct 26, 2020

uranusjr commented Oct 26, 2020

reflechant commented Oct 26, 2020 • edited Loading

pfmoore commented Oct 26, 2020

uranusjr commented Oct 26, 2020 • edited Loading

reflechant commented Oct 26, 2020

uranusjr commented Oct 26, 2020 • edited by pradyunsg Loading

pradyunsg commented Oct 26, 2020

uranusjr commented Oct 28, 2020 • edited Loading

reflechant commented Nov 17, 2020

Masynchin commented Jul 16, 2021

pfmoore commented Jul 16, 2021

Masynchin commented Jul 16, 2021 • edited Loading

methane commented May 5, 2022

uranusjr commented May 5, 2022

reflechant commented Oct 26, 2020 •

edited

Loading

uranusjr commented Oct 26, 2020 •

edited

Loading

uranusjr commented Oct 26, 2020 •

edited by pradyunsg

Loading

uranusjr commented Oct 28, 2020 •

edited

Loading

Masynchin commented Jul 16, 2021 •

edited

Loading