Installation can fail in Python3 in Windows due to UnicodeDecodeError

[memsharded]

$ pip install wrapt

Will fail in Windows using a Visual Studio in some languages, because it will try to output some accented characters, and pip will fail with UnicodeDecodeError to decode them properly, and wrapt will not be properly installed.

Exception:
Traceback (most recent call last):
  File "c:\users\memsharded\envs\conan3\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 11: invalid continuation byte

The failure is in pip/compat/__init__.py file:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
               return s.decode('utf_8')

I have just workarounded it with:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
            try:
               return s.decode('utf_8')
            except Exception as e:
               return "Cannot decode this string"

It seems this will be improved in pip 10 (not yet released): pypa/pip#4110 (comment)

[IceflowRE]

Bump.

[GrahamDumpleton]

@IceflowRE What are you expecting me to do? As far as I can see this was advising me that due to an issue with pip there could be issues. If it was an issue with pip I can't do much about it.

[IceflowRE]

Maybe implement the workaround as long as pip10 is not released?

And if its an issue you cant do something (or dont want) close it with an comment, because without commenting it looks like you dont care.

[memsharded]

If a workaround is possible, that would be great. I didn't submit the issue asking for a fix, but more like as information for users, trying to contribute with a possible workaround. So, until pip 10 is released, this is an open issue for wrapt users, and it shouldn't be closed, IMHO.

[mkpoli]

Thank you for the information and workaround, it did help me.

[9468305]

I also meet this issue.
I guess it's caused by Windows console encode fomat - GBK default.
I rounded it by running pip install wrapt under Git Bash for Windows(MSYS, MinGW) console, which encode format is UTF-8 default.
Then everything is fine.

[tjenssen]

To solve this, change you code page to 866

chcp 866

[netzulo]

Please. need patch there! :)

have fresh python install 3.6.3 on my windows 10 and got the same error trying to use pylint plugin on IDE visual code , editor try to autoinstall on my new virtualenv pylint using python.exe for env

• im opening administrator prompt
• reinstalled python , just one version 3.6.3
• checked python env vars on windows
• reeboted
• checked python env vars on windows AGAIN
• opened new administrator prompt

and then...

C:\WINDOWS\system32>python --version
Python 3.6.3

C:\WINDOWS\system32>pip --version
pip 9.0.1 from d:\0.programms\python36\lib\site-packages (python 3.6)

C:\WINDOWS\system32>python
Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\WINDOWS\system32>pip install -U wrapt
Collecting wrapt
  Using cached wrapt-1.10.11.tar.gz
Installing collected packages: wrapt
  Running setup.py install for wrapt ... error
Exception:
Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "d:\0.programms\python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "d:\0.programms\python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

C:\WINDOWS\system32>

SOLVED

ty to : @GrahamDumpleton , @esabouraud

Just was a problem with symlinks from combination of windows10+gitSCM tools , just open Git Bash prompt and install it !

[GrahamDumpleton]

@netzulo From what I understand of this issue, it is ultimately a problem in pip. Try the workaround above. Unless someone can point me to what in the wrapt package might be triggering it, there isn't much I can do.

[esabouraud]

I did fresh installs of python 3.6.3 on Windows 7 and Windows 10, and had this issue occur only on Windows 10.

Changing the codepage with chcp did not solve the issue for me. However running pip from "Git Bash" as @9468305 suggested did the trick.

[netzulo]

@GrahamDumpleton thank you for your answer, im on it yet

SOLVED

Edited report 3 comments UP

[MarKett]

Thank you for this post and workaround! Changing the code page did the trick for me.

[achamoux]

Faced the same problems during months.
Now, installing wrapt via pip AND git-bash.exe worked like a charm. Visual Studio Code then installed pylint without any problem.
Tnx !!

[pradyunsg]

Hey there! A pip maintainer here.

This isn't exactly a pip issue but there's a patch that works around this in pip 10 which is in beta right now. If you wanna try it out, you can install pip 10 using get-pip.py or via pip install --pre -U pip.

[memsharded]

Hi @pradyunsg

Any news on this?

I have to disagree :), I think that yes, it is a pip issue. While it is installing a python package with native extension, if the compiler building the extension outputs some message with localized characters, pip is completely aborting installation. And that package won't install correctly, because pip is not resilient to encoding errors in an output that is irrelevant for the pip functioning.

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this? Thanks!

[GrahamDumpleton]

Oh, I was under impression that newer version of pip that addressed this had come out, so hadn't given it priority.

Been snowed under with some work, but will try and get onto it.

[memsharded]

Oh, my bad, I have just checked my recent py3.6 and it comes with pip 10 now by default, I hadn't realize it. This is good news! Sorry for the noise.

[pradyunsg]

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this?

It's been out of beta since mid-April. We're actually a few days away from our next release. :)

[GrahamDumpleton]

Should be addressed by 7bd955d

[GrahamDumpleton]

Closing this issue as assumed fix some time back in pip or by wrapt changes.