Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation can fail in Python3 in Windows due to UnicodeDecodeError #112

Open
memsharded opened this issue Oct 18, 2017 · 19 comments

Comments

Projects
None yet
@memsharded
Copy link

commented Oct 18, 2017

$ pip install wrapt

Will fail in Windows using a Visual Studio in some languages, because it will try to output some accented characters, and pip will fail with UnicodeDecodeError to decode them properly, and wrapt will not be properly installed.

Exception:
Traceback (most recent call last):
  File "c:\users\memsharded\envs\conan3\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 11: invalid continuation byte

The failure is in pip/compat/__init__.py file:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
               return s.decode('utf_8')

I have just workarounded it with:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
            try:
               return s.decode('utf_8')
            except Exception as e:
               return "Cannot decode this string"

It seems this will be improved in pip 10 (not yet released): pypa/pip#4110 (comment)

@IceflowRE

This comment has been minimized.

Copy link

commented Nov 9, 2017

Bump.

@GrahamDumpleton

This comment has been minimized.

Copy link
Owner

commented Nov 9, 2017

@IceflowRE What are you expecting me to do? As far as I can see this was advising me that due to an issue with pip there could be issues. If it was an issue with pip I can't do much about it.

@IceflowRE

This comment has been minimized.

Copy link

commented Nov 9, 2017

Maybe implement the workaround as long as pip10 is not released?

And if its an issue you cant do something (or dont want) close it with an comment, because without commenting it looks like you dont care.

@memsharded

This comment has been minimized.

Copy link
Author

commented Nov 9, 2017

If a workaround is possible, that would be great. I didn't submit the issue asking for a fix, but more like as information for users, trying to contribute with a possible workaround. So, until pip 10 is released, this is an open issue for wrapt users, and it shouldn't be closed, IMHO.

@mkpoli

This comment has been minimized.

Copy link

commented Nov 12, 2017

Thank you for the information and workaround, it did help me.

@9468305

This comment has been minimized.

Copy link

commented Nov 28, 2017

I also meet this issue.
I guess it's caused by Windows console encode fomat - GBK default.
I rounded it by running pip install wrapt under Git Bash for Windows(MSYS, MinGW) console, which encode format is UTF-8 default.
Then everything is fine.

@tjenssen

This comment has been minimized.

Copy link

commented Dec 4, 2017

To solve this, change you code page to 866

chcp 866

@netzulo

This comment has been minimized.

Copy link

commented Dec 11, 2017

Please. need patch there! :)

have fresh python install 3.6.3 on my windows 10 and got the same error trying to use pylint plugin on IDE visual code , editor try to autoinstall on my new virtualenv pylint using python.exe for env

  • im opening administrator prompt
  • reinstalled python , just one version 3.6.3
  • checked python env vars on windows
  • reeboted
  • checked python env vars on windows AGAIN
  • opened new administrator prompt

and then...

C:\WINDOWS\system32>python --version
Python 3.6.3

C:\WINDOWS\system32>pip --version
pip 9.0.1 from d:\0.programms\python36\lib\site-packages (python 3.6)

C:\WINDOWS\system32>python
Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\WINDOWS\system32>pip install -U wrapt
Collecting wrapt
  Using cached wrapt-1.10.11.tar.gz
Installing collected packages: wrapt
  Running setup.py install for wrapt ... error
Exception:
Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "d:\0.programms\python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "d:\0.programms\python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

C:\WINDOWS\system32>

SOLVED

ty to : @GrahamDumpleton , @esabouraud

Just was a problem with symlinks from combination of windows10+gitSCM tools , just open Git Bash prompt and install it !

@GrahamDumpleton

This comment has been minimized.

Copy link
Owner

commented Dec 11, 2017

@netzulo From what I understand of this issue, it is ultimately a problem in pip. Try the workaround above. Unless someone can point me to what in the wrapt package might be triggering it, there isn't much I can do.

@esabouraud

This comment has been minimized.

Copy link

commented Dec 14, 2017

I did fresh installs of python 3.6.3 on Windows 7 and Windows 10, and had this issue occur only on Windows 10.

Changing the codepage with chcp did not solve the issue for me. However running pip from "Git Bash" as @9468305 suggested did the trick.

@netzulo

This comment has been minimized.

Copy link

commented Dec 21, 2017

@GrahamDumpleton thank you for your answer, im on it yet

SOLVED

Edited report 3 comments UP

@MarKett

This comment has been minimized.

Copy link

commented Jan 9, 2018

Thank you for this post and workaround! Changing the code page did the trick for me.

@achamoux

This comment has been minimized.

Copy link

commented Feb 4, 2018

Faced the same problems during months.
Now, installing wrapt via pip AND git-bash.exe worked like a charm. Visual Studio Code then installed pylint without any problem.
Tnx !!

@pradyunsg

This comment has been minimized.

Copy link

commented Apr 4, 2018

Hey there! A pip maintainer here.

This isn't exactly a pip issue but there's a patch that works around this in pip 10 which is in beta right now. If you wanna try it out, you can install pip 10 using get-pip.py or via pip install --pre -U pip.

@memsharded

This comment has been minimized.

Copy link
Author

commented Jul 11, 2018

Hi @pradyunsg

Any news on this?

I have to disagree :), I think that yes, it is a pip issue. While it is installing a python package with native extension, if the compiler building the extension outputs some message with localized characters, pip is completely aborting installation. And that package won't install correctly, because pip is not resilient to encoding errors in an output that is irrelevant for the pip functioning.

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this? Thanks!

@GrahamDumpleton

This comment has been minimized.

Copy link
Owner

commented Jul 11, 2018

Oh, I was under impression that newer version of pip that addressed this had come out, so hadn't given it priority.

Been snowed under with some work, but will try and get onto it.

@memsharded

This comment has been minimized.

Copy link
Author

commented Jul 11, 2018

Oh, my bad, I have just checked my recent py3.6 and it comes with pip 10 now by default, I hadn't realize it. This is good news! Sorry for the noise.

@pradyunsg

This comment has been minimized.

Copy link

commented Jul 12, 2018

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this?

It's been out of beta since mid-April. We're actually a few days away from our next release. :)

@GrahamDumpleton

This comment has been minimized.

Copy link
Owner

commented Aug 22, 2018

Should be addressed by 7bd955d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.