Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation can fail in Python3 in Windows due to UnicodeDecodeError #112

Closed
memsharded opened this issue Oct 18, 2017 · 20 comments
Closed

Installation can fail in Python3 in Windows due to UnicodeDecodeError #112

memsharded opened this issue Oct 18, 2017 · 20 comments

Comments

@memsharded
Copy link

@memsharded memsharded commented Oct 18, 2017

$ pip install wrapt

Will fail in Windows using a Visual Studio in some languages, because it will try to output some accented characters, and pip will fail with UnicodeDecodeError to decode them properly, and wrapt will not be properly installed.

Exception:
Traceback (most recent call last):
  File "c:\users\memsharded\envs\conan3\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 11: invalid continuation byte

The failure is in pip/compat/__init__.py file:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
               return s.decode('utf_8')

I have just workarounded it with:

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            return s.decode(sys.__stdout__.encoding)
        except UnicodeDecodeError:
            try:
               return s.decode('utf_8')
            except Exception as e:
               return "Cannot decode this string"

It seems this will be improved in pip 10 (not yet released): pypa/pip#4110 (comment)

@IceflowRE
Copy link

@IceflowRE IceflowRE commented Nov 9, 2017

Bump.

@GrahamDumpleton
Copy link
Owner

@GrahamDumpleton GrahamDumpleton commented Nov 9, 2017

@IceflowRE What are you expecting me to do? As far as I can see this was advising me that due to an issue with pip there could be issues. If it was an issue with pip I can't do much about it.

@IceflowRE
Copy link

@IceflowRE IceflowRE commented Nov 9, 2017

Maybe implement the workaround as long as pip10 is not released?

And if its an issue you cant do something (or dont want) close it with an comment, because without commenting it looks like you dont care.

@memsharded
Copy link
Author

@memsharded memsharded commented Nov 9, 2017

If a workaround is possible, that would be great. I didn't submit the issue asking for a fix, but more like as information for users, trying to contribute with a possible workaround. So, until pip 10 is released, this is an open issue for wrapt users, and it shouldn't be closed, IMHO.

@mkpoli
Copy link

@mkpoli mkpoli commented Nov 12, 2017

Thank you for the information and workaround, it did help me.

@9468305
Copy link

@9468305 9468305 commented Nov 28, 2017

I also meet this issue.
I guess it's caused by Windows console encode fomat - GBK default.
I rounded it by running pip install wrapt under Git Bash for Windows(MSYS, MinGW) console, which encode format is UTF-8 default.
Then everything is fine.

@tjenssen
Copy link

@tjenssen tjenssen commented Dec 4, 2017

To solve this, change you code page to 866

chcp 866

@netzulo
Copy link

@netzulo netzulo commented Dec 11, 2017

Please. need patch there! :)

have fresh python install 3.6.3 on my windows 10 and got the same error trying to use pylint plugin on IDE visual code , editor try to autoinstall on my new virtualenv pylint using python.exe for env

  • im opening administrator prompt
  • reinstalled python , just one version 3.6.3
  • checked python env vars on windows
  • reeboted
  • checked python env vars on windows AGAIN
  • opened new administrator prompt

and then...

C:\WINDOWS\system32>python --version
Python 3.6.3

C:\WINDOWS\system32>pip --version
pip 9.0.1 from d:\0.programms\python36\lib\site-packages (python 3.6)

C:\WINDOWS\system32>python
Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\WINDOWS\system32>pip install -U wrapt
Collecting wrapt
  Using cached wrapt-1.10.11.tar.gz
Installing collected packages: wrapt
  Running setup.py install for wrapt ... error
Exception:
Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\0.programms\python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "d:\0.programms\python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "d:\0.programms\python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "d:\0.programms\python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "d:\0.programms\python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 61: invalid start byte

C:\WINDOWS\system32>

SOLVED

ty to : @GrahamDumpleton , @esabouraud

Just was a problem with symlinks from combination of windows10+gitSCM tools , just open Git Bash prompt and install it !

@GrahamDumpleton
Copy link
Owner

@GrahamDumpleton GrahamDumpleton commented Dec 11, 2017

@netzulo From what I understand of this issue, it is ultimately a problem in pip. Try the workaround above. Unless someone can point me to what in the wrapt package might be triggering it, there isn't much I can do.

@esabouraud
Copy link

@esabouraud esabouraud commented Dec 14, 2017

I did fresh installs of python 3.6.3 on Windows 7 and Windows 10, and had this issue occur only on Windows 10.

Changing the codepage with chcp did not solve the issue for me. However running pip from "Git Bash" as @9468305 suggested did the trick.

@netzulo
Copy link

@netzulo netzulo commented Dec 21, 2017

@GrahamDumpleton thank you for your answer, im on it yet

SOLVED

Edited report 3 comments UP

@MarKett
Copy link

@MarKett MarKett commented Jan 9, 2018

Thank you for this post and workaround! Changing the code page did the trick for me.

@achamoux
Copy link

@achamoux achamoux commented Feb 4, 2018

Faced the same problems during months.
Now, installing wrapt via pip AND git-bash.exe worked like a charm. Visual Studio Code then installed pylint without any problem.
Tnx !!

@pradyunsg
Copy link

@pradyunsg pradyunsg commented Apr 4, 2018

Hey there! A pip maintainer here.

This isn't exactly a pip issue but there's a patch that works around this in pip 10 which is in beta right now. If you wanna try it out, you can install pip 10 using get-pip.py or via pip install --pre -U pip.

@memsharded
Copy link
Author

@memsharded memsharded commented Jul 11, 2018

Hi @pradyunsg

Any news on this?

I have to disagree :), I think that yes, it is a pip issue. While it is installing a python package with native extension, if the compiler building the extension outputs some message with localized characters, pip is completely aborting installation. And that package won't install correctly, because pip is not resilient to encoding errors in an output that is irrelevant for the pip functioning.

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this? Thanks!

@GrahamDumpleton
Copy link
Owner

@GrahamDumpleton GrahamDumpleton commented Jul 11, 2018

Oh, I was under impression that newer version of pip that addressed this had come out, so hadn't given it priority.

Been snowed under with some work, but will try and get onto it.

@memsharded
Copy link
Author

@memsharded memsharded commented Jul 11, 2018

Oh, my bad, I have just checked my recent py3.6 and it comes with pip 10 now by default, I hadn't realize it. This is good news! Sorry for the noise.

@pradyunsg
Copy link

@pradyunsg pradyunsg commented Jul 12, 2018

It would be great if pip 10 was out of beta and become mainstream, any roadmap for this?

It's been out of beta since mid-April. We're actually a few days away from our next release. :)

@GrahamDumpleton
Copy link
Owner

@GrahamDumpleton GrahamDumpleton commented Aug 22, 2018

Should be addressed by 7bd955d

@GrahamDumpleton
Copy link
Owner

@GrahamDumpleton GrahamDumpleton commented Aug 5, 2021

Closing this issue as assumed fix some time back in pip or by wrapt changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
12 participants