Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install codepage error (sqlalchemy,windows) #4110

Closed
Goblenus opened this issue Nov 12, 2016 · 23 comments
Closed

pip install codepage error (sqlalchemy,windows) #4110

Goblenus opened this issue Nov 12, 2016 · 23 comments
Labels
auto-locked Outdated issues that have been locked by automation C: encoding Related to text encoding and likely, UnicodeErrors
Milestone

Comments

@Goblenus
Copy link

I have changed code page in my cmd to 65001 (it's utf-8), and while installing sqlalchemy by pip:

c:\Program Files\Python35\Scripts>pip3.exe install sqlalchemy

I obtained next error:

Collecting sqlalchemy
  Using cached SQLAlchemy-1.1.3.tar.gz
Installing collected packages: sqlalchemy
  Running setup.py install for sqlalchemy ... error
Exception:
Traceback (most recent call last):
  File "c:\program files\python35\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
  File "c:\program files\python35\lib\encodings\cp65001.py", line 17, in decode
    return codecs.code_page_decode(65001, input, errors, True)
UnicodeDecodeError: 'CP_UTF8' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python35\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\program files\python35\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\program files\python35\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "c:\program files\python35\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "c:\program files\python35\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "c:\program files\python35\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 34: invalid continuation byte

To solve this, change you code page to 866

chcp 866

And try to install sqlalchemy once again.

Issue on sqlalchemy: https://bitbucket.org/zzzeek/sqlalchemy/issues/3851/pip-install-error-windows-10

@sakurai-youhei
Copy link

+1 for fix; I've seen many (now pysandbox) recently because of code page 932 on my Windows...

C:\Users\sakurai>python --version
Python 3.6.0

C:\Users\sakurai>python -m pip --version
pip 9.0.1 from C:\Python36\lib\site-packages (python 3.6)

@xavfernandez xavfernandez added the C: encoding Related to text encoding and likely, UnicodeErrors label Mar 24, 2017
@xavfernandez
Copy link
Member

Related to #4251

@AraHaan
Copy link
Contributor

AraHaan commented May 14, 2017

I seen this issue also happen with packages that have cython extension modules (aiohttp, yarl) in packages on 3.6 when they did not have wheels for 3.6 on PyPi. Note this bug does not appear when pip installing bdist_wheel's even if there are no c/cython extension modules.

@XCanG
Copy link

XCanG commented May 20, 2017

I want to say some words about that error. In short: Some days ago I can't update aiohttp and today I can't install compleately new package.
About aiohttp problem I already write issue here and I solve it: aio-libs/aiohttp#1885
Today I try to talk in IRC on #python channel and I get help and we find source of that error.
Here I quote problem and source from chat log:

[22:41:26] <moreati> XCanG: please pastebin what you ran, the output and the errors
[22:59:06] <XCanG> moreati: here I write all about my bug with pip what I know: https://pastebin.com/khmaK0Pu
[22:59:09] <infobob> https://paste.pound-python.org/show/aoxHulQphZhCtlgWk739/ (repasted for XCanG)
[23:00:50] <ChrisWarrick> XCanG: There is some broken file that pip tries to read on your system
[23:02:42] <moreati> XCanG: run pip freeze to get a list of pip-installed packages
[23:03:15] <moreati> XCanG: then run pip uninstall -y <those packages>
[23:03:48] <ChrisWarrick> XCanG: I’d do my favorite way of debugging, print debugging
[23:04:05] <ChrisWarrick> XCanG: open C:\python\python36\lib\site-packages\pip\compat\__init__.py, and before line 75, add: print(s)
[23:04:14] <ChrisWarrick> XCanG: then see/tell us what it prints
[23:09:52] <XCanG> ChrisWarrick print printed that:   Running setup.py install for ujson ... /b'   \xd1\xee\xe7\xe4\xe0\xe5\xf2\xf1\xff \xe1\xe8\xe1\xeb\xe8\xee\xf2\xe5\xea\xe0 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.lib \xe8 \xee\xe1\xfa\xe5\xea\xf2 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.exp\r\n'
[23:10:27] <XCanG> and error on 0xd1
[23:11:18] <ChrisWarrick> XCanG: Which language do you speak/is your system set to?
[23:11:36] <XCanG> my system language is russian
[23:12:26] <ChrisWarrick> XCanG: Your problem is “Создаетсябиблиотека”. Report a bug in pip, pastebin traceback, make sure to mention your system is in Russian, and that the string is CP1251.
...
[23:18:22] <ChrisWarrick> XCanG: There seems to be a bug in pip with the handling of Russian (non-latin) path names

My system specs writed in that pastebin url and as Chris say, my system language is Russian and string generated in CP1251. Link to pastebin: https://pastebin.com/khmaK0Pu

If you need anything else, then ask me, I will reply. Hope that helps with solving that error.

@pfmoore
Copy link
Member

pfmoore commented May 20, 2017

Fundamentally, I believe the problem is that Visual C can produce output that is in a mix of the ANSI and the OEM codepages when run with stdout directed to a pipe. This means that the output is in practice un-decodable to Unicode.

At the moment, pip fails with a UnicodeError as a result of this. We're working on a patch that allows pip to produce output without failing, although the output will still contain mojibake as it is the data itself that is in error.

@XCanG
Copy link

XCanG commented May 21, 2017

I write fix for me.
In /site-packages/pip/compat/__init__.py
I add:

from locale import getpreferredencoding

...

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            ss = []
            l = getpreferredencoding()
            for i in range(len(s)):
                if 255 >= s[i] >= 128:
                    ss.append(s[i:i+1].decode(l))
                else:
                    ss.append(s[i:i+1].decode(sys.__stdout__.encoding))
            s = "".join(ss)
            del ss
            return s
        except UnicodeDecodeError:
            return s.decode('utf_8')

    def native_str(s, replace=False):
        if isinstance(s, bytes):
            return s.decode('utf-8', 'replace' if replace else 'strict')
        return s

So I locale.getpreferredencoding() return my cp1251
Then I check if characters between 128 (\x80) and 255 (\xff) I convert it from my local code page, else I convert by utf-8, then join converted string and return.

Also, what is actually in output (log by print("// LOGGING /", s, end="")):

pip install ujson
Collecting ujson
  Using cached ujson-1.35.tar.gz
// LOGGING / running egg_info
// LOGGING / creating pip-egg-info\ujson.egg-info
// LOGGING / writing pip-egg-info\ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to pip-egg-info\ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to pip-egg-info\ujson.egg-info\top_level.txt
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / Installing collected packages: ujson
  Running setup.py install for ujson ... // LOGGING / running install
-// LOGGING / running build
// LOGGING / running build_ext
// LOGGING / building 'ujson' extension
// LOGGING / creating build
\// LOGGING / creating build\temp.win-amd64-3.6
// LOGGING / creating build\temp.win-amd64-3.6\Release
// LOGGING / creating build\temp.win-amd64-3.6\Release\python
// LOGGING / creating build\temp.win-amd64-3.6\Release\lib
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/ujson.c /Fobuild\temp.win-amd64-3.6\Release\./python/ujson.obj -D_GNU_SOURCE
// LOGGING / ujson.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/objToJSON.c /Fobuild\temp.win-amd64-3.6\Release\./python/objToJSON.obj -D_GNU_SOURCE
|// LOGGING / objToJSON.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/JSONtoObj.c /Fobuild\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj -D_GNU_SOURCE
// LOGGING / JSONtoObj.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsonenc.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj -D_GNU_SOURCE
/// LOGGING / ultrajsonenc.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsondec.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj -D_GNU_SOURCE
// LOGGING / ultrajsondec.c
// LOGGING / creating G:\AppData\Local\Temp\pip-build-pgfn501d\ujson\build\lib.win-amd64-3.6
-// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python\python36\libs /LIBPATH:c:\python\python36\PCbuild\amd64 "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\um\x64" /EXPORT:PyInit_ujson build\temp.win-amd64-3.6\Release\./python/ujson.obj build\temp.win-amd64-3.6\Release\./python/objToJSON.obj build\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj /OUT:build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib
// LOGGING /    Создается библиотека build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib и объект build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.exp
// LOGGING / Создание кода
// LOGGING / Создание кода завершено
\// LOGGING / running install_lib
// LOGGING / copying build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd -> c:\python\python36\Lib\site-packages
// LOGGING / running install_egg_info
// LOGGING / running egg_info
// LOGGING / writing ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to ujson.egg-info\top_level.txt
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / Copying ujson.egg-info to c:\python\python36\Lib\site-packages\ujson-1.35-py3.6.egg-info
// LOGGING / running install_scripts
// LOGGING / writing list of installed files to 'G:\AppData\Local\Temp\pip-hxlakqjj-record\install-record.txt'
// LOGGING /done
Successfully installed ujson-1.35

C:\Windows\system32>

As you can see, here is 3 lines what contain my local language.

@pfmoore
Copy link
Member

pfmoore commented May 27, 2017

This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10. I am closing this issue, but if you are still able to reproduce the problem in the development version of pip, please reopen it with a description of now to replicate the issue.

@hasufell
Copy link

This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10.

So... pip on windows is basically broken since more than half a year without a convenient way for a temporary fix? pip-10 is not released and I don't see any pre-release either.

@sakurai-youhei
Copy link

I personally agree with hasufell’s point. Newbies (I guess they’d be students) have been wasting time due to this issue in Japan because of majority of Japanese Windows (cp932). I wish pip were not a blocker but a helper of newbies’ start-up in Python’s ecosystem.

@pradyunsg
Copy link
Member

pradyunsg commented Oct 19, 2017

@sakurai-youhei I think there'll be a 10.0 soon. Until then, you could use pip's development version by installing it as follows:

pip install -U git+https://github.com/pypa/pip

@AraHaan
Copy link
Contributor

AraHaan commented Oct 19, 2017

@pradyunsg you forgot an --upgrade there.

@pradyunsg
Copy link
Member

Thanks for catching that @AraHaan!

@shimizukawa
Copy link
Contributor

I've installed 10.0.0.dev0 with bellow command because Windows locks pip.exe and "users" doesn't have git client. It worked.

python -m pip install -U https://github.com/pypa/pip/archive/master.zip

@pradyunsg
Copy link
Member

Thanks @shimizukawa!

tbh, I'd not actually put in the amount of thought that I should have in my comment above. Your command is indeed better than what I'd suggested. I might just start suggesting the same soon. :)

@uranusjr
Copy link
Member

uranusjr commented Jan 4, 2018

So a new year has come and there is still no concrete plan for Pip 10. Would it be plausible to have a 9.0.2 to include this fix, please? This is clearly an issue that affects people, and one those affected are highly unlikely to know how to solve.

@paulorodriguesxv
Copy link

paulorodriguesxv commented Jan 4, 2018

I have to agree with @uranusjr. I've been watching this issue about one year, it's so long time and some times a think it's kinda windows-users-bullying ?

@sakurai-youhei
Copy link

+1 for 9.0.2

@pfmoore
Copy link
Member

pfmoore commented Jan 4, 2018

windows-users-bullying

Not sure what you mean by that. I'm a windows user and a pip dev, and I implemented the fix for this issue. I agree it's high priority to get it released (for some of our users - like most encoding issues, it's user-specific - if it hits you you get it all the time, but if it doesn't you're not clear what the fuss is about). If the pip release process was structured that way, I'd support the idea of a bugfix release, but unfortunately, it's not - we release what's currently on the master branch at any time, and we currently have some new features on master that need a bit of work to complete (PEP 517/518 support in particular, and also the hiding of the internal API, which while appropriate for pip 10, isn't really acceptable for a 9.0.x release).

So while I'd agree that this fix is one I'd be happy to include in a 9.0.2 release, we don't really have an easy way to cut a 9.0.2 release at this time.

One of our goals for a more sustainable pip development process should probably be structuring things so we have a better workflow to support bugfix releases. But we have extremely limited developer resources, and basically no commercial or vendor support, so "big picture" questions like this are hard to find time to address. If anyone wants to raise the issue with whoever provides commercial support for their use of Python/pip, asking them to contribute towards the sustainability of Python's package management infrastructure, that would be a good long-term step to avoiding this type of situation in the future.

@paulorodriguesxv
Copy link

Hi @pfmoore. I understand your viewpoint, and you're right about it. About the "windows-users-bullying" I was just kidding, sorry for that. :-)

@pfmoore
Copy link
Member

pfmoore commented Jan 4, 2018

No problem - I just wasn't sure what you meant and whether there was a genuine issue we could address underlying your comment.

@XCanG
Copy link

XCanG commented Jan 7, 2018

@uranusjr @paulorodriguesxv If you still have this issue, try my code (other fixes not help to me, so I code it by myself). The code showed, probably, above #4110 (comment).

@paulorodriguesxv
Copy link

thanks @XCanG

@lock
Copy link

lock bot commented Jun 2, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 2, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation C: encoding Related to text encoding and likely, UnicodeErrors
Projects
None yet
Development

Successfully merging a pull request may close this issue.