UnicodeDecodeError when install simplejson under python 3.6.0

[Ducatel]

Hi,
There is a UnicodeDecodeError exception when I try to install simplejson.

Error stack

(project_virtual_env) J:\....\project>pip install simplejson
Collecting simplejson
  Using cached simplejson-3.11.1.tar.gz
Installing collected packages: simplejson
  Running setup.py install for simplejson ... error
Exception:
Traceback (most recent call last):
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 66: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 66: invalid continuation byte

Step to reproduce

SET VIRTUAL_ENV_PATH=%TEMP%\project_virtual_env
J:\....\project>python.exe -m venv "%VIRTUAL_ENV_PATH%"
J:\....\project>"%VIRTUAL_ENV_PATH%\Scripts\activate.bat"  "%VIRTUAL_ENV_PATH%\pyvenv.cfg"
J:\....\project>pip install simplejson

Extra info

• Python x64 3.6.0
• Pip 9.0.1
• Windows 10 x64
• Issue already publish on UnicodeDecodeError when install simplejson under python 3.6.0 simplejson/simplejson#182

[pfmoore]

Is your username actually "me", or is it something with a non-ASCII character in it? Also, can you confirm what your code page is set to (chcp from a command prompt).

[Ducatel]

me is the username in my VM.

C:\Users\me>chcp
Page de codes active : 850

[pfmoore]

Ah, OK. In a VM, "me" as a username is reasonable :-)

This looks like one of the issues that is fixed by #4486, which is in the development version of pip but not yet in a release. So it should be fixed in pip 10, but unfortunately we don't have a timescale for when that will be released at the moment.

[Ducatel]

Ah, OK. In a VM, "me" as a username is reasonable :-)

😄

There is a way to upgrade pip to dev version ?

[pfmoore]

You can install from github, via python -m pip install -U "git+https://github.com/pypa/pip"

[Ducatel]

Outch the installation produce some error but if a retry, it's working.

Collecting git+https://github.com/pypa/pip
  Cloning https://github.com/pypa/pip to c:\users\me\appdata\local\temp\pip-f7cc2o50-build
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Running setup.py install for pip ... done
Exception:
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Accès refusé: 'C:\\Users\\me\\AppData\\Local\\Temp\\pip-jdl6_f6j-uninstall\\users\\me\\appdata\\local\\temp\\project_virtual_env\\scripts\\pip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\basecommand.py", line 215, in main
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\commands\install.py", line 342, in run
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\req\req_set.py", line 795, in install
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\req\req_install.py", line 767, in commit_uninstall
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\req\req_uninstall.py", line 142, in commit
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\_vendor\six.py", line 686, in reraise
    raise value
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
  File "C:\Program Files\Python36\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Python36\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python36\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python36\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 3 more times]
  File "C:\Program Files\Python36\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip\utils\__init__.py", line 114, in rmtree_errorhandler
PermissionError: [WinError 5] Accès refusé: 'C:\\Users\\me\\AppData\\Local\\Temp\\pip-jdl6_f6j-uninstall\\users\\me\\appdata\\local\\temp\\project_virtual_env\\scripts\\pip.exe'

> pip install -U "git+https://github.com/pypa/pip"
Collecting git+https://github.com/pypa/pip
  Cloning https://github.com/pypa/pip to c:\users\dducatel\appdata\local\temp\pip-req-build-4miiiygy
Installing collected packages: pip
  Found existing installation: pip 10.0.0.dev0
    Uninstalling pip-10.0.0.dev0:
      Successfully uninstalled pip-10.0.0.dev0
  Running setup.py install for pip ... done
  
>pip --version
pip 10.0.0.dev0 from c:\users\me\appdata\local\temp\project_virtual_env\lib\site-packages\pip (python 3.6)

And after that when I did the installation of simplejson, it's working ;)
So I will wait for the next release of pip 👍

[pradyunsg]

Outch the installation produce some error

Seems to me like that might have been caused by something transient like an antivirus locking up the executable maybe.

[Ducatel]

Seems to me like that might have been caused by something transient like an antivirus locking up the executable maybe.

No, I do not have any antivirus. For me It's like a too long path for windows .....

[pfmoore]

Thanks for confirming the fix works!

[Ducatel]

No problem ;)