UnicodeDecodeError on Python 3.1 in backwardcompat.py #326

Closed
kumar303 opened this Issue Jul 26, 2011 · 17 comments

Comments

Projects
None yet
6 participants
Contributor

kumar303 commented Jul 26, 2011

To reproduce using tox

  1. create a temp directory
  2. add this tox.ini :

[testenv]
deps= coverage ==3.5

  1. add a dummy setup.py file:

from distutils.core import setup
setup(name='foo')

  1. Run tox -e py31

I get this traceback:

[TOX] ERROR: /Users/kumar/tmp/pip-coverage/.tox/py31/log$ ../bin/pip install --download-cache=/Users/kumar/tmp/pip-coverage/.tox/_download coverage ==3.5 >1.log
Downloading/unpacking coverage==3.5
  Storing download in cache at /Users/kumar/tmp/pip-coverage/.tox/_download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fc%2Fcoverage%2Fcoverage-3.5.tar.gz
  Running setup.py egg_info for package coverage

    no previously-included directories found matching 'test'
Installing collected packages: coverage
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    /usr/bin/cc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python3/3.1.2/include/python3.1 -c coverage/tracer.c -o build/temp.macosx-10.4-x86_64-3.1/coverage/tracer.o
    In file included from /usr/local/Cellar/python3/3.1.2/include/python3.1/Python.h:108,
                     from coverage/tracer.c:3:
Exception:
Traceback (most recent call last):
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/commands/install.py", line 228, in run
    requirement_set.install(install_options, global_options)
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 1100, in install
    requirement.install(install_options, global_options)
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 572, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/__init__.py", line 230, in call_subprocess
    line = console_to_str(stdout.readline())
  File "/Users/kumar/tmp/pip-coverage/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/backwardcompat.py", line 60, in console_to_str
    return s.decode(console_encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 76: ordinal not in range(128)

tox -e py32 works!!

Contributor

pnasrat commented Jul 31, 2011

Replicated outside of tox with virtualenv and python3.1 dumped output to pip.log and looked up the characters - gcc is outputting smart quotes and so on.

x = '\xe2\x80\x98'
x.decode('utf-8')
u'\u2018'
print _

Contributor

pnasrat commented Jul 31, 2011

pnasrat@paul-nasrats-macbook:pip (develop)$ python3.1 -c 'import sys; print(sys.stdout.encoding)'
UTF-8
pnasrat@paul-nasrats-macbook:pip (develop)$ python3.2 -c 'import sys; print(sys.stdout.encoding)'
UTF-8

Contributor

pnasrat commented Jul 31, 2011

Looks like the native extension compilation on 3.2 succeeds

Downloading/unpacking coverage
  Downloading coverage-3.5.tar.gz (110Kb): 110Kb downloaded
  Running setup.py egg_info for package coverage

    no previously-included directories found matching 'test'
Installing collected packages: coverage
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c coverage/tracer.c -o build/temp.macosx-10.6-intel-3.2/coverage/tracer.o
    gcc-4.2 -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-3.2/coverage/tracer.o -o build/lib.macosx-10.6-intel-3.2/coverage/tracer.so

    no previously-included directories found matching 'test'
    Installing coverage script to /Users/pnasrat/Development/pip/32/bin
Successfully installed coverage
Cleaning up...
>>> import coverage
>>> import coverage.tracer
>>> coverage.tracer.__file__
'/Users/pnasrat/Development/pip/32/lib/python3.2/site-packages/coverage/tracer.so'
Python 3.1.4 (v3.1.4:c918ec9f3a76, Jun 11 2011, 14:37:17) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import coverage
>>> import coverage.tracer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named tracer
Contributor

pnasrat commented Jul 31, 2011

So it looks like the compile on 3.1 python from upstream is doing:

gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1 -c coverage/tracer.c -o build/temp.macosx-10.3-fat-3.1/coverage/tracer.o
 ls -1 /Developer/SDKs/
MacOSX10.5.sdk
MacOSX10.6.sdk
pnasrat@paul-nasrats-macbook:pip (develop)$ python3.2 -c 'import distutils.sysconfig as dc; print(dc.get_config_var("PY_CFLAGS"))'
-fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk
pnasrat@paul-nasrats-macbook:pip (develop)$ python3.1 -c 'import distutils.sysconfig as dc; print(dc.get_config_var("PY_CFLAGS"))'
-fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -O2 -DNDEBUG -g -O3 -I. -IInclude -I/Users/sysadmin/build/v3.1.4/Include  -DPy_BUILD_CORE 
pnasrat@paul-nasrats-macbook:pip (develop)$ python2.7 -c 'import distutils.sysconfig as dc; print(dc.get_config_var("PY_CFLAGS"))'
-fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I. -IInclude -I/Users/ronald/Projects/python/r271/Include  -DPy_BUILD_CORE
Contributor

pnasrat commented Jul 31, 2011

@kumar303 from your tox python 3.1 can you do

python3.1 -c 'import sys; print(sys.stdout.encoding)'

I don't get the failure with pip in a 3.1 virtualenv but I can see why you are getting it due to the gcc failure on Snow Leopard with coverage.tracer and non-existant SDK.

Contributor

pnasrat commented Jul 31, 2011

@kumar303 also can you try running your config with:

export LC_ALL=C
tox -e py31
Contributor

kumar303 commented Jul 31, 2011

Here's the first one:

$ /Users/kumar/tmp/pip-coverage/.tox/py31/bin/python3.1 -c 'import sys; print(sys.stdout.encoding)'
utf-8

And after setting LC_ALL=C tox --recreate -e py31 does in fact succeed.

Huh, weird, I don't know why console_encoding is set to ascii otherwise.

pnasrat was assigned Aug 1, 2011

Contributor

pnasrat commented Aug 1, 2011

Thanks for the information. It may be an issue with tox which I'll also look into, but we should ensure we can handle this.

Contributor

embray commented Aug 18, 2011

This is actually due to the same issue I reported in issue #325. It's actually a bug in the subprocess module.

Contributor

embray commented Aug 18, 2011

And actually, for what it's worth, I just add some -Wno- options to gcc to squelch the warnings that contain non-ASCII characters. Setting LC_ALL=C would do the trick too, as you discovered.

Contributor

msabramo commented Oct 25, 2011

This appears to work around the issue described in tox:

https://bitbucket.org/hpk42/tox/pull-request/5/fix-issue-10-unicodedecodeerror-from-pip

but this could probably be fixed in pip and that would prevent the problem in other tools that call pip and would allow piping and redirecting pip. This would allow stuff like the following to work:

$ bin/pip install --upgrade PyYAML | egrep '^(Installing|UnicodeDecodeError)'
Installing collected packages: PyYAML
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 60:
ordinal not in range(128)
Contributor

msabramo commented Oct 26, 2011

The changes in the following tree seem to eliminate this problem:

https://github.com/msabramo/pip/tree/fix-issue-326-UnicodeDecodeError

Pull request: #374

Contributor

msabramo commented Oct 29, 2011

I just added some commits that add a test for this issue and fix Python 2 compatibility in one of my previous commits.

Contributor

msabramo commented Feb 7, 2012

@carljm just merged Pull request: #374 into pypa:develop -- Folks may want to try with this change and see if it fixes the original issue here.

Contributor

qwcode commented Oct 5, 2013

see #1224, and specifically #1224 (comment)
#1224 may have reversed the fix in #374.

Contributor

msabramo commented Oct 8, 2013

It appears to me from a quick test, that #1224 did not reverse the fix in #374 (or maybe something else changed, but in any case, things seem to work properly as far as I can tell):

(note that pip is broken on Python 3.1 because of the missing ssl.match_hostname function. I am using the change in #1227 to work around this here)

(py31.venv)marca@marca-mac:~/dev/git-repos/pip$ which pip
/Users/marca/dev/git-repos/pip/py31.venv/bin/pip
(py31.venv)marca@marca-mac:~/dev/git-repos/pip$ git branch
  develop
  display_download_full_url
* monkeypatch_ssl_match_hostname
  pr_1170
  solve_ssl_match_hostname_error_on_py31
  use_ssl_match_hostname_from_vendor_if_not_found

(py31.venv)marca@marca-mac:~/dev/git-repos/pip$ git --no-pager log -2
commit d4ea4e63252f9d4f8385088debd6095194c8a433
Author: Marc Abramowitz <marc@marc-abramowitz.com>
Date:   Mon Oct 7 15:36:53 2013 -0700

    If we don't have ssl.match_hostname, monkeypatch it in, so that it's
    available to vendored libraries like distlib

commit 6cd3e999545cdc821401d58727eb9cfb1cad5145
Author: Marcus Smith <qwcode@gmail.com>
Date:   Sat Oct 5 22:51:02 2013 -0700

    update changelog

(py31.venv)marca@marca-mac:~/dev/git-repos/pip$ pip install http://marc-abramowitz.com/download/python/broken-0.2broken.tar.gz
Downloading/unpacking http://marc-abramowitz.com/download/python/broken-0.2broken.tar.gz
  Using download cache from /Users/marca/.pip/download-cache/http%3A%2F%2Fmarc-abramowitz.com%2Fdownload%2Fpython%2Fbroken-0.2broken.tar.gz
  Running setup.py (path:/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build/setup.py) egg_info for package from http://marc-abramowitz.com/download/python/broken-0.2broken.tar.gz

    file broken.py (for module broken) not found
Installing collected packages: broken
  Running setup.py install for broken

    This package prints out UTF-8 stuff like:
    * return type of ‘main’ is not ‘int’
    * Björk Guðmundsdóttir [ˈpjœr̥k ˈkvʏðmʏntsˌtoʊhtɪr]Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build/setup.py", line 13, in <module>
        raise FakeError('this package designed to fail on install')
    __main__.FakeError: this package designed to fail on install
    Complete output from command /Users/marca/dev/git-repos/pip/py31.venv/bin/python -c "import setuptools;__file__='/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-4D5og8-record/install-record.txt --single-version-externally-managed --install-headers /Users/marca/dev/git-repos/pip/py31.venv/include/site/python3.1:


This package prints out UTF-8 stuff like:

* return type of ‘main’ is not ‘int’

* Björk Guðmundsdóttir [ˈpjœr̥k ˈkvʏðmʏntsˌtoʊhtɪr]Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build/setup.py", line 13, in <module>

    raise FakeError('this package designed to fail on install')

__main__.FakeError: this package designed to fail on install

----------------------------------------
Cleaning up...
Command /Users/marca/dev/git-repos/pip/py31.venv/bin/python -c "import setuptools;__file__='/var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-4D5og8-record/install-record.txt --single-version-externally-managed --install-headers /Users/marca/dev/git-repos/pip/py31.venv/include/site/python3.1 failed with error code 1 in /var/folders/nk/8f8f6wjn7v3b9cb2gjqf6vph0000gp/T/pip-IccA3M-build
Storing complete log in /Users/marca/.pip/pip.log

No UnicodeDecodeError here so this is good.

qwcode closed this Oct 10, 2013

when i run pip install gnureadline throws an error
/usr/bin/ld: cannot find -lncurses

collect2: error: ld returned 1 exit status

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1


Cleaning up...
Command /home/administrator/projects/rapidpro/env/bin/python -c "import setuptools, tokenize;file='/home/administrator/projects/rapidpro/env/build/gnureadline/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-21Niad-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/administrator/projects/rapidpro/env/include/site/python2.7 failed with error code 1 in /home/administrator/projects/rapidpro/env/build/gnureadline
Traceback (most recent call last):
File "/home/administrator/projects/rapidpro/env/bin/pip", line 11, in
sys.exit(main())
File "/home/administrator/projects/rapidpro/env/local/lib/python2.7/site-packages/pip/init.py", line 185, in main
return command.main(cmd_args)
File "/home/administrator/projects/rapidpro/env/local/lib/python2.7/site-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 66: ordinal not in range(128)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment