New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip wheel fails with UnicodeDecodeError when setup.py and default encoding don't match #2042

Closed
SpotlightKid opened this Issue Sep 17, 2014 · 9 comments

Comments

Projects
None yet
5 participants
@SpotlightKid

SpotlightKid commented Sep 17, 2014

When trying to build a wheel for a package, whose setup.py has a non-ascii encoding (even with proper encoding declaration), it fails with the following error:

$ pip wheel --wheel-dir .  -v pysmfDownloading/unpacking pysmf
  Downloading pysmf-0.1.1.tar.gz (50kB): 
  Downloading from URL https://pypi.python.org/packages/source/p/pysmf/pysmf-0.1.1.tar.gz#md5=8354f3cd7752b12e2668d4ed9aaa25c9 (from https://pypi.python.org/simple/pysmf/)
...Downloading pysmf-0.1.1.tar.gz (50kB): 50kB downloaded
  Storing download in cache at ./.pip_download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpysmf%2Fpysmf-0.1.1.tar.gz
  Running setup.py (path:/home/chris/tmp/pip_build_chris/pysmf/setup.py) egg_info for package pysmf
    running egg_info
    creating pip-egg-info/pysmf.egg-info
    writing pip-egg-info/pysmf.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pysmf.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pysmf.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
Building wheels for collected packages: pysmf
  Running setup.py bdist_wheel for pysmf
  Destination directory: /home/chris
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1623: ordinal not in range(128)
  Complete output from command /usr/bin/python -c "import setuptools;__file__='/home/chris/tmp/pip_build_chris/pysmf/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /home/chris:
  Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode

    return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1623: ordinal not in range(128)

----------------------------------------
  Failed building wheel for pysmf
Failed to build pysmf
Cleaning up...
  Removing temporary dir /home/chris/tmp/pip_build_chris...

The offending code is calls to read() in the command to read and run setup.py. pip passes this to Python on the command line. In plain, unraveled code it would look like:

import setuptools
__file__ = 'setup.py'
fp = open(__file__)
setup = fp.read()
setup = setup.replace('\r\n', '\n')
code = compile(setup, __file__, 'exec')
exec(code)

The fp.read() call uses the system dependent default encoding (locale.getpreferredencoding()), since the open() call didn't specify an encoding. Therefore the above pip wheel invocation failed on my Arm system, but not on my Linux x86_64 laptop.

To fix this, pip would need to honor the source encoding declaration in setup.py, which is a rather complicated logic. Why does pip read & compile the setup.py file itself anyway? Could it not just call subprocess.check_call([sys.executable, 'setup.py' 'bdist_wheel'])?

@nils-werner

This comment has been minimized.

Show comment
Hide comment
@nils-werner

nils-werner Apr 6, 2015

Note this only happens when using Python 3.

nils-werner commented Apr 6, 2015

Note this only happens when using Python 3.

@hansmbakker

This comment has been minimized.

Show comment
Hide comment
@hansmbakker

hansmbakker Apr 22, 2015

I'm also having trouble building wheels out of phue and python-nest using pip3. This is because their setup.py's are utf8.

hansmbakker commented Apr 22, 2015

I'm also having trouble building wheels out of phue and python-nest using pip3. This is because their setup.py's are utf8.

@gumblex

This comment has been minimized.

Show comment
Hide comment
@gumblex

gumblex Nov 24, 2015

The offending code is at https://github.com/pypa/pip/blob/develop/pip/wheel.py#L687
@SpotlightKid The 'complicated logic' is implemented in tokenize.detect_encoding (we can use tokenize.open)

gumblex commented Nov 24, 2015

The offending code is at https://github.com/pypa/pip/blob/develop/pip/wheel.py#L687
@SpotlightKid The 'complicated logic' is implemented in tokenize.detect_encoding (we can use tokenize.open)

@SpotlightKid

This comment has been minimized.

Show comment
Hide comment
@SpotlightKid

SpotlightKid Nov 24, 2015

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

SpotlightKid commented Nov 24, 2015

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

Since sdist install works, we could use the same trick used for sdist install: https://github.com/pypa/pip/blob/e42e822/pip/req/req_install.py#L478-L482

Contributor

xavfernandez commented Nov 24, 2015

Since sdist install works, we could use the same trick used for sdist install: https://github.com/pypa/pip/blob/e42e822/pip/req/req_install.py#L478-L482

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

Because we want to use setuptools for all setup.py invocation, this is mandatory to be able to perform a clean uninstall.

Contributor

xavfernandez commented Nov 24, 2015

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

Because we want to use setuptools for all setup.py invocation, this is mandatory to be able to perform a clean uninstall.

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

Also related to #1233

Contributor

xavfernandez commented Nov 24, 2015

Also related to #1233

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 24, 2015

Use a single setuptools shim
for all setup.py invocations
bdist_wheel will now use tokenize in Python 3 just like for install
fixes pypa#2042
@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

@gumblex could you try #3265 (or https://github.com/xavfernandez/pip/tree/setuptool_shim) and confirm it is fixed ?

Contributor

xavfernandez commented Nov 24, 2015

@gumblex could you try #3265 (or https://github.com/xavfernandez/pip/tree/setuptool_shim) and confirm it is fixed ?

@gumblex

This comment has been minimized.

Show comment
Hide comment
@gumblex

gumblex Nov 25, 2015

It works.

gumblex commented Nov 25, 2015

It works.

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 25, 2015

Use a single setuptools shim
for all setup.py invocations
bdist_wheel will now use tokenize in Python 3 just like for install
fixes pypa#2042

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 26, 2015

Use a single setuptools shim
for all setup.py invocations
bdist_wheel will now use tokenize in Python 3 just like for install
fixes pypa#2042

@xavfernandez xavfernandez added the T: bug label Nov 26, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment