problems with LC_ALL=C #761

kmike · 2013-01-03T10:47:35Z

There is a pattern of using open(path, 'r').read() without explicit encoding in pip:

This pattern causes issues under Python 3.x with ASCII locale because file contents is decoded using ascii in this case and it fails for non-ascii data.

The first occurance (in setup.py) is clearly wrong IMHO: the utility function is used for reading pip's own index.txt and news.txt files which are encoded to utf8. It may cause the following exception:

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 871: ordinal not in range(128)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/var/folders/_5/cbsg50991szfp1r9nwxpx8580000gq/T/pip-61p_z7-build/setup.py", line 31, in <module>

    "\n\n" + read("docs", "news.txt"))

  File "/var/folders/_5/cbsg50991szfp1r9nwxpx8580000gq/T/pip-61p_z7-build/setup.py", line 9, in read

    return codecs.open(os.path.join(os.path.abspath(os.path.dirname(__file__)), *parts), 'r').read()

  File "/Users/kmike/svn/pip/.tox/py32-ascii/lib/python3.2/encodings/ascii.py", line 26, in decode

    return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 871: ordinal not in range(128)

if the following is added to pip's own tox.ini:

[testenv:py32-ascii]
basepython = python3.2
setenv = LC_ALL=C

The second is more tricky and I didn't debug it. It causes the following exception:

Unpacking /Users/kmike/svn/DAWG/.tox/dist/DAWG-0.5.3.zip
  Running setup.py egg_info for package from file:///Users/kmike/svn/DAWG/.tox/dist/DAWG-0.5.3.zip

Exception:
Traceback (most recent call last):
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/basecommand.py", line 107, in main
    status = self.run(options, args)
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/commands/install.py", line 256, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/req.py", line 1042, in prepare_files
    req_to_install.run_egg_info()
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/req.py", line 241, in run_egg_info
    "%(Name)s==%(Version)s" % self.pkg_info())
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/req.py", line 334, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/site-packages/pip-1.2.1-py3.2.egg/pip/req.py", line 274, in egg_info_data
    data = fp.read()
  File "/Users/kmike/svn/DAWG/.tox/py32-locale/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2130: ordinal not in range(128)

in https://github.com/kmike/DAWG testing suite (https://github.com/kmike/DAWG/blob/master/tox.ini).
DAWG package has a non-ascii README.rst (which is loaded to long_description, binary under Python 2.x and unicode under Python 3.x).

Under Python 2.x this works fine because req.py doesn't try to decode the data.

The text was updated successfully, but these errors were encountered:

hltbra · 2013-03-28T00:46:14Z

It seems the first point, the issue on setup.py, is solved by @qwcode. But the second point of req.py and fp.read() seems to be buggy yet.

kmike · 2013-03-28T00:53:25Z

For the record: I gave up on this :) I don't know how to make reliably installable packages with non-ascii metadata under Python 2.x.

As for the first point, I think the commited solution is fragile (because non-ascii chars could accidently be introduced again) and it is better to explicitly decode news.txt from ascii to prevent such errors in future.

kmike · 2013-03-28T00:57:37Z

Also, I think pip should have Travis/tox environment with LC_ALL=C to test against this.

domenkozar · 2013-08-08T22:18:01Z

Same here:

$ LANG="POSIX" pip install -e .
Obtaining file:///home/ielectric/dev/pyramid_jinja2
  Running setup.py egg_info for package from file:///home/ielectric/dev/pyramid_jinja2

Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/req.py", line 1047, in prepare_files
    req_to_install.run_egg_info()
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/req.py", line 262, in run_egg_info
    "%(Name)s==%(Version)s" % self.pkg_info())
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/req.py", line 355, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/site-packages/pip/req.py", line 295, in egg_info_data
    data = fp.read()
  File "/home/ielectric/dev/pyramid_jinja2/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1087: ordinal not in range(128)

domenkozar · 2013-10-16T16:55:20Z

cc @pauleveritt

pauleveritt · 2013-10-16T17:05:44Z

Indeed, this was the source of my problem. In one shell I had this in my environment:

LANG=en_US.UTF-8

...and pip worked. In another shell I didn't, and got the UnicodeDecodeError exception.

With that, building and uploading wheels to PyPI is only one "python setup.py bdist_wheel" away. Also, changes LC_ALL from C to UTF-8 to get around a bug in pip: pypa/pip#761 Change-Id: I508ab6436c29b23f6b8d56c0ec63eb26f3568819

rbarrois · 2013-12-26T00:23:32Z

Hi there,

This issue seems to still be running; the way I see it, there seems to be two options:

Consider that all setup.py files are UTF-8, expect users to use a UTF-8 locale, and don't change anything
Consider that all setup.py files are UTF-8, and add explicit codecs.open(__file__, 'r', 'utf-8') instead of open(__file__, 'r') in pip/req.py
Prepare for non-UTF-8, non-ascii setup.py, and emulate Python's handling of coding: utf-8 & co markers.

It seems that fixing the manually written micro-scripts around lines 600 and 285, as well as egg_info_data at line 296 of req.py are enough to install packages with UTF-8 setup.py and metadata with LC_ALL=C.

I'd like to contribute a patch to fix this issue; which option would you prefer?

pradyunsg · 2017-11-05T14:23:32Z

Is this still a thing?

auvipy · 2019-05-07T18:35:07Z

@pradyunsg I haven't face this in recent times. I saw this before though. This might not be existing anymore. But in the face of ambiguity, refuse the temptation to guess.

gutsytechster · 2020-04-21T17:19:49Z

Can anyone tell me how to reproduce this(if it's still a bug)? The initial comment seems to be referring to the older piece of code and I really am not able to understand much out ot it.

Though if it's not present, we can then close it. :)

uranusjr · 2020-04-21T17:32:07Z

I believe this part of the code base has been removed. pip now uses distlib to read legacy metadata (egg-info), which always uses UTF-8.

pradyunsg · 2020-04-21T21:29:05Z

In that case, let's close this! :)

kmike added a commit to kmike/pip that referenced this issue Jan 3, 2013

index.txt, news.txt and __init__.py are UTF-8 (see pypa#761)

136a981

kmike mentioned this issue Jan 4, 2013

index.txt, news.txt and __init__.py are UTF-8 (see #761) #762

Closed

rbarrois mentioned this issue Dec 26, 2013

pip install factory_boy error on python3.3 FactoryBoy/factory_boy#118

Closed

chriskuehl mentioned this issue Jan 23, 2015

Replace n-dash in HISTORY to workaround pip bug psf/requests#2420

Merged

kmike mentioned this issue Jan 4, 2016

Fix #463 by using io.open Lasagne/Lasagne#562

Merged

paveldedik mentioned this issue Sep 24, 2019

license: readable with encoding=ascii paveldedik/neon-py#4

Merged

parmentelat mentioned this issue Sep 26, 2019

UnicodeDecodeError: 'ascii' codec can't decode byte damianavila/RISE#514

Closed

pradyunsg closed this as completed Apr 21, 2020

douglatornell mentioned this issue Jun 18, 2020

python3.4 installation of nosy fails, non-ascii in CHANGELOG douglatornell/nosy#3

Open

github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

problems with LC_ALL=C #761

problems with LC_ALL=C #761

kmike commented Jan 3, 2013

hltbra commented Mar 28, 2013

kmike commented Mar 28, 2013

kmike commented Mar 28, 2013

domenkozar commented Aug 8, 2013

domenkozar commented Oct 16, 2013

pauleveritt commented Oct 16, 2013

rbarrois commented Dec 26, 2013

pradyunsg commented Nov 5, 2017

auvipy commented May 7, 2019

gutsytechster commented Apr 21, 2020

uranusjr commented Apr 21, 2020

pradyunsg commented Apr 21, 2020

problems with LC_ALL=C #761

problems with LC_ALL=C #761

Comments

kmike commented Jan 3, 2013

hltbra commented Mar 28, 2013

kmike commented Mar 28, 2013

kmike commented Mar 28, 2013

domenkozar commented Aug 8, 2013

domenkozar commented Oct 16, 2013

pauleveritt commented Oct 16, 2013

rbarrois commented Dec 26, 2013

pradyunsg commented Nov 5, 2017

auvipy commented May 7, 2019

gutsytechster commented Apr 21, 2020

uranusjr commented Apr 21, 2020

pradyunsg commented Apr 21, 2020