New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error running pip install -r <requirements file> #2865

Closed
miloandmilk opened this Issue Jun 3, 2015 · 4 comments

Comments

Projects
None yet
4 participants
@miloandmilk

miloandmilk commented Jun 3, 2015

Python 3.4 / Windows 7 / Powershell / pip version 7.0.3

Running

pip install -r .\stable-req.txt

with stable-req.txt containing requests==2.7.0

results in the following exception:

Traceback (most recent call last):
  File "D:\frh\env2\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "D:\frh\env2\lib\site-packages\pip\commands\install.py", line 268, in run
    wheel_cache
  File "D:\frh\env2\lib\site-packages\pip\basecommand.py", line 287, in populate_requirement_set
    wheel_cache=wheel_cache):
  File "D:\frh\env2\lib\site-packages\pip\req\req_file.py", line 86, in parse_requirements
    for req in req_iter:
  File "D:\frh\env2\lib\site-packages\pip\req\req_file.py", line 130, in process_line
    wheel_cache=wheel_cache
  File "D:\frh\env2\lib\site-packages\pip\req\req_install.py", line 207, in from_line
    wheel_cache=wheel_cache)
  File "D:\frh\env2\lib\site-packages\pip\req\req_install.py", line 66, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2960, in parse
    reqs = list(parse_requirements(s))
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2904, in parse_requirements
    "version spec")
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2869, in scan_list
    raise ValueError(msg, line, "at", line[p:])
ValueError: ('Expected version spec in', 'ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00'
, 'at', '\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00')

Reproduced on two pcs and one Windows Server 2008 r2

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Jun 9, 2015

Contributor

looks like a big pile of encoding confusion. Is that UCS32 perhaps?

Contributor

rbtcollins commented Jun 9, 2015

looks like a big pile of encoding confusion. Is that UCS32 perhaps?

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Aug 19, 2015

Contributor
>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8')
b'\xc3\xbf\xc3\xber\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00'
>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8').decode('utf16')
'뿃뻃requests==2.7.0'

and ÿþ seems to corresponds to the UTF-16 (LE) BOM (cf https://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding), which would match.

So I'd guess the req file was encoded in utf16 but python opened it as utf8 (returned from locale.getpreferredencoding() if I understand correctly https://docs.python.org/3.4/library/functions.html#open).

Not sure what pip could do to improve the user experience on such case ...

Contributor

xavfernandez commented Aug 19, 2015

>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8')
b'\xc3\xbf\xc3\xber\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00'
>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8').decode('utf16')
'뿃뻃requests==2.7.0'

and ÿþ seems to corresponds to the UTF-16 (LE) BOM (cf https://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding), which would match.

So I'd guess the req file was encoded in utf16 but python opened it as utf8 (returned from locale.getpreferredencoding() if I understand correctly https://docs.python.org/3.4/library/functions.html#open).

Not sure what pip could do to improve the user experience on such case ...

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Aug 20, 2015

Member

Yeah, that's a UTF-16 file. Was it created by redirection in Powershell (some_cmd > stable-req.txt) ? As that is typically where I end up with UTF-16 files that I didn't want ;-)

Auto-detecting (guessing) encodings is generally a bad idea, and probably something pip shouldn't get into, but I guess that looking for a BOM when reading the requirements file, and choosing the encoding from that if it's found, would be a useful feature to help with this common Windows issue. (Although it's mainly a Windows issue, the BOM detection can be added for all platforms, as I can't see how it would cause a problem on any other platform).

Member

pfmoore commented Aug 20, 2015

Yeah, that's a UTF-16 file. Was it created by redirection in Powershell (some_cmd > stable-req.txt) ? As that is typically where I end up with UTF-16 files that I didn't want ;-)

Auto-detecting (guessing) encodings is generally a bad idea, and probably something pip shouldn't get into, but I guess that looking for a BOM when reading the requirements file, and choosing the encoding from that if it's found, would be a useful feature to help with this common Windows issue. (Although it's mainly a Windows issue, the BOM detection can be added for all platforms, as I can't see how it would cause a problem on any other platform).

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Feb 12, 2016

Contributor

Duplicate of #733

Contributor

xavfernandez commented Feb 12, 2016

Duplicate of #733

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment