New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip freeze crashes with Invalid requirement, parse error at "'-lxc==0'" #3681

Closed
ssbarnea opened this Issue May 12, 2016 · 39 comments

Comments

Projects
None yet
@ssbarnea
Contributor

ssbarnea commented May 12, 2016

  • Pip version: 8.1.2
  • Python version: 3.4
  • Operating System: Ubuntu 15.10

Description:

While trying to run pip freeze we got a very long stacktrace that is ending with the line below:

file "/var/lib/jenkins/jobs/exampletech/jobs/example-portal/branches/devops/workspace/.tox/py34-linux/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-lxc==0.'"

While I suspect that one of the packages installed from public pypi.python.org may have some weird version, i do find unacceptable to block pip freeze from working completely.

@ssbarnea

This comment has been minimized.

Show comment
Hide comment
@ssbarnea

ssbarnea May 12, 2016

Contributor

I was able to discover that the weird package is -lxc==0.1 and is installed as system package. pip freeze works at the system level but if you create a virtual environment that is using --system-site-packages, and try to use the pip from inside the environment you will get this error.

Contributor

ssbarnea commented May 12, 2016

I was able to discover that the weird package is -lxc==0.1 and is installed as system package. pip freeze works at the system level but if you create a virtual environment that is using --system-site-packages, and try to use the pip from inside the environment you will get this error.

@ssbarnea ssbarnea changed the title from pip freeze crashes while to pip freeze crashes with Invalid requirement, parse error at "'-lxc==0'" May 12, 2016

@bricewge

This comment has been minimized.

Show comment
Hide comment
@bricewge

bricewge May 15, 2016

I have the same issue with debian-8.4, it happen with pip from the debian repository 1.5.6 and also with the last 8.1.2 from pypi. However 8.8.1 from pypi is working and pip3 freeze return -lxc==0.1 on the first line.

bricewge commented May 15, 2016

I have the same issue with debian-8.4, it happen with pip from the debian repository 1.5.6 and also with the last 8.1.2 from pypi. However 8.8.1 from pypi is working and pip3 freeze return -lxc==0.1 on the first line.

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler May 18, 2016

I've not seen this happen myself yet, but some contributors to my project (using tox) saw this happen with -sqlite3:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2719, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2445, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1823, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-sqlite3'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/__init__.py", line 264, in from_dist
    req = dist.as_requirement()
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-sqlite3'"

The-Compiler commented May 18, 2016

I've not seen this happen myself yet, but some contributors to my project (using tox) saw this happen with -sqlite3:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2719, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2445, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1823, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-sqlite3'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/__init__.py", line 264, in from_dist
    req = dist.as_requirement()
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/home/elronnd/qutebrowser/.venv/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-sqlite3'"
@Elronnd

This comment has been minimized.

Show comment
Hide comment
@Elronnd

Elronnd May 18, 2016

As said user (contributor? not hardly), I just want to note that the OS is FreeBSD 10.3, and that I get the error on python versions 3.4 and 3.5.

Elronnd commented May 18, 2016

As said user (contributor? not hardly), I just want to note that the OS is FreeBSD 10.3, and that I get the error on python versions 3.4 and 3.5.

@diogobaeder

This comment has been minimized.

Show comment
Hide comment
@diogobaeder

diogobaeder May 18, 2016

Same thing here, and I confirm that in pip 8.1.1 I can get normal output from pip freeze.

It's pretty weird for a dependency to get listed starting with a dash, though.

diogobaeder commented May 18, 2016

Same thing here, and I confirm that in pip 8.1.1 I can get normal output from pip freeze.

It's pretty weird for a dependency to get listed starting with a dash, though.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 18, 2016

Member

Hmm, a dependency starting with a - is invalid... why do they have that? Is that inside of the requirements.txt or is it getting added somewhere?

Member

dstufft commented May 18, 2016

Hmm, a dependency starting with a - is invalid... why do they have that? Is that inside of the requirements.txt or is it getting added somewhere?

@bricewge

This comment has been minimized.

Show comment
Hide comment
@bricewge

bricewge May 18, 2016

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

bricewge commented May 18, 2016

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 18, 2016

Member

Probably should file a ticket with Debian on the lxc package. Names with a leading or trailing character that isn't an ascii alpha numeric are not valid. They worked previously due to setuptools attempting to accept whatever garbage was there (no matter how nonsensical) but things have gotten stricter in the past couple years in an attempt to better define behavior.

Sent from my iPhone

On May 18, 2016, at 6:11 AM, bricewge notifications@github.com wrote:

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

You are receiving this because you commented.
Reply to this email directly or view it on GitHub

Member

dstufft commented May 18, 2016

Probably should file a ticket with Debian on the lxc package. Names with a leading or trailing character that isn't an ascii alpha numeric are not valid. They worked previously due to setuptools attempting to accept whatever garbage was there (no matter how nonsensical) but things have gotten stricter in the past couple years in an attempt to better define behavior.

Sent from my iPhone

On May 18, 2016, at 6:11 AM, bricewge notifications@github.com wrote:

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

You are receiving this because you commented.
Reply to this email directly or view it on GitHub

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 18, 2016

Member

That being said. We should probably just issue a warning and skip this package rather than bomb out completely.

Sent from my iPhone

On May 18, 2016, at 6:11 AM, bricewge notifications@github.com wrote:

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

You are receiving this because you commented.
Reply to this email directly or view it on GitHub

Member

dstufft commented May 18, 2016

That being said. We should probably just issue a warning and skip this package rather than bomb out completely.

Sent from my iPhone

On May 18, 2016, at 6:11 AM, bricewge notifications@github.com wrote:

For -lxc==0.1 it come from /usr/lib/python3/dist-packages/_lxc-0.1.egg-info owned by the package lxc of debian stable. Here is it's content:

Metadata-Version: 1.0
Name: _lxc
Version: 0.1
Summary: LXC
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

You are receiving this because you commented.
Reply to this email directly or view it on GitHub

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco May 18, 2016

Member

Names with a leading or trailing character that isn't an ascii alpha numeric are not valid... things have gotten stricter in the past couple years in an attempt to better define behavior.

_lxc seems to be a pretty reasonable use of the leading underscore. It's very common practice in Python to precede a name with a leading underscore to signal that it's not a public, official name. I suspect Debian did this to avoid clashing with a potential public lxc package. Regardless, it seems reasonable to me that any valid Python identifier should be allowed for a package name. The name _lxc is a valid Python package name, so should probably also be allowed as a distribution name. I suspect the stricter imposition here will lead to names like x_lxc or xlxc, which I don't see as better behavior, but rather worse.

In any case, I'm not familiar with the motivations for adding these restrictions, so maybe there's a good reason behind it. I just wanted to share my reaction to the condition.

Member

jaraco commented May 18, 2016

Names with a leading or trailing character that isn't an ascii alpha numeric are not valid... things have gotten stricter in the past couple years in an attempt to better define behavior.

_lxc seems to be a pretty reasonable use of the leading underscore. It's very common practice in Python to precede a name with a leading underscore to signal that it's not a public, official name. I suspect Debian did this to avoid clashing with a potential public lxc package. Regardless, it seems reasonable to me that any valid Python identifier should be allowed for a package name. The name _lxc is a valid Python package name, so should probably also be allowed as a distribution name. I suspect the stricter imposition here will lead to names like x_lxc or xlxc, which I don't see as better behavior, but rather worse.

In any case, I'm not familiar with the motivations for adding these restrictions, so maybe there's a good reason behind it. I just wanted to share my reaction to the condition.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 18, 2016

Member

@jaraco The reason for it is largely to make it harder to make names that appear to be one thing but are actually another thing and in this vein it's entirely unlike Python identifiers because you don't have to worry about people trying to attack you using a similar identifier. This also goes back to some history in the setuptools behavior too which is why we went a bit stricter overall.

For instance, folks create create a guide telling people to copy/paste pip install _Django and it's not entirely unreasonable for people to miss the fact that there is a _ there and then have them install _Django from PyPI (which is obviously not the same thing as Django. This is made worse by the fact that older versions of setuptools and pip would attempt to escape most characters as a -, so you could have something like pip install ԁjango (which is not the same as pip install django, the first character is not the ascii letterd) which would be escaped as-jango. This is also the reason why we want to reduce runs of these characters down to a single one so thatdjango_toolsanddjango__tools`` don't resolve to different things.

In addition to all of that, I'm not sure it makes sense to have this value there at all. Either _lxc is something that they expect people to be able to interact with as part of the dependency of their Python program (so it'll be in install_requires, or __requires__, or whatever) in which case I'd argue that it is not a "private, unofficial" name, but a public name that just happens to not be on PyPI (assuming it is in fact, not on PyPI under a different name) or it's not something that people are expected to interact with in that way, and they should just omit the .egg-info directory all together since it's not really a package at that point.

Member

dstufft commented May 18, 2016

@jaraco The reason for it is largely to make it harder to make names that appear to be one thing but are actually another thing and in this vein it's entirely unlike Python identifiers because you don't have to worry about people trying to attack you using a similar identifier. This also goes back to some history in the setuptools behavior too which is why we went a bit stricter overall.

For instance, folks create create a guide telling people to copy/paste pip install _Django and it's not entirely unreasonable for people to miss the fact that there is a _ there and then have them install _Django from PyPI (which is obviously not the same thing as Django. This is made worse by the fact that older versions of setuptools and pip would attempt to escape most characters as a -, so you could have something like pip install ԁjango (which is not the same as pip install django, the first character is not the ascii letterd) which would be escaped as-jango. This is also the reason why we want to reduce runs of these characters down to a single one so thatdjango_toolsanddjango__tools`` don't resolve to different things.

In addition to all of that, I'm not sure it makes sense to have this value there at all. Either _lxc is something that they expect people to be able to interact with as part of the dependency of their Python program (so it'll be in install_requires, or __requires__, or whatever) in which case I'd argue that it is not a "private, unofficial" name, but a public name that just happens to not be on PyPI (assuming it is in fact, not on PyPI under a different name) or it's not something that people are expected to interact with in that way, and they should just omit the .egg-info directory all together since it's not really a package at that point.

@diogobaeder

This comment has been minimized.

Show comment
Hide comment
@diogobaeder

diogobaeder May 18, 2016

I understand the motivation and support the decision of disallowing strange characters in package names, and I do find it ugly and not-pythonic to start a package with underscore; but I think underscores shouldn't be disallowed, since they're not ambiguous for the human eye. Not even in the start of the package name - though I personally find it ugly.

diogobaeder commented May 18, 2016

I understand the motivation and support the decision of disallowing strange characters in package names, and I do find it ugly and not-pythonic to start a package with underscore; but I think underscores shouldn't be disallowed, since they're not ambiguous for the human eye. Not even in the start of the package name - though I personally find it ugly.

@bricewge

This comment has been minimized.

Show comment
Hide comment
@bricewge

bricewge commented May 18, 2016

@ssbarnea

This comment has been minimized.

Show comment
Hide comment
@ssbarnea

ssbarnea May 18, 2016

Contributor

The problem is very easy to replicate with any recent Ubuntu distribution (like 15.10). Just do

apt-get install python-lxc
# create a virtual environment that has --system-site-packages and activate it
pip freeze

While I totally agree that the naming is not good, I find unacceptable to for pip to crash. That's what we need to fix first, the pip failure.

We will always have broken package names, as we cannot control the ability of people to make mistakes. What we can do is to improve our ability to deal with them.

Contributor

ssbarnea commented May 18, 2016

The problem is very easy to replicate with any recent Ubuntu distribution (like 15.10). Just do

apt-get install python-lxc
# create a virtual environment that has --system-site-packages and activate it
pip freeze

While I totally agree that the naming is not good, I find unacceptable to for pip to crash. That's what we need to fix first, the pip failure.

We will always have broken package names, as we cannot control the ability of people to make mistakes. What we can do is to improve our ability to deal with them.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore May 18, 2016

Member

Agreed pip shouldn't crash. A warning and skipping the package seems like a reasonable fix (as @dstufft suggested). PRs, as usual, welcome :-)

Also, help in making people more aware of the standards - either in the form of passing the word around, or PRs to documentation to clarify it, or as in this case bug reports to projects that don't conform - is hugely appreciated. The biggest problem we have in Python packaging is making people aware when we tighten up risks/loopholes (as in this case), or enforce long-standing but previously ignored rules. So getting the word out is a great help.

Member

pfmoore commented May 18, 2016

Agreed pip shouldn't crash. A warning and skipping the package seems like a reasonable fix (as @dstufft suggested). PRs, as usual, welcome :-)

Also, help in making people more aware of the standards - either in the form of passing the word around, or PRs to documentation to clarify it, or as in this case bug reports to projects that don't conform - is hugely appreciated. The biggest problem we have in Python packaging is making people aware when we tighten up risks/loopholes (as in this case), or enforce long-standing but previously ignored rules. So getting the word out is a great help.

@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw May 19, 2016

There are a couple of problems with this issue that I'd like to untangle.

Clearly pip shouldn't crash. I'd argue that it should never crash. There's really very little an end user can do when it does except get frustrated. In this case, there's no obvious repair the end user can guess to do based on the error. Where it finds bogus data, pip should complain, maybe try to offer a way out, and just continue on, ignoring the bogus data.

It's also the case that the problem is very far removed from the source of the problem. What did the lxc developers do wrong to create this situation? Why did the tools allow them to create a bogus .egg-info file that can break end users without the lxc maintainers being aware of the problem? How do we stop them from creating the bogus data in the first place?

I'm in the camp that disallowing _lxc as a package name seems very weird. @dstufft mentioned to me that package names and module names aren't the same thing, pointing to the fact that foo-bar is a valid package name but not a valid module name. And re: the point above about attempts to disguise malicious packages with almost correct looking package names. Fair points all, but I'm not sure the specific case of using a leading underscore should be disallowed, and if it must be then we have to make sure such packages can never be created, and won't break a system if they are specially crafted to be malicious.

warsaw commented May 19, 2016

There are a couple of problems with this issue that I'd like to untangle.

Clearly pip shouldn't crash. I'd argue that it should never crash. There's really very little an end user can do when it does except get frustrated. In this case, there's no obvious repair the end user can guess to do based on the error. Where it finds bogus data, pip should complain, maybe try to offer a way out, and just continue on, ignoring the bogus data.

It's also the case that the problem is very far removed from the source of the problem. What did the lxc developers do wrong to create this situation? Why did the tools allow them to create a bogus .egg-info file that can break end users without the lxc maintainers being aware of the problem? How do we stop them from creating the bogus data in the first place?

I'm in the camp that disallowing _lxc as a package name seems very weird. @dstufft mentioned to me that package names and module names aren't the same thing, pointing to the fact that foo-bar is a valid package name but not a valid module name. And re: the point above about attempts to disguise malicious packages with almost correct looking package names. Fair points all, but I'm not sure the specific case of using a leading underscore should be disallowed, and if it must be then we have to make sure such packages can never be created, and won't break a system if they are specially crafted to be malicious.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore May 19, 2016

Member

I'm OK with changing the PEP (can't see a reference to the number here, I presume @dstufft would know) to allow leading underscores. I also agree that pip shouldn't crash, IMO it should warn and skip any invalid package names (PRs welcome, as usual :-))

As far as not being allowed to build/upload/etc packages with invalid names, that's an entirely reasonable suggestion, but not one for the pip tracker. Someone would have to raise issues against setuptools, and distutils (at a minimum - what about tools like flit?) saying that they don't enforce PEP xxx. I don't know if PyPI allows registration of new non-compliant names, nor do I know what it does about existing ones (I suspect nothing, as the PyPI codebase is pretty much on life support). I assume that Warehouse enforces the rules.

But I don't think pip should be forced to continue allowing legacy package names until everyone else changes. Someone has to be the first to implement new standards, why not pip? We've got a history of pip driving adoption of new standards which has so far worked well, no reason to alter that now.

Member

pfmoore commented May 19, 2016

I'm OK with changing the PEP (can't see a reference to the number here, I presume @dstufft would know) to allow leading underscores. I also agree that pip shouldn't crash, IMO it should warn and skip any invalid package names (PRs welcome, as usual :-))

As far as not being allowed to build/upload/etc packages with invalid names, that's an entirely reasonable suggestion, but not one for the pip tracker. Someone would have to raise issues against setuptools, and distutils (at a minimum - what about tools like flit?) saying that they don't enforce PEP xxx. I don't know if PyPI allows registration of new non-compliant names, nor do I know what it does about existing ones (I suspect nothing, as the PyPI codebase is pretty much on life support). I assume that Warehouse enforces the rules.

But I don't think pip should be forced to continue allowing legacy package names until everyone else changes. Someone has to be the first to implement new standards, why not pip? We've got a history of pip driving adoption of new standards which has so far worked well, no reason to alter that now.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 19, 2016

Member

PEP 508 is the most relevant one to this particular behavior, though PEP 508 copied the restrictions from PEP 426. PyPI does not allow invalid names for a good long time now (I want to say a year+ but I could be misremembering) and when I made that change we also scrubbed the hand full of invalid names from the database.

Changing the restriction would require updating the PEP (since we don't want to start going back down the path of implementation defined behavior) though I'm not sure if it's a good idea-- I definitely think we should not relax the restrictions on PyPI and I think it is simpler if the same restrictions exist everywhere instead of people having to remember which set of restrictions apply where (transitional issues with some pieces getting updated faster than others aside). In any case, discussions about whether or not to change the PEP belong on distutils-sig rather than the pip issue tracker.

I also agree that pip should not relax the rules unless the PEP changes. Historically we've found that the only way we really push people forward is to force their hand through breaking the edge cases we no longer support. If we don't, people just never change their behavior and we're never able to deprecate anything.

I also agree that setuptools should start to enforce not using an invalid name (though at this point, if people are going to try and get the PEP changed, we should wait until that gets decided if we will or not).

Member

dstufft commented May 19, 2016

PEP 508 is the most relevant one to this particular behavior, though PEP 508 copied the restrictions from PEP 426. PyPI does not allow invalid names for a good long time now (I want to say a year+ but I could be misremembering) and when I made that change we also scrubbed the hand full of invalid names from the database.

Changing the restriction would require updating the PEP (since we don't want to start going back down the path of implementation defined behavior) though I'm not sure if it's a good idea-- I definitely think we should not relax the restrictions on PyPI and I think it is simpler if the same restrictions exist everywhere instead of people having to remember which set of restrictions apply where (transitional issues with some pieces getting updated faster than others aside). In any case, discussions about whether or not to change the PEP belong on distutils-sig rather than the pip issue tracker.

I also agree that pip should not relax the rules unless the PEP changes. Historically we've found that the only way we really push people forward is to force their hand through breaking the edge cases we no longer support. If we don't, people just never change their behavior and we're never able to deprecate anything.

I also agree that setuptools should start to enforce not using an invalid name (though at this point, if people are going to try and get the PEP changed, we should wait until that gets decided if we will or not).

@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw May 19, 2016

Okay, if PEP 508/426 already define the name limitations, and PyPI already adheres to these, then I think you're right that we're down to two things that need to be done: make pip more robust in the face of invalid names, and make setuptools reject invalid names at the start. (Admittedly the latter isn't relevant for this project/issue.)

warsaw commented May 19, 2016

Okay, if PEP 508/426 already define the name limitations, and PyPI already adheres to these, then I think you're right that we're down to two things that need to be done: make pip more robust in the face of invalid names, and make setuptools reject invalid names at the start. (Admittedly the latter isn't relevant for this project/issue.)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 19, 2016

Member

Not to be dismissive or anything, but the quickest way to get this resolved would be if someone else can make a PR for it. I can commit to reviewing it if someone does, otherwise I'll try to get to it as quickly as I can but I have a lot of competing priorities at the moment.

Member

dstufft commented May 19, 2016

Not to be dismissive or anything, but the quickest way to get this resolved would be if someone else can make a PR for it. I can commit to reviewing it if someone does, otherwise I'll try to get to it as quickly as I can but I have a lot of competing priorities at the moment.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft May 26, 2016

Member

Fixed by #3704

Member

dstufft commented May 26, 2016

Fixed by #3704

@waynew

This comment has been minimized.

Show comment
Hide comment
@waynew

waynew Aug 3, 2016

I'm actually still getting this error on Arch Linux:

⚘ python -m pip --version                                                                                                                                                                                                                                                      $? 2  12:21:26
pip 8.1.2 from /usr/lib/python3.5/site-packages (python 3.5)
⚘ python -m pip freeze                                                                                                                                                                                                                                                               12:21:19
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2719, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2445, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1823, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-lxc==0.'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.5/site-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/lib/python3.5/site-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/lib/python3.5/site-packages/pip/__init__.py", line 264, in from_dist
    req = dist.as_requirement()
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-lxc==0.'"

waynew commented Aug 3, 2016

I'm actually still getting this error on Arch Linux:

⚘ python -m pip --version                                                                                                                                                                                                                                                      $? 2  12:21:26
pip 8.1.2 from /usr/lib/python3.5/site-packages (python 3.5)
⚘ python -m pip freeze                                                                                                                                                                                                                                                               12:21:19
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2719, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 2445, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pyparsing.py", line 1823, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-lxc==0.'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.5/site-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/lib/python3.5/site-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/lib/python3.5/site-packages/pip/__init__.py", line 264, in from_dist
    req = dist.as_requirement()
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/lib/python3.5/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-lxc==0.'"
@diogobaeder

This comment has been minimized.

Show comment
Hide comment
@diogobaeder

diogobaeder Aug 3, 2016

The fix was committed to master, but not yet released. I suggest you use 8.1.1 instead of 8.1.2 (I'm using it locally, haven't got any problems so far, but I'm eager to see a 8.1.3 release)

diogobaeder commented Aug 3, 2016

The fix was committed to master, but not yet released. I suggest you use 8.1.1 instead of 8.1.2 (I'm using it locally, haven't got any problems so far, but I'm eager to see a 8.1.3 release)

@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw Aug 3, 2016

FWIW, I cherrypicked the fix into the Debian/Ubuntu version of the package.

warsaw commented Aug 3, 2016

FWIW, I cherrypicked the fix into the Debian/Ubuntu version of the package.

@waynew

This comment has been minimized.

Show comment
Hide comment
@waynew

waynew Aug 5, 2016

@diogobaeder That actually works for me. I guess I'll hang out at 8.1.1 until 8.1.3 is released 😛

waynew commented Aug 5, 2016

@diogobaeder That actually works for me. I guess I'll hang out at 8.1.1 until 8.1.3 is released 😛

@diogobaeder

This comment has been minimized.

Show comment
Hide comment
@diogobaeder

diogobaeder Aug 5, 2016

Me too, that's what I'm doing.

Nice job, @warsaw :-)

diogobaeder commented Aug 5, 2016

Me too, that's what I'm doing.

Nice job, @warsaw :-)

@Mausy5043

This comment has been minimized.

Show comment
Hide comment
@Mausy5043

Mausy5043 Aug 8, 2016

Just installed pip3 on Ubuntu 16.10.1.

$ pip3 list
-lxc (0.1)
apt-xapian-index (0.47)
chardet (2.3.0)
command-not-found (0.3)
devscripts (2.16.2ubuntu3)
language-selector (0.1)
Magic-file-extensions (0.2)
mysqlclient (1.3.7)
pip (8.1.1)
pycurl (7.43.0)
pygobject (3.20.0)
python-apt (1.1.0b1)
python-debian (0.1.27)
python-systemd (231)
setuptools (20.7.0)
six (1.10.0)
ufw (0.35)
unattended-upgrades (0.1)
wheel (0.29.0)
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

pip3 freeze falls over as discussed in this thread.

uname: Linux boson 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

EDIT: Also: I expect that -- as I installed pip3 -- the proposed "You should consider upgrading via the 'pip install --upgrade pip' command" will not work for me?

EDIT2: Also funny to see that:

$ python -m pip --version
/usr/bin/python: No module named pip
$ python -m pip3 --version
/usr/bin/python: No module named pip3
$ python3 -m pip --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)

Mausy5043 commented Aug 8, 2016

Just installed pip3 on Ubuntu 16.10.1.

$ pip3 list
-lxc (0.1)
apt-xapian-index (0.47)
chardet (2.3.0)
command-not-found (0.3)
devscripts (2.16.2ubuntu3)
language-selector (0.1)
Magic-file-extensions (0.2)
mysqlclient (1.3.7)
pip (8.1.1)
pycurl (7.43.0)
pygobject (3.20.0)
python-apt (1.1.0b1)
python-debian (0.1.27)
python-systemd (231)
setuptools (20.7.0)
six (1.10.0)
ufw (0.35)
unattended-upgrades (0.1)
wheel (0.29.0)
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

pip3 freeze falls over as discussed in this thread.

uname: Linux boson 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

EDIT: Also: I expect that -- as I installed pip3 -- the proposed "You should consider upgrading via the 'pip install --upgrade pip' command" will not work for me?

EDIT2: Also funny to see that:

$ python -m pip --version
/usr/bin/python: No module named pip
$ python -m pip3 --version
/usr/bin/python: No module named pip3
$ python3 -m pip --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)
@waynew

This comment has been minimized.

Show comment
Hide comment
@waynew

waynew Aug 8, 2016

@Mausy5043 That's exactly what you should see.

pip is a package that's installed to a particular python version. AFAIK for most linux distros, pip is installed with python and pip3 is installed with python3. Note, that it's still the pip package, regardless of python or python3. The only difference is that with pip3 it does something like ln -s /path/to/python/bin/pip /usr/bin/pip3.

waynew commented Aug 8, 2016

@Mausy5043 That's exactly what you should see.

pip is a package that's installed to a particular python version. AFAIK for most linux distros, pip is installed with python and pip3 is installed with python3. Note, that it's still the pip package, regardless of python or python3. The only difference is that with pip3 it does something like ln -s /path/to/python/bin/pip /usr/bin/pip3.

@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw Aug 9, 2016

@Mausy5043 are you sure you tried this on an up-to-date Ubuntu 16.10? I just tried this in a Yakkety chroot and pip3 freeze with python3-lxc installed works fine. You do get a warning:

$ pip3 list
-lxc (0.1)
nose2 (0.6.5)
pip (8.1.2)
setuptools (20.10.1)
six (1.10.0)
wheel (0.29.0)

$ pip3 freeze
Could not parse requirement: -lxc
nose2==0.6.5
six==1.10.0

However, I noticed that the patch is not installed on the Ubuntu 16.04 LTS (Xenial) version, so you do get that crash. I've already cherry picked one fix into the Xenial SRU, but I missed this one.

LP: #1611503 tracks the SRU for this problem.

warsaw commented Aug 9, 2016

@Mausy5043 are you sure you tried this on an up-to-date Ubuntu 16.10? I just tried this in a Yakkety chroot and pip3 freeze with python3-lxc installed works fine. You do get a warning:

$ pip3 list
-lxc (0.1)
nose2 (0.6.5)
pip (8.1.2)
setuptools (20.10.1)
six (1.10.0)
wheel (0.29.0)

$ pip3 freeze
Could not parse requirement: -lxc
nose2==0.6.5
six==1.10.0

However, I noticed that the patch is not installed on the Ubuntu 16.04 LTS (Xenial) version, so you do get that crash. I've already cherry picked one fix into the Xenial SRU, but I missed this one.

LP: #1611503 tracks the SRU for this problem.

@Mausy5043

This comment has been minimized.

Show comment
Hide comment
@Mausy5043

Mausy5043 Aug 10, 2016

@warsaw : This is what I have:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

pip3 says:

$ pip3 list
-lxc (0.1)
apt-xapian-index (0.47)
chardet (2.3.0)
command-not-found (0.3)
devscripts (2.16.2ubuntu3)
language-selector (0.1)
Magic-file-extensions (0.2)
mysqlclient (1.3.7)
pip (8.1.1)
pycurl (7.43.0)
pygobject (3.20.0)
python-apt (1.1.0b1)
python-debian (0.1.27)
python-systemd (231)
setuptools (20.7.0)
six (1.10.0)
ufw (0.35)
unattended-upgrades (0.1)
wheel (0.29.0)
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ pip3 freeze
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1137, in parseString
    raise exc
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1127, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2412, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2669, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2395, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1005, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1785, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-lxc==0.'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/lib/python3/dist-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 260, in from_dist
    req = dist.as_requirement()
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-lxc==0.'"
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Mausy5043 commented Aug 10, 2016

@warsaw : This is what I have:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

pip3 says:

$ pip3 list
-lxc (0.1)
apt-xapian-index (0.47)
chardet (2.3.0)
command-not-found (0.3)
devscripts (2.16.2ubuntu3)
language-selector (0.1)
Magic-file-extensions (0.2)
mysqlclient (1.3.7)
pip (8.1.1)
pycurl (7.43.0)
pygobject (3.20.0)
python-apt (1.1.0b1)
python-debian (0.1.27)
python-systemd (231)
setuptools (20.7.0)
six (1.10.0)
ufw (0.35)
unattended-upgrades (0.1)
wheel (0.29.0)
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ pip3 freeze
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1137, in parseString
    raise exc
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1127, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2412, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2669, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 2395, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1005, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1785, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-lxc==0.'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/freeze.py", line 85, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/lib/python3/dist-packages/pip/operations/freeze.py", line 47, in freeze
    dependency_links
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 260, in from_dist
    req = dist.as_requirement()
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2562, in as_requirement
    return Requirement.parse(spec)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2833, in parse
    req, = parse_requirements(s)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-lxc==0.'"
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw Aug 10, 2016

Release: 16.04

Now it makes sense! Thanks. A fix for Xenial is in the SRU pipeline. If you have xenial-updates enabled, you should get the fix soon. It has to be approved by the SRU team, but I expect there shouldn't be any hiccups with that.

warsaw commented Aug 10, 2016

Release: 16.04

Now it makes sense! Thanks. A fix for Xenial is in the SRU pipeline. If you have xenial-updates enabled, you should get the fix soon. It has to be approved by the SRU team, but I expect there shouldn't be any hiccups with that.

@Mausy5043

This comment has been minimized.

Show comment
Hide comment
@Mausy5043

Mausy5043 Aug 10, 2016

@warsaw: Thanks. Just to be sure: how do I make sure I have "updates enabled"?

Mausy5043 commented Aug 10, 2016

@warsaw: Thanks. Just to be sure: how do I make sure I have "updates enabled"?

@warsaw

This comment has been minimized.

Show comment
Hide comment
@warsaw

warsaw Aug 10, 2016

@Mausy5043 Take a look at /etc/apt/sources.list. It should look something like this:

deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial main restricted

deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates main restricted

deb http://us.archive.ubuntu.com/ubuntu/ xenial universe
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial universe
deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe

deb http://us.archive.ubuntu.com/ubuntu/ xenial multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates multiverse

deb http://us.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
deb-src http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse
deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse

I'm using the USA mirror, so YMMV. This page might also help.

warsaw commented Aug 10, 2016

@Mausy5043 Take a look at /etc/apt/sources.list. It should look something like this:

deb http://us.archive.ubuntu.com/ubuntu/ xenial main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial main restricted

deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates main restricted

deb http://us.archive.ubuntu.com/ubuntu/ xenial universe
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial universe
deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe

deb http://us.archive.ubuntu.com/ubuntu/ xenial multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-updates multiverse

deb http://us.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
deb-src http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse
deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse

I'm using the USA mirror, so YMMV. This page might also help.

@111pontes

This comment has been minimized.

Show comment
Hide comment
@111pontes

111pontes Aug 17, 2016

Also affects Trusty. Added note to LP: #1611503.

111pontes commented Aug 17, 2016

Also affects Trusty. Added note to LP: #1611503.

@ssbarnea

This comment has been minimized.

Show comment
Hide comment
@ssbarnea

ssbarnea Oct 28, 2016

Contributor

Can we please reopen this bug and have it transformed into a warning ASAP? Using pip became a nightmare due to this and we all see it appearing more and more often.

Contributor

ssbarnea commented Oct 28, 2016

Can we please reopen this bug and have it transformed into a warning ASAP? Using pip became a nightmare due to this and we all see it appearing more and more often.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Oct 28, 2016

Contributor

@ssbarnea it was fixed in the pull request a while back, see the diff of #3704

Contributor

RonnyPfannschmidt commented Oct 28, 2016

@ssbarnea it was fixed in the pull request a while back, see the diff of #3704

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Oct 28, 2016

Contributor

@ssbarnea its not very nice to copy&paste the same comment across a range of issues

Contributor

RonnyPfannschmidt commented Oct 28, 2016

@ssbarnea its not very nice to copy&paste the same comment across a range of issues

@diogobaeder

This comment has been minimized.

Show comment
Hide comment
@diogobaeder

diogobaeder Oct 28, 2016

The issue was fixed more than 5 months ago, however there was no release for it yet. Why not release version 8.1.3 already?

diogobaeder commented Oct 28, 2016

The issue was fixed more than 5 months ago, however there was no release for it yet. Why not release version 8.1.3 already?

@laerus

This comment has been minimized.

Show comment
Hide comment
@laerus

laerus Jun 1, 2017

still happening on debian-stretch

> pip freeze > /dev/null
Could not parse requirement: -lxc

> pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)

> apt list python3-lxc 
Listing... Done
python3-lxc/testing,now 1:2.0.7-2 amd64 [installed,automatic]

laerus commented Jun 1, 2017

still happening on debian-stretch

> pip freeze > /dev/null
Could not parse requirement: -lxc

> pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)

> apt list python3-lxc 
Listing... Done
python3-lxc/testing,now 1:2.0.7-2 amd64 [installed,automatic]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment