Error with `import pip` in pip 9.0.2

[Miserlou]

Some users are now experiencing an import error when calling import pip in a function with pip 9.0.2. Downgrading to 9.0.1 fixes the issue.

Trace: https://user-images.githubusercontent.com/2273226/37558391-5e41fc94-2a24-11e8-9fdc-5884445e829a.png

More details here:
Miserlou/Zappa#1446

[gs-jackal]

I can confirm this as well. Just about to file same issue.

[gs-jackal]

I can confirm this as well. Just about to file same issue.

[alex]

This is a dupe of #5079 -- importing pip is not a supported use case (and never has been).

[alex]

This is a dupe of #5079 -- importing pip is not a supported use case (and never has been).

[Miserlou]

Come on.. making a huge breaking change in a single x.x.PATCH version bump? For a top level method that's called "main"? I think that's pretty poor form..

[Miserlou]

Come on.. making a huge breaking change in a single x.x.PATCH version bump? For a top level method that's called "main"? I think that's pretty poor form..

[FlipperPA]

This also broke Anaconda and I came up with the same solution as @Miserlou:

ContinuumIO/anaconda-issues#8911

Errors are being reported across a lot of projects.

[FlipperPA]

This also broke Anaconda and I came up with the same solution as @Miserlou:

ContinuumIO/anaconda-issues#8911

Errors are being reported across a lot of projects.

[RayJ414]

This error also bothered me when I was trying to use pip to update my anaconda-navigator.

[RayJ414]

This error also bothered me when I was trying to use pip to update my anaconda-navigator.

[Miserlou]

Is there any chance of us getting a 9.0.3 that fixes this - ideally quite soon?

This is already affecting lots of major projects (Anaconda, SpaCy, Zappa), and there are more than 850k+ uses of this on GitHub alone that are now broken by this supposed "patch" version update.

Can you at least revert this massive change in 9.0.3 and then announce to downstreams your intention to change this behavior for a future 10.x.x or at least 9.x.x release?

[Miserlou]

Is there any chance of us getting a 9.0.3 that fixes this - ideally quite soon?

This is already affecting lots of major projects (Anaconda, SpaCy, Zappa), and there are more than 850k+ uses of this on GitHub alone that are now broken by this supposed "patch" version update.

Can you at least revert this massive change in 9.0.3 and then announce to downstreams your intention to change this behavior for a future 10.x.x or at least 9.x.x release?

[RemiCardona]

We don't want to use an older version either, but that's exactly what we ended up doing. 9.0.2 does not exist inside our CI environment, at least for now.

[RemiCardona]

We don't want to use an older version either, but that's exactly what we ended up doing. 9.0.2 does not exist inside our CI environment, at least for now.

[stmcginnis]

Also seeing this hit in some OpenStack projects.

[stmcginnis]

Also seeing this hit in some OpenStack projects.

[pfmoore]

Pip 10 is due out in about a month. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage.

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

Proposals for reintroducing some level of API support are certainly an option (see #5080 for example) but at this stage, it's too late for any such changes to make it into pip 10.

[pfmoore]

Pip 10 is due out in about a month. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage.

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

Proposals for reintroducing some level of API support are certainly an option (see #5080 for example) but at this stage, it's too late for any such changes to make it into pip 10.

[stmcginnis]

With no warnings being raised by the code for those using the old way, nothing denoting this as "internal" with starting with _, and this just being a bugfix bump, I actually think it's quite easy to see this as a sudden, unexpected breakage.

[stmcginnis]

With no warnings being raised by the code for those using the old way, nothing denoting this as "internal" with starting with _, and this just being a bugfix bump, I actually think it's quite easy to see this as a sudden, unexpected breakage.

[Miserlou]

Yes, this is the type of change that people could expect from a MAJOR version update. That would be fine.

Unfortunately, this massive change came in a PATCH update, which is supposed to fix things, not break them. This is the exact opposite of semantic versioning. And, as a result, we are seeing massive damage across the Python ecosystem.

You should revert this change ASAP in another PATCH update, then have the major breaking change come with the MAJOR version update. Now that you've already broken everything for everybody prematurely, I think they will be more aware that the actual major change is coming.

I also think it's a cop out to say that this was documented and announced already, considering that it was not documented in the stable documentation, and that the announcement said that the break would happen for the major version, not for in a patch a month prior.

Please, just save everybody a massive headache, revert this problem and start actually adhering to semantic versioning.

[Miserlou]

Yes, this is the type of change that people could expect from a MAJOR version update. That would be fine.

Unfortunately, this massive change came in a PATCH update, which is supposed to fix things, not break them. This is the exact opposite of semantic versioning. And, as a result, we are seeing massive damage across the Python ecosystem.

You should revert this change ASAP in another PATCH update, then have the major breaking change come with the MAJOR version update. Now that you've already broken everything for everybody prematurely, I think they will be more aware that the actual major change is coming.

I also think it's a cop out to say that this was documented and announced already, considering that it was not documented in the stable documentation, and that the announcement said that the break would happen for the major version, not for in a patch a month prior.

Please, just save everybody a massive headache, revert this problem and start actually adhering to semantic versioning.

[pfmoore]

@Miserlou OK, I take your point - we targeted the internal renaming change for pip 10 because it's a major version. I don't really know the drivers for the patch release - @dstufft pinged me about it and it's apparently to avoid significant breakage for Mac OS users when an imminent deadline for TLS support hits us, which is before the pip 10 release. We expected the problems to be significant enough to warrant a short-term patch release. There was certainly no intention of breaking existing usage.

I have to leave the decision of a follow-up 9.0.3 to @dstufft - I don't have the details of what went into 9.0.2 or know whether it's even possible to identify how to fix this issue. And I can't judge whether pulling 9.0.2 altogether will be an overall benefit, as I have no idea how many people are affected by the Mac OS issue. I understand that (at least for some people) pinning to 9.0.1 is a solution, so that may end up being the least bad option.

[pfmoore]

@Miserlou OK, I take your point - we targeted the internal renaming change for pip 10 because it's a major version. I don't really know the drivers for the patch release - @dstufft pinged me about it and it's apparently to avoid significant breakage for Mac OS users when an imminent deadline for TLS support hits us, which is before the pip 10 release. We expected the problems to be significant enough to warrant a short-term patch release. There was certainly no intention of breaking existing usage.

I have to leave the decision of a follow-up 9.0.3 to @dstufft - I don't have the details of what went into 9.0.2 or know whether it's even possible to identify how to fix this issue. And I can't judge whether pulling 9.0.2 altogether will be an overall benefit, as I have no idea how many people are affected by the Mac OS issue. I understand that (at least for some people) pinning to 9.0.1 is a solution, so that may end up being the least bad option.

[alex]

The macOS TLS issue will affect everyone using a system Python on macOS<10.13

[alex]

The macOS TLS issue will affect everyone using a system Python on macOS<10.13

[pradyunsg]

I have to leave the decision of a follow-up 9.0.3 to @dstufft

I'm of the same position as @pfmoore.

[pradyunsg]

I have to leave the decision of a follow-up 9.0.3 to @dstufft

I'm of the same position as @pfmoore.

[ViktorHaag]

Workaround, if you have it available to you, is to check the order of imports and test moving importing pip above other package imports (specifically, importing pip before importing requests seems to solve some cases). (Note that this still implies use of pip's internals, which isn't officially supported.)

[ViktorHaag]

Workaround, if you have it available to you, is to check the order of imports and test moving importing pip above other package imports (specifically, importing pip before importing requests seems to solve some cases). (Note that this still implies use of pip's internals, which isn't officially supported.)

[anx-ckreuzberger]

Same issue here with pip 9.0.2 in a gitlab-ci with docker python 3.4: KeyError

  File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'

[anx-ckreuzberger]

Same issue here with pip 9.0.2 in a gitlab-ci with docker python 3.4: KeyError

  File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'

[anx-ckreuzberger]

FYI, 9.0.2 was released with broken builts:

Travis-CI Reference: https://travis-ci.org/pypa/pip/builds/354616774?utm_source=github_status&utm_medium=notification

Granted the errors might be unrelated, though this just smells like a "rushed release"...

[anx-ckreuzberger]

FYI, 9.0.2 was released with broken builts:

Travis-CI Reference: https://travis-ci.org/pypa/pip/builds/354616774?utm_source=github_status&utm_medium=notification

Granted the errors might be unrelated, though this just smells like a "rushed release"...

[anx-ckreuzberger]

@pfmoore

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

[anx-ckreuzberger]

@pfmoore

If @dstufft wants to make an emergency 9.0.3 release, I'm fine with that. But it will only be a very short term benefit, and I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

[RonnyPfannschmidt]

pip always was documented as NOT having a consumable python api, i'm disappointed in people that even at that point try to turn around the "told you so since quite some years" karma

[RonnyPfannschmidt]

pip always was documented as NOT having a consumable python api, i'm disappointed in people that even at that point try to turn around the "told you so since quite some years" karma

[anx-ckreuzberger]

@RonnyPfannschmidt This is like saying "We've told you a 100 times you are not allowed to drive over the speed limit" and then enforcing the speed limit by replacing all engine-cars with flintstone-cars, so people can not go over the speed limit anymore.

[anx-ckreuzberger]

@RonnyPfannschmidt This is like saying "We've told you a 100 times you are not allowed to drive over the speed limit" and then enforcing the speed limit by replacing all engine-cars with flintstone-cars, so people can not go over the speed limit anymore.

[RonnyPfannschmidt]

you just demonstrated perfectly that turning around of words that so bad - the telling was "dont rely on it it will break" - now it broke and suddenly the project that told you before that it will break is at fault

that just putting blame because you dont like to be at fault

[RonnyPfannschmidt]

you just demonstrated perfectly that turning around of words that so bad - the telling was "dont rely on it it will break" - now it broke and suddenly the project that told you before that it will break is at fault

that just putting blame because you dont like to be at fault

[anx-ckreuzberger]

@RonnyPfannschmidt

that just putting blame because you dont like to be at fault

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip. Maybe, I am... I should have screened those 3rd-party packages for those errors and submitted an issue or even better a PR.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

[anx-ckreuzberger]

@RonnyPfannschmidt

that just putting blame because you dont like to be at fault

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip. Maybe, I am... I should have screened those 3rd-party packages for those errors and submitted an issue or even better a PR.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

[ubernostrum]

If a third-party package you use breaks because it relies on undocumented, un-guaranteed internal APIs, it seems to me that you should file a bug against that package.

There's an easy-to-discover line between what parts of pip are and aren't intended for third-party consumption, and I don't support trying to force its developers to maintain internal APIs that never should have been consumed by third-party packages. I don't support treating this as the fault of the pip developers and sending angry users their way. If you want to go be angry at someone, go be angry at any packages you use which relied on internal APIs, and push for their maintainers to fix their own code.

[ubernostrum]

If a third-party package you use breaks because it relies on undocumented, un-guaranteed internal APIs, it seems to me that you should file a bug against that package.

There's an easy-to-discover line between what parts of pip are and aren't intended for third-party consumption, and I don't support trying to force its developers to maintain internal APIs that never should have been consumed by third-party packages. I don't support treating this as the fault of the pip developers and sending angry users their way. If you want to go be angry at someone, go be angry at any packages you use which relied on internal APIs, and push for their maintainers to fix their own code.

[pfmoore]

@anx-ckreuzberger

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

While I understand the frustration here, I'm getting increasingly annoyed by the willingness of people to blame the pip maintainers for things we never promised.

If you want to make accusations like that, please back them up with

1. A pointer to a statement from any of the pip maintainers that we support use of the pip internal API in user code.
2. A pointer to a statement from any of the pip maintainers that pip uses semantic versioning.

I don't think you'll find that evidence...

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip.

No, but accusing the pip maintainers rather than those projects is not acceptable. We're trying to help you but can't be held responsible for what those projects do. Air your grievances with them.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

We tried to provide a warning. We sent announcements out months ago, we added docs explaining the situation, and we've consistently replied to issues on the tracker stating that use of internal APIs is unsupported. Adding deprecations is far from being as easy as you suggest, as pip itself would spew those warnings (or there'd be a way for pip to switch them off, which others would just copy - we're already hearing of people planning to just import pip._internal, so even that change won't stop them 😞).

As to "fixing" this, I'll happily admit that 9.0.2 was released quickly to address a pressing issue, and we didn't anticipate this problem. Maybe releasing a 9.0.3 with a lifespan of 2-3 weeks is a reasonable thing to do, I don't think so myself, but I've stated that we'll consider that. I can't personally agree to do it, as I'm not the one involved in the 9.0.x releases. I'm working on pip 10, which will make all of this debate irrelevant, so this is likely my last word on this issue - I need to focus on things related to that release.

My personal advice:

1. Pin to 9.0.1 if you're affected by this and need a workaround now.
2. Be prepared for pip 10, when all the dependencies you currently have that are broken because they use pip internal APIs will break again. Push those dependencies to take action on the information we've been giving for months, and be glad you got advance warning that it would happen.
3. If pip 9.0.3 gets released, remove the pin.
4. Please, test the pip 10 beta when it comes out, and report any issues to the relevant parties (3rd party projects if they are calling pip internally, us if you are calling pip yourself).

[pfmoore]

@anx-ckreuzberger

I can not believe that this is a statement by the owner of this repository... You literally just said "f*ck semantic versioning" and "who needs a deprecation policy anyway".

While I understand the frustration here, I'm getting increasingly annoyed by the willingness of people to blame the pip maintainers for things we never promised.

If you want to make accusations like that, please back them up with

1. A pointer to a statement from any of the pip maintainers that we support use of the pip internal API in user code.
2. A pointer to a statement from any of the pip maintainers that pip uses semantic versioning.

I don't think you'll find that evidence...

So you are saying that I'm at fault for using 3rd-party packages that rely on an undocumented feature of pip.

No, but accusing the pip maintainers rather than those projects is not acceptable. We're trying to help you but can't be held responsible for what those projects do. Air your grievances with them.

But let's face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It's a matter of "can you please fix this and provide a proper warning/deprecation".

We tried to provide a warning. We sent announcements out months ago, we added docs explaining the situation, and we've consistently replied to issues on the tracker stating that use of internal APIs is unsupported. Adding deprecations is far from being as easy as you suggest, as pip itself would spew those warnings (or there'd be a way for pip to switch them off, which others would just copy - we're already hearing of people planning to just import pip._internal, so even that change won't stop them 😞).

As to "fixing" this, I'll happily admit that 9.0.2 was released quickly to address a pressing issue, and we didn't anticipate this problem. Maybe releasing a 9.0.3 with a lifespan of 2-3 weeks is a reasonable thing to do, I don't think so myself, but I've stated that we'll consider that. I can't personally agree to do it, as I'm not the one involved in the 9.0.x releases. I'm working on pip 10, which will make all of this debate irrelevant, so this is likely my last word on this issue - I need to focus on things related to that release.

My personal advice:

1. Pin to 9.0.1 if you're affected by this and need a workaround now.
2. Be prepared for pip 10, when all the dependencies you currently have that are broken because they use pip internal APIs will break again. Push those dependencies to take action on the information we've been giving for months, and be glad you got advance warning that it would happen.
3. If pip 9.0.3 gets released, remove the pin.
4. Please, test the pip 10 beta when it comes out, and report any issues to the relevant parties (3rd party projects if they are calling pip internally, us if you are calling pip yourself).

[thet]

I've experienced the same problem with pip 9.0.2 and Python 2.7.14 within a docker container.
I cannot reproduce the problem on my dev machine outside a docker container though.
I searched for a pip import (grep'd for import.pip, from.pip, \'pip, \"pip) but couldn't find anything.
We at Plone are using a mechanism to automatically include configurations from packages included in a setuptools setup.py file, which sounds suspicious - but again - this one is just using __import__ and nothign from pip AFAIcansee.

This the relevant part of the traceback:

  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/zcml.py", line 51, in includeDependenciesDirective
    info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml'])
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/dependency.py", line 26, in includableInfo
    module = resolve(dotted_name)
  File "/home/plone/.buildout/shared-eggs/zope.dottedname-4.2-py2.7.egg/zope/dottedname/resolve.py", line 39, in resolve
    found = __import__(used)
  File "/plone/buildout/lib/python2.7/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/plone/buildout/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/plone/buildout/lib/python2.7/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/site.zcml", line 15.2-15.55
    ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/package-includes/002-bda.aaf.site-configure.zcml", line 1.0-1.56
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure.zcml", line 11.2-11.48
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure-dependencies.zcml", line 10.2-10.39
    ZopeXMLConfigurationError: File "/plone/buildout/src-addons/plone.app.mosaic/src/plone/app/mosaic/configure.zcml", line 9.2-9.39
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.tiles-3.0.3-py2.7.egg/plone/app/tiles/configure.zcml", line 18.2-18.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.z3cform/plone/app/z3cform/configure.zcml", line 10.2-10.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.widgets/plone/app/widgets/configure.zcml", line 12.2-12.41
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/Products.CMFPlone-5.1.0.1-py2.7.egg/Products/CMFPlone/configure.zcml", line 15.2-15.46
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.contenttypes-1.4.9-py2.7.egg/plone/app/contenttypes/configure.zcml", line 10.2-10.37
    KeyError: 'pip._vendor.urllib3.contrib'

[thet]

I've experienced the same problem with pip 9.0.2 and Python 2.7.14 within a docker container.
I cannot reproduce the problem on my dev machine outside a docker container though.
I searched for a pip import (grep'd for import.pip, from.pip, \'pip, \"pip) but couldn't find anything.
We at Plone are using a mechanism to automatically include configurations from packages included in a setuptools setup.py file, which sounds suspicious - but again - this one is just using __import__ and nothign from pip AFAIcansee.

This the relevant part of the traceback:

  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/plone/.buildout/shared-eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/zcml.py", line 51, in includeDependenciesDirective
    info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml'])
  File "/home/plone/.buildout/shared-eggs/z3c.autoinclude-0.3.7-py2.7.egg/z3c/autoinclude/dependency.py", line 26, in includableInfo
    module = resolve(dotted_name)
  File "/home/plone/.buildout/shared-eggs/zope.dottedname-4.2-py2.7.egg/zope/dottedname/resolve.py", line 39, in resolve
    found = __import__(used)
  File "/plone/buildout/lib/python2.7/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/plone/buildout/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/plone/buildout/lib/python2.7/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/plone/buildout/lib/python2.7/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/site.zcml", line 15.2-15.55
    ZopeXMLConfigurationError: File "/plone/buildout/parts/instance/etc/package-includes/002-bda.aaf.site-configure.zcml", line 1.0-1.56
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure.zcml", line 11.2-11.48
    ZopeXMLConfigurationError: File "/plone/buildout/src-aaf/bda.aaf.site/src/bda/aaf/site/configure-dependencies.zcml", line 10.2-10.39
    ZopeXMLConfigurationError: File "/plone/buildout/src-addons/plone.app.mosaic/src/plone/app/mosaic/configure.zcml", line 9.2-9.39
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.tiles-3.0.3-py2.7.egg/plone/app/tiles/configure.zcml", line 18.2-18.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.z3cform/plone/app/z3cform/configure.zcml", line 10.2-10.41
    ZopeXMLConfigurationError: File "/plone/buildout/src/plone.app.widgets/plone/app/widgets/configure.zcml", line 12.2-12.41
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/Products.CMFPlone-5.1.0.1-py2.7.egg/Products/CMFPlone/configure.zcml", line 15.2-15.46
    ZopeXMLConfigurationError: File "/home/plone/.buildout/shared-eggs/plone.app.contenttypes-1.4.9-py2.7.egg/plone/app/contenttypes/configure.zcml", line 10.2-10.37
    KeyError: 'pip._vendor.urllib3.contrib'

[jensens]

As far as I understand just importing pip - without using it - breaks already. This is not a good citizens behavior in Python land. Not supporting using it/ not providing a public API is one thing, just breaking on import another. This affects a lot of auto-inspecting code.

[jensens]

As far as I understand just importing pip - without using it - breaks already. This is not a good citizens behavior in Python land. Not supporting using it/ not providing a public API is one thing, just breaking on import another. This affects a lot of auto-inspecting code.

[jensens]

In order to make this do-not-import-pip mantra more visible, what about renaming the namespace to "_pip". This indicates that the whole is not for public use on a name level.
Also, it would be easier to tell auto-inspecting code not to look at stuff starting with underscore. Well, latter would need changes in the involvement inspection code too. But at least there's a chance to automate by it (by convention) without managing blacklists.

[jensens]

In order to make this do-not-import-pip mantra more visible, what about renaming the namespace to "_pip". This indicates that the whole is not for public use on a name level.
Also, it would be easier to tell auto-inspecting code not to look at stuff starting with underscore. Well, latter would need changes in the involvement inspection code too. But at least there's a chance to automate by it (by convention) without managing blacklists.

[dstufft]

Ok, last thing I swear-- pip 10 already moved all relevant code to pip._internal (we didn't use _pip because that would break python -m pip, which is supported).

[dstufft]

Ok, last thing I swear-- pip 10 already moved all relevant code to pip._internal (we didn't use _pip because that would break python -m pip, which is supported).

[dstufft]

Can anyone verify if #5100 solves this for them?

[dstufft]

Can anyone verify if #5100 solves this for them?

[dstufft]

Scratch that, I think #5100 is wrong, could you verify #5101 instead please.

[dstufft]

Scratch that, I think #5100 is wrong, could you verify #5101 instead please.

[thet]

@dstufft I can confirm, the fix in #5101 solves the problem for me.

[thet]

@dstufft I can confirm, the fix in #5101 solves the problem for me.

[msarahan]

Thanks @dstufft for your time in addressing this. I will work with the Anaconda teams to communicate this issue and help them move away from importing pip.

[msarahan]

Thanks @dstufft for your time in addressing this. I will work with the Anaconda teams to communicate this issue and help them move away from importing pip.

[vdog]

For the record in this thread, #5101 solved my issue as well.

[vdog]

For the record in this thread, #5101 solved my issue as well.

[RemiCardona]

Ditto, #5101 allows the import to work inside our tool. Caveat emptor: I haven't tested the patched pip nor our tool extensively.

[RemiCardona]

Ditto, #5101 allows the import to work inside our tool. Caveat emptor: I haven't tested the patched pip nor our tool extensively.

[dstufft]

This should be fixed in 9.0.3.

[dstufft]

This should be fixed in 9.0.3.

[diginc]

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

[diginc]

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

[Miserlou]

Hooray! Thanks for this!

I'd also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

[Miserlou]

Hooray! Thanks for this!

I'd also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

[dstufft]

pip list --format=json?

[dstufft]

pip list --format=json?

[drunkwcodes]

Firstly thank you all contributing to pip people for it totally covers all of my use cases with a few user-friendly options and arguments.

Due to this "surprisingly widely adopted and useful undefined behavior," do we need to make a piplib as libgit2 for git? Please note that libgit2 is not sharing any code with git, and is maintained by other group from git's. And it is good for git ecosystem. Maybe piplib will cover some interesting use cases which we didn't expect.

Here is a similar story: https://public-inbox.org/git/1267957655.3759.29.camel@mattotaupa/T/

[drunkwcodes]

Firstly thank you all contributing to pip people for it totally covers all of my use cases with a few user-friendly options and arguments.

Due to this "surprisingly widely adopted and useful undefined behavior," do we need to make a piplib as libgit2 for git? Please note that libgit2 is not sharing any code with git, and is maintained by other group from git's. And it is good for git ecosystem. Maybe piplib will cover some interesting use cases which we didn't expect.

Here is a similar story: https://public-inbox.org/git/1267957655.3759.29.camel@mattotaupa/T/

[sruggier]

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there's no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it's probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don't go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

[sruggier]

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there's no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it's probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don't go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

[drunkwcodes]

@sruggier The packages metioned are all good, and WheelFile.install() also needs more promotion. But the one-stop service pip.main() provided is irreplaceable with all of the above. It's still worth trying.

The most important is that I hope these needs are held by another project, and pip10 will arrive smoothly without extra guarantees. The deprecation and minimizing the code base are the whole points for a major release.

Concrete implementation details for a permanent infrastructure "software" is ridiculous. You can't judge maintainers by the documented abuse case and hold back the wheel of pip.

If you insist to use pip as a lib, a lot of assumptions will need to be reconsidered.

[drunkwcodes]

@sruggier The packages metioned are all good, and WheelFile.install() also needs more promotion. But the one-stop service pip.main() provided is irreplaceable with all of the above. It's still worth trying.

The most important is that I hope these needs are held by another project, and pip10 will arrive smoothly without extra guarantees. The deprecation and minimizing the code base are the whole points for a major release.

Concrete implementation details for a permanent infrastructure "software" is ridiculous. You can't judge maintainers by the documented abuse case and hold back the wheel of pip.

If you insist to use pip as a lib, a lot of assumptions will need to be reconsidered.

[pfmoore]

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn't promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it's not hard to implement in a 3rd party module.

[pfmoore]

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn't promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it's not hard to implement in a 3rd party module.

[honnibal]

I appreciate the work you all do on pip, and know it's not easy. But for the record:

I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10

spaCy has moved away from import pip. But some people are still using spaCy 1, which did import pip --- and for obvious reasons, didn't pin pip down to a patch release. If the change had come in at v10, our old versions wouldn't have been affected. We've just issued a hotfix to address this.

[honnibal]

I appreciate the work you all do on pip, and know it's not easy. But for the record:

I'm a little disappointed that people haven't already moved away from import pip. The people hit by this issue will still need to prepare for pip 10

spaCy has moved away from import pip. But some people are still using spaCy 1, which did import pip --- and for obvious reasons, didn't pin pip down to a patch release. If the change had come in at v10, our old versions wouldn't have been affected. We've just issued a hotfix to address this.

[kalefranz]

proper instruction on how to programmatically inspect python environments in a pip10 world

What is PyPA's position on distlib? Pip is obviously using it in some capacity, but so too is it using packaging, which distlib purports to deprecate.

If there isn't an official position, then at least current thoughts and opinions from pip's core maintainers would be much appreciated. If there are relevant discussions on this topic elsewhere, a simple pointer to other discussions is good enough for me.

[kalefranz]

proper instruction on how to programmatically inspect python environments in a pip10 world

What is PyPA's position on distlib? Pip is obviously using it in some capacity, but so too is it using packaging, which distlib purports to deprecate.

If there isn't an official position, then at least current thoughts and opinions from pip's core maintainers would be much appreciated. If there are relevant discussions on this topic elsewhere, a simple pointer to other discussions is good enough for me.

[pfmoore]

I'm not aware that distlib deprecates packaging. It mentions "distutils2/packaging" but distutils2 was something different.

My personal view is that both distlib and packaging are perfectly reasonable things to use. You should evaluate them yourself to confirm they meet your needs, obviously, just like any other dependency you rely on.

[pfmoore]

I'm not aware that distlib deprecates packaging. It mentions "distutils2/packaging" but distutils2 was something different.

My personal view is that both distlib and packaging are perfectly reasonable things to use. You should evaluate them yourself to confirm they meet your needs, obviously, just like any other dependency you rely on.

[kalefranz]

Maybe deprecates is too strong of word then. The source of my current understanding:

https://distlib.readthedocs.io/en/latest/overview.html#distlib-evolved-out-of-packaging

[kalefranz]

Maybe deprecates is too strong of word then. The source of my current understanding:

https://distlib.readthedocs.io/en/latest/overview.html#distlib-evolved-out-of-packaging

[pfmoore]

Ah, that's a different "packaging" - it was the proposed stdlib module that was intended to replace distutils. It's completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

[pfmoore]

Ah, that's a different "packaging" - it was the proposed stdlib module that was intended to replace distutils. It's completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

[tuukkamustonen]

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that :) See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status

[tuukkamustonen]

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that :) See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status