opscode & openstack distribute upgrade issues

[qwcode]

<dstufft> http://tickets.opscode.com/browse/COOK-3284 /cc qwcode
<qwcode> I'll look into it, but "`pip install --upgrade supervisor`) causes distribute 0.7.3 to be installed" should also thereby install setuptools-0.8 making pip operable
<dstufft> qwcode: openstack had an issue too
<qwcode> I'll try to recreate
<dstufft> In openstack case they had virtualenv --distribute
<dstufft> and tried to install d2to1 inside of it
<dstufft> (along with a bunch of other stuff)
<dstufft> and distribute patched things so setuptool s== distribute previously
<dstufft> lemme get a console for it sec
<dstufft> http://logs.openstack.org/35845/1/gate/gate-requirements-install/211/console.html
<dstufft> https://review.openstack.org/#/c/35899/ was their proposed fix It hink

[coderanger]

Just checking in as the Opscode guy that tracks Python stuffs, also @cwjohnston as our initial reporter.

[qwcode]

ok, looking at this now

[dhellmann]

The other option we have may be to stop using distribute when setting up the virtualenv: https://review.openstack.org/35905

I think for that to work we also need to have everyone update to the latest tox.

[qwcode]

to approximate the opscode scenario, I did the following w/o a problem, so there must be more to the story? what should I do different?

• create a --distribute virtualenv using 1.9.1 (which uses pip-1.3.1/distribute-0.6.34)
• pip install distribute==0.6.49 (this will only work in py2 btw)
• pip install supervisor
• pip install -U supervisor

the result (with some edits for brevity) is what I would expect and want.

drwxr-xr-x  2 qwcode qwcode 4.0K 2013-07-05 16:29 distribute-0.7.3-py2.6.egg-info
drwxr-xr-x  4 qwcode qwcode 4.0K 2013-07-05 16:28 pip-1.3.1-py2.6.egg
drwxr-xr-x  3 qwcode qwcode 4.0K 2013-07-05 16:29 setuptools-0.8-py2.6.egg-info
drwxr-xr-x  2 qwcode qwcode 4.0K 2013-07-05 16:28 supervisor-3.0b2-py2.6.egg-info

the purpose of distribute-0.7.3 is just to depend on setuptools, and get it installed.

[qwcode]

@dhellmann, I looked at the console output
http://logs.openstack.org/35845/1/gate/gate-requirements-install/211/console.html

I think I see what's happening, and this may be the opscode issue as well if I was recreating their situation better.

pip builds an install list that ends up including distribute-0.7.3 and setuptools-0.8 (like we want), but they are separated by other dependencies in the list, and in between there is no usable setuptools installed.

When I considered upgrades with Jason (setuptools maintainer), I was worried mostly about the case of a user using an older pip to intentionally upgrade to setuptools/distribute in isolation. That does work.

This is a case where a package upgrade is forcing the distribute upgrade (as part of a large dependency set) prior to the user intentionally doing it first in isolation.

so what's the answer?

• for older pips:
  • to prevent in new environments, you can perform the upgrade first in isolation (i.e. run pip install -U setuptools separately), or you can use a setuptools-based venv in py2.
  • if you've already broken an environment, it will require manual installs to fix.
• for pip-1.4, there are already tweaks in place that would prevent this I think (I can confirm and post later), but pip-1.4 users should already be setup to use setuptools-0.8 and upgrades won't involve this awkward double-project transition and won't fail.

I'll look at this again this weekend and confirm my thinking, and make doc updates.

cc @jaraco

[qwcode]

for pip-1.4, there are already tweaks in place that would prevent this I think (I can confirm and post later)

confirmed

[qwcode]

added a note to the changelog about this for now.
0e342bc

[santagada]

Maybe this explanation should be part of the pip documentation, and of other documentations that use pip like the hitchhikers guide.

[qwcode]

it's a note in the pip-1.4 changelog ( 0e342bc), but I agree a section in our cookbook that explains the whole confusing deal would be nice. I'll put something together.

the hitchhikers guide has been "replaced" by the "Python Packaging User Guide", but that's still in the works
https://python-packaging-user-guide.readthedocs.org/en/latest/

[qwcode]

opened #1064 with a better title and description for this problem to make it for easiser for users to find the solution.
posted a note on distutils-sig and python-virtualenv lists.

[kennethreitz]

To be clear, v1.4 fixes this issue, correct?

[qwcode]

yes, the unreleased pip-1.4 fixes this. (release tentatively set for monday)

[kennethreitz]

Fantastic, I'll start testing it out. If all goes well, I might be able to roll it out on Monday.