opscode & openstack distribute upgrade issues #1033

Closed
qwcode opened this Issue Jul 5, 2013 · 13 comments

Comments

Projects
None yet
5 participants
Contributor

qwcode commented Jul 5, 2013

<dstufft> http://tickets.opscode.com/browse/COOK-3284 /cc qwcode
<qwcode> I'll look into it, but "`pip install --upgrade supervisor`) causes distribute 0.7.3 to be installed" should also thereby install setuptools-0.8 making pip operable
<dstufft> qwcode: openstack had an issue too
<qwcode> I'll try to recreate
<dstufft> In openstack case they had virtualenv --distribute
<dstufft> and tried to install d2to1 inside of it
<dstufft> (along with a bunch of other stuff)
<dstufft> and distribute patched things so setuptool s== distribute previously
<dstufft> lemme get a console for it sec
<dstufft> http://logs.openstack.org/35845/1/gate/gate-requirements-install/211/console.html
<dstufft> https://review.openstack.org/#/c/35899/ was their proposed fix It hink

Just checking in as the Opscode guy that tracks Python stuffs, also @cwjohnston as our initial reporter.

Contributor

qwcode commented Jul 5, 2013

ok, looking at this now

The other option we have may be to stop using distribute when setting up the virtualenv: https://review.openstack.org/35905

I think for that to work we also need to have everyone update to the latest tox.

Contributor

qwcode commented Jul 5, 2013

to approximate the opscode scenario, I did the following w/o a problem, so there must be more to the story? what should I do different?

  • create a --distribute virtualenv using 1.9.1 (which uses pip-1.3.1/distribute-0.6.34)
  • pip install distribute==0.6.49 (this will only work in py2 btw)
  • pip install supervisor
  • pip install -U supervisor

the result (with some edits for brevity) is what I would expect and want.

drwxr-xr-x  2 qwcode qwcode 4.0K 2013-07-05 16:29 distribute-0.7.3-py2.6.egg-info
drwxr-xr-x  4 qwcode qwcode 4.0K 2013-07-05 16:28 pip-1.3.1-py2.6.egg
drwxr-xr-x  3 qwcode qwcode 4.0K 2013-07-05 16:29 setuptools-0.8-py2.6.egg-info
drwxr-xr-x  2 qwcode qwcode 4.0K 2013-07-05 16:28 supervisor-3.0b2-py2.6.egg-info

the purpose of distribute-0.7.3 is just to depend on setuptools, and get it installed.

Contributor

qwcode commented Jul 6, 2013

@dhellmann, I looked at the console output
http://logs.openstack.org/35845/1/gate/gate-requirements-install/211/console.html

I think I see what's happening, and this may be the opscode issue as well if I was recreating their situation better.

pip builds an install list that ends up including distribute-0.7.3 and setuptools-0.8 (like we want), but they are separated by other dependencies in the list, and in between there is no usable setuptools installed.

When I considered upgrades with Jason (setuptools maintainer), I was worried mostly about the case of a user using an older pip to intentionally upgrade to setuptools/distribute in isolation. That does work.

This is a case where a package upgrade is forcing the distribute upgrade (as part of a large dependency set) prior to the user intentionally doing it first in isolation.

so what's the answer?

  • for older pips:
    • to prevent in new environments, you can perform the upgrade first in isolation (i.e. run pip install -U setuptools separately), or you can use a setuptools-based venv in py2.
    • if you've already broken an environment, it will require manual installs to fix.
  • for pip-1.4, there are already tweaks in place that would prevent this I think (I can confirm and post later), but pip-1.4 users should already be setup to use setuptools-0.8 and upgrades won't involve this awkward double-project transition and won't fail.

I'll look at this again this weekend and confirm my thinking, and make doc updates.

cc @jaraco

Contributor

qwcode commented Jul 6, 2013

for pip-1.4, there are already tweaks in place that would prevent this I think (I can confirm and post later)

confirmed

Contributor

qwcode commented Jul 6, 2013

added a note to the changelog about this for now.
0e342bc

@qwcode qwcode closed this Jul 13, 2013

Maybe this explanation should be part of the pip documentation, and of other documentations that use pip like the hitchhikers guide.

Contributor

qwcode commented Jul 18, 2013

it's a note in the pip-1.4 changelog ( 0e342bc), but I agree a section in our cookbook that explains the whole confusing deal would be nice. I'll put something together.

the hitchhikers guide has been "replaced" by the "Python Packaging User Guide", but that's still in the works
https://python-packaging-user-guide.readthedocs.org/en/latest/

Contributor

qwcode commented Jul 19, 2013

opened #1064 with a better title and description for this problem to make it for easiser for users to find the solution.
posted a note on distutils-sig and python-virtualenv lists.

Contributor

kennethreitz commented Jul 19, 2013

To be clear, v1.4 fixes this issue, correct?

Contributor

qwcode commented Jul 19, 2013

yes, the unreleased pip-1.4 fixes this. (release tentatively set for monday)

Contributor

kennethreitz commented Jul 19, 2013

Fantastic, I'll start testing it out. If all goes well, I might be able to roll it out on Monday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment