New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(pypy/debian only) setuptools broken during an upgrade #1632

Closed
fungi opened this Issue Mar 10, 2014 · 22 comments

Comments

Projects
None yet
8 participants
@fungi
Contributor

fungi commented Mar 10, 2014

On some systems, and so far I've been unable to determine what makes them "special" in this way, upgrading setuptools with pip at the same time as upgrading particular other packages causes their setup.py invocation to fail with "error: option --single-version-externally-managed not recognized". Upgrading them independently of setuptools (either before or after) does not exhibit this behavior.

I saw it briefly weeks ago, but the release of setuptools 2.2 seemed to cause it to cease. The most recent occurrences seem to possibly coincide with Saturday's release of setuptools 3.1, but that could simply be a red herring.

I have narrowed it down to a simple means of recreating this on the affected systems where we're seeing it currently: within a pypy virtualenv, use pip to upgrade setuptools and mccabe together on the same command line. Examples...

Working consistently on Ubuntu 12.04 VMs in one service provider: http://paste.openstack.org/show/73058/

Failing consistently on seemingly-identical Ubuntu 12.04 VMs in another service provider: http://paste.openstack.org/show/73060/

I discussed this with Alex Gaynor just now, and we agree that this feels like some sort of strangeness in pip which is getting triggered by hash ordering differences in different systems/interpreters rather than being a PyPy-specific bug. On the systems where this fails in a pypy virtualenv, it succeeds normally in a python2.7 or python3.3 virtualenv.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 18, 2014

Member
<fungi>     dstufft: inspecting the virtualenv shows that the new setuptools installed after it throws that error
<fungi>     dstufft: foo/bin/pypy -c 'import setuptools;print(setuptools.__version__)' says 3.3 but list says setuptools (3.1)
<fungi>     dstufft: another point of interest, 'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described
<fungi>     dstufft: also, downgrading pip to 1.3.1 in the virtualenv before installing/upgrading setuptools and other packages seems to cause this problem not to happen
Member

dstufft commented Mar 18, 2014

<fungi>     dstufft: inspecting the virtualenv shows that the new setuptools installed after it throws that error
<fungi>     dstufft: foo/bin/pypy -c 'import setuptools;print(setuptools.__version__)' says 3.3 but list says setuptools (3.1)
<fungi>     dstufft: another point of interest, 'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described
<fungi>     dstufft: also, downgrading pip to 1.3.1 in the virtualenv before installing/upgrading setuptools and other packages seems to cause this problem not to happen
@fungi

This comment has been minimized.

Show comment
Hide comment
@fungi

fungi Mar 18, 2014

Contributor

This started impacting systems on our remaining provider over the weekend, and seems coincidentally timed to the release of setuptools 3.3 (no idea whether that could be related to the new shift in behavior).

Contributor

fungi commented Mar 18, 2014

This started impacting systems on our remaining provider over the weekend, and seems coincidentally timed to the release of setuptools 3.3 (no idea whether that could be related to the new shift in behavior).

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft
Member

dstufft commented Mar 19, 2014

/cc @jaraco

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Mar 19, 2014

Member

It sounds like the issue is related to something about how the setuptools package is updated during the installation process. It sounds as if the issue is not related to a specific setuptools version, but rather the way in which the version changes. I would not be surprised if this issue is related to the way the 'sandbox' is created during builds, though I really don't have any insight to add yet.

I can say that it seems unlikely that the recent changes would have affected the way builds are done with respect to that command line argument.

Member

jaraco commented Mar 19, 2014

It sounds like the issue is related to something about how the setuptools package is updated during the installation process. It sounds as if the issue is not related to a specific setuptools version, but rather the way in which the version changes. I would not be surprised if this issue is related to the way the 'sandbox' is created during builds, though I really don't have any insight to add yet.

I can say that it seems unlikely that the recent changes would have affected the way builds are done with respect to that command line argument.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 19, 2014

Contributor

here's the story:

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use __pycache__, e.g. on debian), that results in an empty __pycache__ folder being written to the dist-info folder. you can see it in a fresh pypy virtualenv for pip and setuptools. i.e. the root of this is not just a setuptools issue. this is true for any wheel install under pypy (that uses __pycache__)

pip uses the dist-info/RECORD file to uninstall things. it doesn't know about this dist-info __pycache__ folder, so it's get left, along with the parent dist-info folder, during an uninstall (or an upgrade, that does an uninstall).

so right now, pip/pypy uninstalls of any wheels, are leaving empty shells of old metadata around, which sets up a failure scenario later due to metadata mismatches.

so back to the setuptools case:

for me, it's not just failing with: `pip install -U setuptools mccabe'

but also much worse, like this:

pip install -U setuptools
pip install mccabe

pip's internal use of setuptools fails with the duplicate metadata. if you remove the old metadata dir, everything is ok.

so, what's the fix? the RECORD file needs to have an entry for the dist-info __pycache__ folder.

and this looks worthy of a patch release.

Contributor

qwcode commented Mar 19, 2014

here's the story:

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use __pycache__, e.g. on debian), that results in an empty __pycache__ folder being written to the dist-info folder. you can see it in a fresh pypy virtualenv for pip and setuptools. i.e. the root of this is not just a setuptools issue. this is true for any wheel install under pypy (that uses __pycache__)

pip uses the dist-info/RECORD file to uninstall things. it doesn't know about this dist-info __pycache__ folder, so it's get left, along with the parent dist-info folder, during an uninstall (or an upgrade, that does an uninstall).

so right now, pip/pypy uninstalls of any wheels, are leaving empty shells of old metadata around, which sets up a failure scenario later due to metadata mismatches.

so back to the setuptools case:

for me, it's not just failing with: `pip install -U setuptools mccabe'

but also much worse, like this:

pip install -U setuptools
pip install mccabe

pip's internal use of setuptools fails with the duplicate metadata. if you remove the old metadata dir, everything is ok.

so, what's the fix? the RECORD file needs to have an entry for the dist-info __pycache__ folder.

and this looks worthy of a patch release.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 19, 2014

Contributor

'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described

I don't see that. breaks for 3.1 too.

Contributor

qwcode commented Mar 19, 2014

'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described

I don't see that. breaks for 3.1 too.

@qwcode qwcode changed the title from setuptools broken when upgraded with other packages to (pypy only) setuptools broken during an upgrade Mar 19, 2014

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Mar 19, 2014

Member

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use pycache, e.g. on debian), that results in an empty pycache folder being written to the dist-info folder

Maybe I'm being dense but isn't this a bug in pypy? Why is it creating a __pycache__ directory in a directory that contains no Python files?

I'm not saying we don't need to work around it, but it's worth a pypy bug report too.

Member

pfmoore commented Mar 19, 2014

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use pycache, e.g. on debian), that results in an empty pycache folder being written to the dist-info folder

Maybe I'm being dense but isn't this a bug in pypy? Why is it creating a __pycache__ directory in a directory that contains no Python files?

I'm not saying we don't need to work around it, but it's worth a pypy bug report too.

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Mar 19, 2014

Member

AFAIK the issue that is *debian pypy is running a Python 2 implementation, yet __pycache__ is really from python 3. Additionally, the API to detect this is in Python 3, but obviously not in Python 2. So pypy is importing a Python 3 feature into Python 2 and without the accompanying API to manage or detect the feature.

I'm guessing they just never guessed that importing __pycache__ into a python 2 implementation would manage to end up screwing up pip while it uninstalls things.

Member

Ivoz commented Mar 19, 2014

AFAIK the issue that is *debian pypy is running a Python 2 implementation, yet __pycache__ is really from python 3. Additionally, the API to detect this is in Python 3, but obviously not in Python 2. So pypy is importing a Python 3 feature into Python 2 and without the accompanying API to manage or detect the feature.

I'm guessing they just never guessed that importing __pycache__ into a python 2 implementation would manage to end up screwing up pip while it uninstalls things.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 19, 2014

Member

I'm not sure the fact that it's a __pycache__ folder really matters, just that there's one being created in the dist-info directory which doesn't happen in CPython.

Member

dstufft commented Mar 19, 2014

I'm not sure the fact that it's a __pycache__ folder really matters, just that there's one being created in the dist-info directory which doesn't happen in CPython.

@arigo

This comment has been minimized.

Show comment
Hide comment
@arigo

arigo Mar 19, 2014

PyPy (version 2.7.x) doesn't contain any line of code with __pycache__ in it. Grepping both the source code and the official binary doesn't find it (just like it doesn't find it on CPython 2.7.x). But Debian patches PyPy and some of their patches does contain __pycache__. Based on this I think it's safe to say that, if anything, it's an issue that should be reported to the Debian maintainers.

arigo commented Mar 19, 2014

PyPy (version 2.7.x) doesn't contain any line of code with __pycache__ in it. Grepping both the source code and the official binary doesn't find it (just like it doesn't find it on CPython 2.7.x). But Debian patches PyPy and some of their patches does contain __pycache__. Based on this I think it's safe to say that, if anything, it's an issue that should be reported to the Debian maintainers.

@arigo

This comment has been minimized.

Show comment
Hide comment
@arigo

arigo Mar 19, 2014

More precisely, grep __pycache__ turns out quite a number of hits in pypy_2.2.1+dfsg-1.debian.tar.gz from the url https://packages.debian.org/source/jessie/pypy for example.

arigo commented Mar 19, 2014

More precisely, grep __pycache__ turns out quite a number of hits in pypy_2.2.1+dfsg-1.debian.tar.gz from the url https://packages.debian.org/source/jessie/pypy for example.

@alex

This comment has been minimized.

Show comment
Hide comment
@alex

alex Mar 19, 2014

Member

Yes, it seems debian (and the PPA) include a set of patches which do cause
pycache to be used.

On Wed, Mar 19, 2014 at 7:40 AM, arigo notifications@github.com wrote:

More precisely, grep pycache turns out quite a number of hits in
pypy_2.2.1+dfsg-1.debian.tar.gz from the url
https://packages.debian.org/source/jessie/pypy for example.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1632#issuecomment-38057950
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

Member

alex commented Mar 19, 2014

Yes, it seems debian (and the PPA) include a set of patches which do cause
pycache to be used.

On Wed, Mar 19, 2014 at 7:40 AM, arigo notifications@github.com wrote:

More precisely, grep pycache turns out quite a number of hits in
pypy_2.2.1+dfsg-1.debian.tar.gz from the url
https://packages.debian.org/source/jessie/pypy for example.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1632#issuecomment-38057950
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 19, 2014

Member

Here's an upstream bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=742132

Probably we should work around this in pip too. It seems like we shouldn't fail because of an empty directory that we installed, even if it was created erroneously.

Member

dstufft commented Mar 19, 2014

Here's an upstream bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=742132

Probably we should work around this in pip too. It seems like we shouldn't fail because of an empty directory that we installed, even if it was created erroneously.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Mar 19, 2014

Member

It's probably reasonable not to worry about changing RECORD, because we don't typically put directory names in there as far as I can see. Instead, let's just delete __pycache__ on the assumption that if the source file is gone, the cache can go too.

How about, if we delete files from a directory, then when we're done if the directory contains nothing but a __pycache__ subdirectory, we delete that? That's a pretty good general principle, __pycache__ directories are intended to be ignorable, after all...

Member

pfmoore commented Mar 19, 2014

It's probably reasonable not to worry about changing RECORD, because we don't typically put directory names in there as far as I can see. Instead, let's just delete __pycache__ on the assumption that if the source file is gone, the cache can go too.

How about, if we delete files from a directory, then when we're done if the directory contains nothing but a __pycache__ subdirectory, we delete that? That's a pretty good general principle, __pycache__ directories are intended to be ignorable, after all...

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 19, 2014

Member

Well the question is, if someone has an empty directory in their dist-info already what will we do? I think ATM we'll install it and then fail in the same way that this fails. IOW this failure isn't specific to __pycache__, it just happened to be exposed by it.

Member

dstufft commented Mar 19, 2014

Well the question is, if someone has an empty directory in their dist-info already what will we do? I think ATM we'll install it and then fail in the same way that this fails. IOW this failure isn't specific to __pycache__, it just happened to be exposed by it.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Mar 19, 2014

Member

We've got to the point where I should look at the code before guessing, but I think that we don't currently record directories in RECORD, only files. That's why I'm a bit reluctant to include __pycache__ as it might expose other issues (trying to calculate the hash of a directory, using unlink rather than rmdir etc). But I take your point.

Member

pfmoore commented Mar 19, 2014

We've got to the point where I should look at the code before guessing, but I think that we don't currently record directories in RECORD, only files. That's why I'm a bit reluctant to include __pycache__ as it might expose other issues (trying to calculate the hash of a directory, using unlink rather than rmdir etc). But I take your point.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 19, 2014

Member

Actually, what's probably a better solution is to make it so we only deal with files. Right now we have this sort of bad state where when installing we'll happily install directories, but uninstall we only uninstall files. My assertion is that if we install something, then we should also uninstall it.

So I would say ideally we should only install files (creating any directories we need to) and we should only uninstall directories (removing any directories that we make empty, or empty sans .pyc and __pycache__ directories.

This should solve the problem that is the cause of this ticket, and make things work in a much saner fashion.

Member

dstufft commented Mar 19, 2014

Actually, what's probably a better solution is to make it so we only deal with files. Right now we have this sort of bad state where when installing we'll happily install directories, but uninstall we only uninstall files. My assertion is that if we install something, then we should also uninstall it.

So I would say ideally we should only install files (creating any directories we need to) and we should only uninstall directories (removing any directories that we make empty, or empty sans .pyc and __pycache__ directories.

This should solve the problem that is the cause of this ticket, and make things work in a much saner fashion.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 19, 2014

Contributor

the API to detect this is in Python 3,
but obviously not in Python 2
So pypy is importing a Python 3 feature into Python 2
and without the accompanying API to manage or detect the feature.

debian hacks the imp lib as well. I discovered this 2 years ago, when I was working another pip uninstall issue related to __pycache__ (#646)

Contributor

qwcode commented Mar 19, 2014

the API to detect this is in Python 3,
but obviously not in Python 2
So pypy is importing a Python 3 feature into Python 2
and without the accompanying API to manage or detect the feature.

debian hacks the imp lib as well. I discovered this 2 years ago, when I was working another pip uninstall issue related to __pycache__ (#646)

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz
Member

Ivoz commented Mar 20, 2014

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 20, 2014

Contributor

for history's sake, here's the pypy thread from 2012, when it was discovered that the debian distro of pypy adds __pycache__.
https://mail.python.org/pipermail/pypy-dev/2012-July/010369.html

the debian maintainer chimes in after a few posts.

Contributor

qwcode commented Mar 20, 2014

for history's sake, here's the pypy thread from 2012, when it was discovered that the debian distro of pypy adds __pycache__.
https://mail.python.org/pipermail/pypy-dev/2012-July/010369.html

the debian maintainer chimes in after a few posts.

@dstufft dstufft added this to the 1.5.5 milestone Mar 28, 2014

@qwcode qwcode changed the title from (pypy only) setuptools broken during an upgrade to (pypy/debian only) setuptools broken during an upgrade Mar 28, 2014

@alex

This comment has been minimized.

Show comment
Hide comment
@alex

alex Apr 25, 2014

Member

Thanks so much for fixing this @qwcode -- can this issue be closed?

Member

alex commented Apr 25, 2014

Thanks so much for fixing this @qwcode -- can this issue be closed?

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Apr 25, 2014

Contributor

yes, closing

Contributor

qwcode commented Apr 25, 2014

yes, closing

@qwcode qwcode closed this Apr 25, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment