setuptools 8.4 breaks non-pip upgrades

[ghost]

Originally reported by: tseaver (Bitbucket: tseaver, GitHub: tseaver)

---

Both zc.buildout and easy_install fail to update to setuptools 8.4 with tracebacks like so:

$ sudo /opt/Python-3.3.3/bin/easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Best match: setuptools 8.4
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.4.zip#md5=41154296b306110e56ebe8dbd35a0d31
Processing setuptools-8.4.zip
Writing /tmp/easy_install-wth6t5/setuptools-8.4/setup.cfg
Running setuptools-8.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-wth6t5/setuptools-8.4/egg-dist-tmp-nl1f7k
Traceback (most recent call last):
File "/opt/Python-3.3.3/lib/python3.3/site-packages/setuptools-8.3-py3.3.egg/pkg_resources/__init__.py", line 2262, in load
AttributeError: 'module' object has no attribute 'write_setup_requirements'

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/314

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@richardipsum can you look into the cause of this and devise a workaround? I suspect the issue has to do with metadata from 8.4 applying to an installation from 8.3 or earlier which doesn't implement the writer.

[ghost]

Original comment by tseaver (Bitbucket: tseaver, GitHub: tseaver):

---

Reproducing:

$ /opt/Python-2.7.9/bin/virtualenv --no-setuptools --no-pip /tmp/xxx
New python executable in xxx/bin/python
$ cd /tmp/xxx
$ wget -o /dev/null https://bootstrap.pypa.io/ez_setup.py
$ bin/python ez_setup.py --version=8.3
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.3.zip
...
Installed /tmp/xxx/lib/python2.7/site-packages/setuptools-8.3-py2.7.egg
Processing dependencies for setuptools==8.3
Finished processing dependencies for setuptools==8.3
$ bin/easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Best match: setuptools 8.4
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-8.4.zip#md5=41154296b306110e56ebe8dbd35a0d31
Processing setuptools-8.4.zip
Writing /tmp/easy_install-XanO1G/setuptools-8.4/setup.cfg
Running setuptools-8.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XanO1G/setuptools-8.4/egg-dist-tmp-KMkFaC
Traceback (most recent call last):
File "bin/easy_install", line 9, in <module>
    load_entry_point('setuptools==8.3', 'console_scripts', 'easy_install')()
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 2152, in main
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 2138, in with_ei_usage
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 2155, in <lambda>
File "/opt/Python-2.7.9/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
File "/opt/Python-2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
File "/opt/Python-2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 375, in run
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 618, in easy_install
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 648, in install_item
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 838, in install_eggs
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1060, in build_and_install
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1045, in run_setup
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 68, in run_setup
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 114, in run
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 67, in runner
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 43, in _execfile
File "/tmp/easy_install-XanO1G/setuptools-8.4/setup.py", line 220, in <module>
File "/opt/Python-2.7.9/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
File "/opt/Python-2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
File "/opt/Python-2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 152, in run
File "/opt/Python-2.7.9/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
File "/opt/Python-2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 167, in run
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2264, in load
ImportError: <module 'setuptools.command.egg_info' from '/tmp/xxx/lib/python2.7/site-packages/setuptools-8.3-py2.7.egg/setuptools/command/egg_info.pyc'> has no 'write_setup_requirements' attribute

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've pulled Setuptools 8.4 from PyPI and pinned the bootstrap script to 8.3 for now while the issue is investigated. I've put a copy of 8.4 in the Downloads section, so the script to reproduce should now use

bin/easy_install https://bitbucket.org/pypa/setuptools/downloads/setuptools-8.4.zip

in place of

bin/easy_install -U setuptools

[ghost]

Original comment by jamilappa (Bitbucket: jamilappa, GitHub: Unknown):

---

The bootstrap script at https://bootstrap.pypa.io/ez_setup.py still refers to 8.4 can you fix this. Thanks.

[ghost]

Original comment by richardipsum (Bitbucket: richardipsum, GitHub: richardipsum):

---

Sorry about this :/
It does look as though easy_install is trying to use the new entry point, which isn't there. I'll try and find a work around.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The bootstrap script at https://bootstrap.pypa.io/ez_setup.py still refers to 8.4

That's probably my oversight. I moved the bootstrap bookmark on the repo and pushed it to github, but I didn't push it to bitbucket, and I'm guessing that's the repo the bootstrap site uses. It should be corrected in the next 12 minutes.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@richardipsum, Don't sweat it. These problems crop up, and there doesn't seem to be anything inherent about your work that was implicated here.

I had some more time to think about this issue this morning, and I think the issue is this - when easy_install is invoked, it imports the installed setuptools (<8.4). That code locates, downloads, and extracts the sdist, and then using the 'sandbox' logic, namely at easy_install.py:1045, the setup.py is invoked. But because setuptools is already in sys.modules, the sandbox invocation works gets the old module and not the one just downloaded, but it still gets the most recent metadata (egg_info).

I imagined there would be a straightforward workaround, but as I think about it more, I don't think there is. Best I can imagine, any technique I've yet conceived would have to have already been present in the old code. I suspect we will have to back out this change, make new releases to support this type of update, allow them to trickle through the community, and only then release the updated behavior.

I wonder if it's possible to reload the setuptools package (and all of its modules) from sys.modules in the sandbox runner. I don't know what happens when modules that are running code are removed from sys.modules. I'm also tempted to abandon the sandbox runner for a subprocess runner, which seems to be the most popular technique for invoking isolated behavior. I don't know what implications that would have.

I invite @pje to comment on the issue, as he may have encountered this in past releases or have other experiences to relate.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@dstufft The bootstrap bookmark is pointing to the 8.3 release in both github and bitbucket. Can you check the bootstrap server to find out why it's not getting that bootstrap script?

[ghost]

Original comment by alex_gaynor (Bitbucket: alex_gaynor, GitHub: Unknown):

---

@jaraco: @gutworth just did something to fix this

[ghost]

Original comment by gutworth (Bitbucket: gutworth, GitHub: gutworth):

---

For some reason, setuptools was checked out to a different revision on the bootstrap server. I recreated the checkout at the correct revision.

[ghost]

Original comment by jbbarth_ (Bitbucket: jbbarth_, GitHub: Unknown):

---

I've pulled Setuptools 8.4 from PyPI and pinned the bootstrap script to 8.3 for now while the issue is investigated.

Just a quick note about this, please be aware that some (private) pypi mirrors don't keep up with that kind of strategies. Actually mine at work is broken because it still has 8.4 in the index, and it tries to download 8.4 from pypi which leads to a 404. And I'm having a hard time understanding how it works.

Don't you think it would be a better strategy to release a 8.4.1 or 8.5 which would be the same as 8.3 ? That's a naive question honestly, I'm a python newbie and I'd understand if you have valid reasons to pull the release like you did.

Good luck for the bugs ! :-)

[ghost]

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

Generally I find the best solution to be to re-release a known good with a higher version number, and then pull the bad version.

[ghost]

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

Oh, and since the bootstrap server is on PSF infra now, there are a number of folks who can poke it in #python-infra on Freenode if i'm not around.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@jbbarth_, Thanks for the feedback. You're right that a new release is warranted. I just haven't had the time to enact one, so to unblock the issue as quickly as possible, I pulled the release.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Remove setup_requirements writer from metadata. Fixes #314.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've filed #315 for tracking a solution to the underlying problem.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Released as 9.0.