Installation issues on Travis CI

[ghost]

Originally reported by: Anonymous

---

Since the upload of version setuptools version 12.1 yesterday, our builds on Travis start to fail:
https://travis-ci.org/bioidiap/bob.core/jobs/50389380

Using the latest bootstrap-buildout.py, we get the following error message:

$ python bootstrap-buildout.py
<string>:144: DeprecationWarning: `use_setuptools` is deprecated. To enforce a specific version of setuptools, use `pkg_resources.require`.
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-12.1.zip
Extracting in /tmp/tmpRCrQju
Now working in /tmp/tmpRCrQju/setuptools-12.1
Building a Setuptools egg in /tmp/tmp7CIXgd
/tmp/tmpRCrQju/setuptools-12.1/setuptools/command/egg_info.py:171: DeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
writer = ep.load(installer=installer)
/tmp/tmp7CIXgd/setuptools-12.1-py2.6.egg
Traceback (most recent call last):
File "bootstrap-buildout.py", line 98, in <module>
ez['use_setuptools'](**setup_args)
File "<string>", line 174, in use_setuptools
File "<string>", line 129, in _do_download
File "build/bdist.linux-x86_64/egg/setuptools/__init__.py", line 11, in <module>
File "build/bdist.linux-x86_64/egg/setuptools/extension.py", line 8, in <module>
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 19, in <module>
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 83, in <module>
AttributeError: 'module' object has no attribute '_vendor'

Interestingly, lines 81 and 82 of pkg_resources/__init__.py (the imports of the _vendor module) pass, but not the actual use of it.
Sorry for not being able to supply more information...

Could you please double-check, what might be going on?

Thank you
Manuel

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/345

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The changes between 12.0.5 and 12.1 were trivially small. I would be surprised if the 12.1 release was a regression from 12.0.5. Do you have reason to believe that (a) the issue did not exist with setuptools 12.0.5 and (b) the bootstrap-buildout has not changed?

[ghost]

Original comment by Anonymous:

---

I am not sure about 12.0.5, but I am sure that 12.0.4 was working: https://travis-ci.org/bioidiap/bob.core/jobs/48035636

[ghost]

Original comment by Anonymous:

---

And I am sure that I haven't changed the bootstrap-buildout.py file (which is still up to date, I have double-checked that).

On the other hand, I am not 100% sure that Travis hasn't updated their system, though I didn't receive a notification...

[ghost]

Original comment by znerol (Bitbucket: znerol, GitHub: znerol):

---

Related issue on travis issue tracker.

[ghost]

Original comment by Anonymous:

---

Indeed, it seems to be an issue with the pre-installed setuptools, which I can reproduce locally.
When using a virtualenv system with pre-installed setuptools version 11.3.1 (in my case) or 12.0.5 (on travis), I get the error:

#!sh
$ /path/to/virtual/system/python bootstrap-buildout.py 
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-12.1.zip
Extracting in /tmp/tmpuCQhNZ
Now working in /tmp/tmpuCQhNZ/setuptools-12.1
Building a Setuptools egg in /tmp/tmpdBboBt
/tmp/tmpdBboBt/setuptools-12.1-py2.7.egg
Traceback (most recent call last):
  File "bootstrap-buildout.py", line 98, in <module>
    ez['use_setuptools'](**setup_args)
  File "<string>", line 174, in use_setuptools
  File "<string>", line 129, in _do_download
  File "build/bdist.linux-x86_64/egg/setuptools/__init__.py", line 11, in <module>
  File "build/bdist.linux-x86_64/egg/setuptools/extension.py", line 8, in <module>
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 19, in <module>
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 83, in <module>
AttributeError: 'module' object has no attribute '_vendor'

So far, my workaround is to use the --setuptools-version option selecting exactly the version that we have:

#!sh
$ /path/to/virtual/system/python bootstrap-buildout.py --setuptools-version 11.3.1

However, we would need to update the travis configuration files each time that they update their system. Since we have more than 40 packages:
https://github.com/idiap/bob/wiki/Packages
you can imaging that I don't want to do that.

The solution in the travis issue tracker would probably work as well, but as it is written there, this is a work-around and no fix. Again, I don't know how long this solution will work.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The issue reported in the travis bug tracker is slightly different, displaying a VersionConflict prior to the AttributeError:

$ python bootstrap.py
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-12.1.zip
Extracting in /tmp/tmpeqvl0x
Now working in /tmp/tmpeqvl0x/setuptools-12.1
Building a Setuptools egg in /tmp/tmp4uaaq8
/tmp/tmp4uaaq8/setuptools-12.1-py3.3.egg
Traceback (most recent call last):
File "<string>", line 162, in use_setuptools
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/pkg_resources/__init__.py", line 918, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages/pkg_resources/__init__.py", line 810, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (setuptools 12.0.5 (/home/travis/virtualenv/python3.3.5/lib/python3.3/site-packages), Requirement.parse('setuptools>=12.1'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bootstrap.py", line 92, in <module>
ez['use_setuptools'](**setup_args)
File "<string>", line 174, in use_setuptools
File "<string>", line 129, in _do_download
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "/tmp/tmp4uaaq8/setuptools-12.1-py3.3.egg/setuptools/__init__.py", line 11, in <module>
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "/tmp/tmp4uaaq8/setuptools-12.1-py3.3.egg/setuptools/extension.py", line 8, in <module>
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "/tmp/tmp4uaaq8/setuptools-12.1-py3.3.egg/setuptools/dist.py", line 19, in <module>
File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File "/tmp/tmp4uaaq8/setuptools-12.1-py3.3.egg/pkg_resources/__init__.py", line 83, in <module>
AttributeError: 'module' object has no attribute '_vendor'
The command "python bootstrap.py" failed and exited with 1 during .

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Aha. The reason that both tracebacks appear in the above report is because it's on Python 3, so the underlying exception isn't swallowed by the subsequent exception. Good work Python 3!

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I think I now know what's happening. In ez_setup.use_setuptools, which is invoked by buildout's bootstrap, the code will use pkg_resources to determine if the installed setuptools is a suitable version. If it is not, it unloads pkg_resources (using del pkg_resources, sys.modules['pkg_resources']). This technique worked before pkg_resources got subpackages (namely _vendor). Now, when the VersionConflict occurs, pkg_resources is unloaded from sys.modules, but pkg_resources._vendor.* is not, so when pkg_resources is reloaded, and the conditional import takes place, the _vendor modules are already in sys.modules, so they're not added to the newly-imported pkg_resources namespace.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Unload all pkg_resources modules and not just the main module. Fixes #345.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Extract function for unloading pkg_resources. Ref #345.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The fix was released as 12.2. Can you confirm the fix in your test runs?

[ghost]

Original comment by Anonymous:

---

I am currently re-running the tests. I'll let you know.

[ghost]

Original comment by Anonymous:

---

Yes, I can confirm that the tests are running now, both on Travis and using the local virtualenv.

Thank you very much for fixing this issue!
Manuel

[ghost]

Original comment by znerol (Bitbucket: znerol, GitHub: znerol):

---

Thanks, that's also working for me.

[ghost]

Original comment by tseaver (Bitbucket: tseaver, GitHub: tseaver):

---

Clears up buildout/buildout#237, too. Thanks, @jaraco!