Buildslave setup fixes

[garetharmstronghp]

Caught this while playing with rpm and buildbot-slave package. Needs to be applied and released otherwise folks are going to get a big surprise when they run the /usr/bin/buildslave command.

[bbslave@pear SIMULAP-]$ buildslave create-slave --umask=022 $(pwd) my.host.org:$port ${host_name}-${type} $USER
Traceback (most recent call last):
  File "/usr/bin/buildslave", line 5, in ?
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.4/site-packages/pkg_resources.py", line 2562, in ?
    working_set.require(__requires__)
  File "/usr/lib/python2.4/site-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.4/site-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: twisted>=2.0.0

[djmitche]

Actually, I think that the master/setup.py is wrong - entry_points is a setup argument, not just some random module-global variable.

Is twisted really not in your python environment?

[djmitche]

Assigning to a local variable name entry_points has no effect. Additionally, setting a console_scripts which overlaps with the scripts argument to setup.py also has no effect. So I removed the entry_points stuff entirely in b2db59f. Does that do the trick for you?

[garetharmstronghp]

Hi Dustin,

sorry for not getting back earlier. Your fix does do the trick, so many thanks. So, sorry I just pushed a quick and dirty fix earlier without doing my homework. I saw it worked on master but not on the slave. In any case, once I finish polishing the new buildbot.spec file that I have been working on, I will push it to the mailing list for discussion.

Aside from that, I think you might need to push out a new buildbot-slave tarball because the buildslave script is broken. I know you are very busy, is this part of the vision or do folks need to hang tight for a while till 0.8.3?

Thanks,

G

[djmitche]

It's OK - I'm famous for not doing my homework (a trait I'm trying to eliminate).

I'm not sure it is broken - I think you just ran into some problems with setuptools. The workaround (using scripts="bin/buildslave" instead of entry_points) just happens to require the constituent packages differently. Let's see if others are seeing the same thing - if it's widespread and completely broken, then I'll push 0.8.2p1 or even 0.8.3. Sound good? Can you poll the list for others suffering from the same thing?

[garetharmstronghp]

Ok, many thanks. I will sound out the list tomorrow morning. I had the problem on Rhel5.5 with setuptools 0.6c9 and with the same on Fedora 12. I will try with distribute 0.6.14 in the morning and see if that changes things. So thanks again for the wisdom and all your work.

[garetharmstronghp]

Hello again,

Sorry for the long sequence of commands below but just trying to confirm by test. Thanks again,

G

I confirm with distribute 0.6.14 installed, I get this

[gareth@localhost] ~/WORK/BuildBot_deps/buildbot-slave-0.8.2
$ python setup.py install --root=toto

$ less toto/usr/bin/buildslave
#!/usr/bin/python

EASY-INSTALL-ENTRY-SCRIPT: 'buildbot-slave==0.8.2','console_scripts','buildslave'

requires = 'buildbot-slave==0.8.2'
import sys
from pkg_resources import load_entry_point

if name == 'main':
sys.exit(
load_entry_point('buildbot-slave==0.8.2', 'console_scripts', 'buildslave')()
)

Even in a virtualenv with distribute, I can confirm the same result.

However, if I get even more brutal and rip out distribute and friends from the virtualenv, things work as expected with 0.8.2

[gareth@localhost] ~/py_virtualenv/slave_test
$ rm -rf lib/python2.6/site-packages/distribute-0.6.14-py2.6.egg

[gareth@localhost] ~/py_virtualenv/slave_test
$ rm -f lib/python2.6/site-packages/setuptools.pth

[gareth@localhost] ~/py_virtualenv/slave_test
$ source bin/activate

(slave_test)[00:35 0.03]
[gareth@localhost] ~/py_virtualenv/slave_test
$ python
Python 2.6.2 (r262:71600, Jun 4 2010, 18:28:58)
[GCC 4.4.3 20100127 (Red Hat 4.4.3-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import setuptools
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named setuptools

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test
$ ls
bin/ buildbot-slave-0.8.2/ include/ lib/ lib64@

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test
$ cd buildbot-slave-0.8.2/

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test/buildbot-slave-0.8.2
$ python setup.py install --root tata
.

[gareth@localhost] ~/py_virtualenv/slave_test/buildbot-slave-0.8.2
$ less tata/home/gareth/py_virtualenv/slave_test/bin/buildslave
#!/home/gareth/py_virtualenv/slave_test/bin/python

from buildslave.scripts import runner
runner.run()

If I now replace buildslave/setup.py with your updates version in a new virtual env with setuptools/distribute. I get the expected buildslave script file. I will still poll the list later this morning but on Fedora/Rhel python-setuptools is nearly always present so I would say go for an update, if feasible.

[djmitche]

By the way, if you indent the command lines four spaces, they look much better. I tried to edit the above, but it didn't work, so here's a formatted version:

Hello again,

Sorry for the long sequence of commands below but just trying to confirm by test. Thanks again,

G

I confirm with distribute 0.6.14 installed, I get this

[gareth@localhost] ~/WORK/BuildBot_deps/buildbot-slave-0.8.2
$ python setup.py install --root=toto

$ less toto/usr/bin/buildslave
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'buildbot-slave==0.8.2','console_scripts','buildslave'
__requires__ = 'buildbot-slave==0.8.2'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('buildbot-slave==0.8.2', 'console_scripts', 'buildslave')()
    )

Even in a virtualenv with distribute, I can confirm the same result.

However, if I get even more brutal and rip out distribute and friends from the virtualenv, things work as expected with 0.8.2

[gareth@localhost] ~/py_virtualenv/slave_test
$ rm -rf lib/python2.6/site-packages/distribute-0.6.14-py2.6.egg

[gareth@localhost] ~/py_virtualenv/slave_test
$ rm -f lib/python2.6/site-packages/setuptools.pth

[gareth@localhost] ~/py_virtualenv/slave_test
$ source bin/activate

(slave_test)[00:35 0.03]
[gareth@localhost] ~/py_virtualenv/slave_test
$ python
Python 2.6.2 (r262:71600, Jun  4 2010, 18:28:58)
[GCC 4.4.3 20100127 (Red Hat 4.4.3-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named setuptools

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test
$ ls
bin/  buildbot-slave-0.8.2/  include/  lib/  lib64@

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test
$ cd buildbot-slave-0.8.2/

(slave_test)[00:39 0.00]
[gareth@localhost] ~/py_virtualenv/slave_test/buildbot-slave-0.8.2
$ python setup.py install --root tata
.
[gareth@localhost] ~/py_virtualenv/slave_test/buildbot-slave-0.8.2
$ less tata/home/gareth/py_virtualenv/slave_test/bin/buildslave
#!/home/gareth/py_virtualenv/slave_test/bin/python
from buildslave.scripts import runner
runner.run()

If I now replace buildslave/setup.py with your updates version in a new virtual env with setuptools/distribute. I get the expected buildslave script file. I will still poll the list later this morning but on Fedora/Rhel python-setuptools is nearly always present so I would say go for an update, if feasible.

[djmitche]

From what I can tell from the above, you're seeing the script with
# EASY-INSTALL-ENTRY-SCRIPT: 'buildbot-slave==0.8.2','console_scripts','buildslave'
and assuming it's broken. I'm not convinced, and that's what I'd like to hear from others about.

In theory, anyway, that script should work just fine. It just takes care to pull Buildbot's dependencies into the "working set" first, munging sys.modules as necessary. That munging doesn't work in all environments. In particular, I don't think it will work in an install root (--root=toto).

You're right that if setuptools is not installed, then the entry_points will not be set, and buildbot-slave will install the shorter bin/buildslave you quote at the end -- a script which seems to work under more circumstances.

So, in lieu of releasing 0.8.2p1, I'd prefer to patch this locally in the RPM. I assume that the exception in the first comment was from an RPM-installed buildbot-slave package?

I know this is frustrating - I find the lack of a completely functional packaging system (despite having a half-dozen of them!) to be Python's greatest weakness. I don't know why the entry_points script wouldn't work..

[djmitche]

We determined in the mailing list that the problem is due to Buildbot expecting a Python package named 'Twisted', while RPM-based systems install Python packages named e.g., Twisted-Words, Twisted-Web, etc.

On master, we've fixed this to use regular old Python scripts, instead of easy install entry points, which basically removes any dependency-checking from the script. That seems to be a functional solution, since it's been in place for master for a while now.

I've only seen one problem with this, and a potentially related problem on Windows, so it's not quite reached my must-make-a-new-release threshold. However, 0.8.3 is coming early in December, so there's not long to wait.

Thanks for hashing this out!