Buildbot reliability

[skrah]

BPO	11962
Nosy	@warsaw, @pitrou, @vstinner, @ned-deily, @skrah
Files	freebsd-amd64-log.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-05-07.09:06:55.947>
created_at = <Date 2011-04-30.06:43:10.842>
labels = ['type-bug', 'tests']
title = 'Buildbot reliability'
updated_at = <Date 2011-05-07.09:06:55.945>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2011-05-07.09:06:55.945>
actor = 'skrah'
assignee = 'none'
closed = True
closed_date = <Date 2011-05-07.09:06:55.947>
closer = 'skrah'
components = ['Tests']
creation = <Date 2011-04-30.06:43:10.842>
creator = 'skrah'
dependencies = []
files = ['21860']
hgrepos = []
issue_num = 11962
keywords = ['buildbot']
message_count = 10.0
messages = ['134839', '134890', '134901', '134922', '134997', '135084', '135085', '135174', '135175', '135421']
nosy_count = 5.0
nosy_names = ['barry', 'pitrou', 'vstinner', 'ned.deily', 'skrah']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11962'
versions = []

[skrah]

The FreeBSD-AMD64 bot exhibits sporadic hanging in unspecific places.
FreeBSD is running under kvm in the background. When the hanging occurs,
the virtual machine uses 100% CPU and I can't log in via ssh, so I have
to kill the kvm process.

The fact that the ssh login fails if a user process is misbehaving
seems like a FreeBSD/kvm issue to me. However, this problem did not
occur when I set up the bot a couple of weeks ago.

I've started a series of older revision builds to see if anything
recent causes this.

[vstinner]

The FreeBSD-AMD64 bot exhibits sporadic hanging in unspecific places.

You can try a shorter regrtest timeout, edit Lib/test/regrtest.py near:

    if hasattr(faulthandler, 'dump_tracebacks_later'):
        timeout = 60*60

(or use --timeout option of the regrtest.py program)

If you have an access to a terminal (using ssh), you can also set a signal to dump the traceback: edit regrtest.py to add "import signal; faulthandler.register(signal.SIGUSR1, all_threads=True)" after "faulthandler.enable()". Then use "kill -USR1 pid" to dump the traceback.

Or the problem is an unlimited loop while dumping the traceback because of a timeout :-D In this case, disable the timeout using --timeout=0 option of regrtest.py.

[skrah]

Thanks Victor, I can try some of that.

Could this also be a problem with the buildbot software or a networking
problem? The Ubuntu PPC bot might have the same issue. Here the tests
appear to be finished but the clean doesn't start:

http://www.python.org/dev/buildbot/all/builders/PPC%20Ubuntu%203.1/builds/387/steps/test/logs/stdio
http://www.python.org/dev/buildbot/all/builders/PPC%20Ubuntu%203.1/builds/387

[ned-deily]

That might be another instance of this:

http://thread.gmane.org/gmane.comp.python.devel/123698

You might want to bring this up on python-dev.

[skrah]

Going through the logs, this indeed looks like a buildbot software
issue to me. I attach the logs that correspond to this incident:

http://www.python.org/dev/buildbot/all/builders/AMD64%20FreeBSD%208.2%203.2/builds/85

After ...

2011-04-30 01:10:56+0200 [Broker,client] closing stdin
2011-04-30 01:10:56+0200 [Broker,client] using PTY: False

... normally you should see:

... [-] command finished with signal None, exit code 0, elapsedTime:

But there is nothing until I restarted the bot.

[skrah]

Another instance:

2011-05-03 20:18:08+0200 [Broker,client] closing stdin
2011-05-03 20:18:08+0200 [Broker,client] using PTY: False
2011-05-03 20:20:38+0200 [-] sending app-level keepalive

Again this is missing:

... [-] command finished with signal None, exit code 0, elapsedTime:

Also, as we speak the Ubuntu PPC bot is hanging as well:

http://www.python.org/dev/buildbot/all/builders/PPC%20Ubuntu%202.7/builds/386/steps/test/logs/stdio

Antoine, do you have access to the server logs for the relevant
times? My bot is on CEST.

[warsaw]

My Ubuntu PPC server is having hardware problems. It will just intermittently shut off. I've reset the SMU and the PRAM, vacuumed out the guts, reseated the RAM, pulled any possibly problematic 3rd party boards, and it still crashes. I was watching the syslog and it didn't look like a thermal shutdown, though it acted like that. The only thing I can think of is a power supply problem, so I'm going to see if I can find an inexpensive replacement. In the meantime, this machine will be offline for a couple of weeks at least.

[skrah]

The FreeBSD bot had these error messages in the log files:

1. kernel: swap_pager: indefinite wait buffer: device
2. Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.p
   v_entry_max sysctl.

I set up the bot from scratch with these changes:

a) Use swap partition (2GB) instead of swap file (2 GB).

b) Use these sysctls:

     kern.ipc.shm_use_phys=1
     vm.pmap.shpgperproc=4096
     vm.pmap.pv_entry_max=16777216

c) Use self-compiled Python2.7 instead of the system Python2.6.

Let's see how that works out. Error 1) is bad, perhaps FreeBSD
does not play well with the qcow2 file system under high load.

[skrah]

On second thought, I don't want to debug possible qcow2 issues, so
I made another change:

d) Use raw format for the image.

[skrah]

I think the FreeBSD bot changes are working out fine. The Ubuntu-PPC
issues were unrelated, so I'm closing this.