Gracefully handle Bugzilla deadlocks

[lmacken]

[2015-09-08 19:29:18][     bodhi   ERROR] Unable to add comment to bug #1260329
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/bodhi/bugs.py", line 77, in comment
bug.addcomment(comment)
File "/usr/lib/python2.7/site-packages/bugzilla/bug.py", line 295, in addcomment
return self.bugzilla.update_bugs(self.bug_id, vals)
File "/usr/lib/python2.7/site-packages/bugzilla/base.py", line 1270, in update_bugs
return self._proxy.Bug.update(tmp)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/site-packages/bugzilla/base.py", line 168, in _ServerProxy__request
ret = ServerProxy._ServerProxy__request(self, methodname, params)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/site-packages/bugzilla/base.py", line 263, in request
return self._request_helper(url, request_body)
File "/usr/lib/python2.7/site-packages/bugzilla/base.py", line 245, in _request_helper
raise sys.exc_info()[1]
Fault: <Fault -32000: 'DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction [for Statement "UPDATE bugs SET delta_ts = ? WHERE bug_id = ?"] at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1305\
n\tBugzilla::Bug::update(\'Bugzilla::Bug=HASH(0x7f90972f8710)\') called at /var/www/html/bugzilla/Bugzilla/WebService/Bug.pm line 768\n\tBugzilla::WebService::Bug::update(\'Bugzilla::WebService::Bug\', \'HASH(0x7f9096fe39f0)\') call
ed at /usr/share/perl5/vendor_perl/SOAP/Lite.pm line 2875\n\teval {...} called at /usr/share/perl5/vendor_perl/SOAP/Lite.pm line 2860\n\teval {...} called at /usr/share/perl5/vendor_perl/SOAP/Lite.pm line 2826\n\tSOAP::Server::handl
e(\'Bugzilla::WebService::Server::XMLRPC=HASH(0x7f9096df9270)\', \'<?xml version=\\\'1.0\\\'?>\\x{a}<methodCall>\\x{a}<methodName>Bug.update</...\') called at /usr/share/perl5/vendor_perl/SOAP/Transport/HTTP.pm line 438\n\tSOAP::Tra
nsport::HTTP::Server::handle(\'Bugzilla::WebService::Server::XMLRPC=HASH(0x7f9096df9270)\') called at /usr/share/perl5/vendor_perl/SOAP/Transport/HTTP.pm line 831\n\tSOAP::Transport::HTTP::Apache::handler(\'Bugzilla::WebService::Ser
ver::XMLRPC=HASH(0x7f9096df9270)\') called at /var/www/html/bugzilla/xmlrpc.cgi line 39\n\tModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_xmlrpc_2ecgi::handler(\'Apache2::RequestRec=SCALAR(0x7f9098333978)\'
) called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204\n\teval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 204\n\tModPerl::RegistryCooker::run(\'Bugzilla::ModPerl::ResponseHandler
=HASH(0x7f9094f6f1d0)\') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 170\n\tModPerl::RegistryCooker::default_handler(\'Bugzilla::ModPerl::ResponseHandler=HASH(0x7f9094f6f1d0)\') called at /usr/lib64/perl5/v
endor_perl/ModPerl/Registry.pm line 31\n\tModPerl::Registry::handler(\'Bugzilla::ModPerl::ResponseHandler\', \'Apache2::RequestRec=SCALAR(0x7f9098333978)\') called at /var/www/html/bugzilla/mo
d_perl.pl line 134\n\tBugzilla::ModPerl::ResponseHandler::handler(\'Bugzilla::ModPerl::ResponseHandler\', \'Apache2::RequestRec=SCALAR(0x7f9098333978)\') called at -e line 0\n\teval {...} called at -e line 0\n'>

[ralphbean]

Is this from the backend, or the frontend?

[lmacken]

I'm seeing it on the backend.

[trishnaguha]

@ralphbean and @lmacken I would like to work on it :) . How can I reproduce the bug?

[lmacken]

@trishnaguha Awesome, thanks for offering to help!

Unfortunately, this bug isn't going to be simple to reproduce, as I think it's a bugzilla server-side race-condition of some sort. Thankfully, it shouldn't be very difficult to fix or at least work around.

So this bug.add_comment call (https://github.com/fedora-infra/bodhi/blob/develop/bodhi/bugs.py#L77) is triggering an xmlrpclib.Fault exception to get thrown (https://docs.python.org/2/library/xmlrpclib.html#fault-objects) when the database is in this deadlocked state. We could potentially catch the fault (maybe look at faultCode too?) and try again (a few times?).