New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: maxdelay.bit_length change causes workers to not reconnect #3876

Closed
moshez opened this Issue Jan 5, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@moshez

moshez commented Jan 5, 2018

See
https://github.com/buildbot/buildbot/blame/3fcf45acd84c1e3e6b3a6c77eda5d1e4dd0f9b3e/worker/buildbot_worker/pb.py#L195 -- this changed the line from passing an int to passing a callable, which cases the following exception

2018-01-05T21:18:39+0000 [Uninitialized] Unhandled Error
	Traceback (most recent call last):
	  File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 311, in runReactorWithLogging
	    reactor.run()
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1243, in run
	    self.mainLoop()
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1255, in mainLoop
	    self.doIteration(t)
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/epollreactor.py", line 235, in doPoll
	    log.callWithLogger(selectable, _drdw, selectable, fd, event)
	--- <exception caught here> ---
	  File "/usr/local/lib/python3.6/site-packages/twisted/python/log.py", line 103, in callWithLogger
	    return callWithContext({"system": lp}, func, *args, **kw)
	  File "/usr/local/lib/python3.6/site-packages/twisted/python/log.py", line 86, in callWithContext
	    return context.call({ILogContext: newCtx}, func, *args, **kw)
	  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
	    return func(*args,**kw)
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 627, in _doReadOrWrite
	    self._disconnectSelectable(selectable, why, inRead)
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 258, in _disconnectSelectable
	    selectable.connectionLost(failure.Failure(why))
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 473, in connectionLost
	    self.failIfNotConnected(error.ConnectError(string=reason))
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 450, in failIfNotConnected
	    self.connector.connectionFailed(failure.Failure(err))
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1128, in connectionFailed
	    self.factory.clientConnectionFailed(self, reason)
	  File "/usr/local/lib/python3.6/site-packages/buildbot_worker/util/_hangcheck.py", line 115, in clientConnectionFailed
	    self._wrapped_factory.clientConnectionFailed(connector, reason)
	  File "/usr/local/lib/python3.6/site-packages/buildbot_worker/pb.py", line 120, in clientConnectionFailed
	    connector, reason)
	  File "/usr/local/lib/python3.6/site-packages/buildbot_worker/pbutil.py", line 64, in clientConnectionFailed
	    self.retry()
	  File "/usr/local/lib/python3.6/site-packages/buildbot_worker/pb.py", line 86, in retry
	    ReconnectingPBClientFactory.retry(self, connector=connector)
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 427, in retry
	    self.delay = min(self.delay * self.factor, self.maxDelay)
	builtins.TypeError: '<' not supported between instances of 'builtin_function_or_method' and 'float'

Relevant buildbot tac


import os

from buildbot_worker.bot import Worker
from twisted.application import service

basedir = '/buildbot-worker'

# note: this line is matched against to check that this is a worker
# directory; do not edit it.
application = service.Application('buildbot-worker')
buildmaster_host = 'localhost'
port = 2020
workername = 'worker'
passwd = 'pass'
keepalive = 600
umask = None
maxdelay = 300
numcpus = None
allow_shutdown = None
maxretries = None

s = Worker(buildmaster_host, port, workername, passwd, basedir,
           keepalive, umask=umask, maxdelay=maxdelay,
           numcpus=numcpus, allow_shutdown=allow_shutdown,
           maxRetries=maxretries)
s.setServiceParent(application)
@tardyp

This comment has been minimized.

Show comment
Hide comment
@tardyp

tardyp Jan 5, 2018

Member

Thanks for the report.

I have no idea why this ".bit_length" sneaked in. I fear this is IDE autocomplete :/

Member

tardyp commented Jan 5, 2018

Thanks for the report.

I have no idea why this ".bit_length" sneaked in. I fear this is IDE autocomplete :/

@tardyp

This comment has been minimized.

Show comment
Hide comment
@tardyp

tardyp Jan 5, 2018

Member

I don't understand how the e2e tests didn't catch that.

Member

tardyp commented Jan 5, 2018

I don't understand how the e2e tests didn't catch that.

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

@tardyp tardyp changed the title from Regression: maxdelay.bit_length change causes workers to not connect to Regression: maxdelay.bit_length change causes workers to not reconnect Jan 6, 2018

@tardyp

This comment has been minimized.

Show comment
Hide comment
@tardyp

tardyp Jan 6, 2018

Member

I guess this is because this problem is only when worker is reconnecting, we don't have e2e tests for that :(

Member

tardyp commented Jan 6, 2018

I guess this is because this problem is only when worker is reconnecting, we don't have e2e tests for that :(

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

tardyp added a commit to tardyp/buildbot that referenced this issue Jan 6, 2018

@tardyp tardyp closed this in #3877 Jan 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment