Exceptions writing http to the client #4045

alexlarsson · 2018-04-12T07:46:02Z

I regularly get an exception in my logs where buildbot fails to write to the client. Here is an example:

2018-04-07 09:19:22+0000 [-] while handling API request
        Traceback (most recent call last):
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
            self.gen.throw(type, value, traceback)
        --- <exception caught here> ---
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 141, in handleErrors
            yield
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/web/server.py", line 234, in write
            http.Request.write(self, data)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/web/http.py", line 1069, in write
            self.channel.writeHeaders(version, code, reason, headers)
        exceptions.AttributeError: 'NoneType' object has no attribute 'writeHeaders'
2018-04-07 09:19:22+0000 [-] While rendering resource:
        Traceback (most recent call last):
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
            deferred.errback()
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/resource.py", line 91, in failHttpError
            f.trap(Error)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
            self.raiseException()
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
            f.trap(Redirect)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/failure.py", line 359, in trap
            self.raiseException()
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 528, in asyncRender
            res = yield self.renderRest(request)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
            self.gen.throw(type, value, traceback)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 172, in handleErrors
            jsonrpccode=JSONRPC_CODES["internal_error"])
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/www/rest.py", line 375, in writeError
            request.write(data)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/web/server.py", line 234, in write
            http.Request.write(self, data)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/web/http.py", line 1086, in write
            self.channel.write(data)
        exceptions.AttributeError: 'NoneType' object has no attribute 'write'

In case you're interested, the config is at https://github.com/flathub/buildbot-config/blob/master/master.cfg but its kinda complicated.

The text was updated successfully, but these errors were encountered:

tardyp · 2018-04-12T08:05:59Z

which version of twisted is that?
Could be a problem in twisted web actually

alexlarsson · 2018-04-12T08:09:47Z

Twisted is version 17.9.0

alexlarsson · 2018-04-12T08:17:30Z

Sometimes I also see this:

2018-04-07 09:19:39+0000 [-] Unhandled error in Deferred:
2018-04-07 09:19:39+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/process/log.py", line 140, in finish
            yield super(PlainLog, self).finish()
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
            return _inlineCallbacks(None, gen, Deferred())
        --- <exception caught here> ---
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/srv/buildbot/sandbox/lib/python2.7/site-packages/buildbot/process/log.py", line 101, in finish
            assert not self.finished
        exceptions.AssertionError:

Dunno if it is related.

marrak · 2018-06-14T10:33:35Z

I also regularly see similar exception in my logs:

2018-06-14 12:05:48+0200 [-] while handling API request
        Traceback (most recent call last):
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
            self.gen.throw(type, value, traceback)
        --- <exception caught here> ---
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 141, in handleErrors
            yield
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/web/server.py", line 236, in write
            http.Request.write(self, data)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/web/http.py", line 1095, in write
            self.channel.writeHeaders(version, code, reason, headers)
        builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

2018-06-14 12:05:48+0200 [-] While rendering resource:
        Traceback (most recent call last):
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
            deferred.errback()
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/resource.py", line 91, in failHttpError
            f.trap(Error)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/python/failure.py", line 371, in trap
            self.raiseException()
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/python/failure.py", line 399, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
            f.trap(Redirect)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/python/failure.py", line 371, in trap
            self.raiseException()
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/python/failure.py", line 399, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/python/failure.py", line 422, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 528, in asyncRender
            res = yield self.renderRest(request)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 453, in renderRest
            request.write(data)
          File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
            self.gen.throw(type, value, traceback)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 172, in handleErrors
            jsonrpccode=JSONRPC_CODES["internal_error"])
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/buildbot/www/rest.py", line 375, in writeError
            request.write(data)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/web/server.py", line 236, in write
            http.Request.write(self, data)
          File "/home/buildmaster/buildbot-work/env/lib/python3.6/site-packages/twisted/web/http.py", line 1110, in write
            self.channel.writeSequence(toChunk(data))
        builtins.AttributeError: 'NoneType' object has no attribute 'writeSequence'

Buildbot 1.2.0, Twisted 18.4.0

tardyp · 2018-06-23T07:11:37Z

Is this exception happening when doing reconfig or restart?

brian-peloton · 2018-06-24T18:25:47Z

I see it when just leaving the master running, and not doing anything I think of as special.

I'm using DockerLatentWorker(build_wait_timeout=0) so there are workers attaching and detaching constantly. I also have builds that print a lot of output which I watch using the HTTP interface, and sometimes close browser tabs before the build being watched has finished. Some/all of these may be correlated with the exceptions, but it's hard to tell because I only notice the exceptions later when I don't have context for exactly what was happening then.

I'm using Buildbot 1.2.0 and Twisted 18.4.0 too.

alexlarsson · 2018-06-25T07:53:08Z

Yeah, this happens regularly during normal usage.

devrajatverma · 2018-07-06T05:40:09Z

Same Here

2018-07-05 18:44:56+0530 [-] while handling API request
	Traceback (most recent call last):
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/twisted/internet/defer.py", line 1442, in gotResult
	    _inlineCallbacks(r, g, deferred)
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
	    result = g.send(result)
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/buildbot/www/rest.py", line 453, in renderRest
	    request.write(data)
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/contextlib.py", line 99, in __exit__
	    self.gen.throw(type, value, traceback)
	--- <exception caught here> ---
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/buildbot/www/rest.py", line 141, in handleErrors
	    yield
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/buildbot/www/rest.py", line 453, in renderRest
	    request.write(data)
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/twisted/web/server.py", line 236, in write
	    http.Request.write(self, data)
	  File "/home/rajat/anaconda3/envs/buildbot/lib/python3.6/site-packages/twisted/web/http.py", line 1095, in write
	    self.channel.writeHeaders(version, code, reason, headers)
	builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

ghost · 2018-07-13T08:34:26Z

Same here, using those versions:

Python version: 3.5.2
Buildbot version: 0.9.15
Twisted version: 17.9.0

The following traces are often visible in the log (when serving web pages I believe (not running builds)):

2018-07-13 10:31:11+0200 [-] while handling API request
	Traceback (most recent call last):
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1442, in gotResult
	    _inlineCallbacks(r, g, deferred)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
	    result = g.send(result)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 453, in renderRest
	    request.write(data)
	  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
	    self.gen.throw(type, value, traceback)
	--- <exception caught here> ---
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 141, in handleErrors
	    yield
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 453, in renderRest
	    request.write(data)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/web/server.py", line 234, in write
	    http.Request.write(self, data)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/web/http.py", line 1069, in write
	    self.channel.writeHeaders(version, code, reason, headers)
	builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

and this one (always I believe) follows:

2018-07-13 10:31:11+0200 [-] While rendering resource:
	Traceback (most recent call last):
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1442, in gotResult
	    _inlineCallbacks(r, g, deferred)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
	    deferred.errback()
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 500, in errback
	    self._startRunCallbacks(fail)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/resource.py", line 91, in failHttpError
	    f.trap(Error)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/python/failure.py", line 359, in trap
	    self.raiseException()
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/python/failure.py", line 385, in raiseException
	    raise self.value.with_traceback(self.tb)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/resource.py", line 84, in failHttpRedirect
	    f.trap(Redirect)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/python/failure.py", line 359, in trap
	    self.raiseException()
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/python/failure.py", line 385, in raiseException
	    raise self.value.with_traceback(self.tb)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 528, in asyncRender
	    res = yield self.renderRest(request)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
	    result = g.send(result)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 453, in renderRest
	    request.write(data)
	  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
	    self.gen.throw(type, value, traceback)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 172, in handleErrors
	    jsonrpccode=JSONRPC_CODES["internal_error"])
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/buildbot/www/rest.py", line 375, in writeError
	    request.write(data)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/web/server.py", line 234, in write
	    http.Request.write(self, data)
	  File "/home/churlin/proventools/buildbot/sandbox/lib/python3.5/site-packages/twisted/web/http.py", line 1084, in write
	    self.channel.writeSequence(toChunk(data))
	builtins.AttributeError: 'NoneType' object has no attribute 'writeSequence'

brian-peloton · 2018-07-13T17:26:47Z

I'm still testing my buildbot config, so there aren't many web browsers interacting with it. Because of that, I've noticed a definite correlation between when I close a buildbot tab or navigate to another buildbot page and these showing up. That means it might be relevant that I'm using Google Chrome 67.0.3396.79 (Official Build) (64-bit) on Debian Jessie.

meffie · 2018-08-07T21:15:29Z

I see the same backtraces.

buildbot 1.3.0
buildbot-console-view 1.3.0
buildbot-grid-view 1.3.0
buildbot-waterfall-view 1.3.0
buildbot-worker 1.3.0
buildbot-www 1.3.0
Twisted 18.7.0
Python 3.5.3

glyph · 2018-09-24T22:23:11Z

This is a bug in Twisted, reported here: https://twistedmatrix.com/trac/ticket/9410

marcin-jedynski · 2018-12-06T17:47:28Z

Any update on that ? Same the same happens here,

yhager · 2019-06-07T19:17:19Z

The twisted bug has been fixed. I've upgraded to buildbot 2.3.1 and twisted 19.2.1, and I'm still seeing these..

glyph · 2019-08-03T19:46:41Z

The twisted bug is fixed but the release containing it hasn't been finalized. Can you try with Twisted==19.7.0rc1 and see if it addresses your issue?

tardyp · 2020-05-23T18:37:17Z

lets close as there has not been new report of this. there is still #4162 though

seankelly mentioned this issue Jun 22, 2018

Exception: NoneType object has no attribute 'writeSequence' #4162

Open

brian-peloton mentioned this issue Jun 22, 2018

Failed API request during self shutdown writeHeaders writeSequence #3705

Closed

seankelly added the external_dep_bug label Sep 25, 2018

tardyp closed this as completed May 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exceptions writing http to the client #4045

Exceptions writing http to the client #4045

alexlarsson commented Apr 12, 2018

tardyp commented Apr 12, 2018

alexlarsson commented Apr 12, 2018

alexlarsson commented Apr 12, 2018

marrak commented Jun 14, 2018

tardyp commented Jun 23, 2018

brian-peloton commented Jun 24, 2018

alexlarsson commented Jun 25, 2018

devrajatverma commented Jul 6, 2018

ghost commented Jul 13, 2018

brian-peloton commented Jul 13, 2018

meffie commented Aug 7, 2018

glyph commented Sep 24, 2018

marcin-jedynski commented Dec 6, 2018

yhager commented Jun 7, 2019

glyph commented Aug 3, 2019

tardyp commented May 23, 2020

Exceptions writing http to the client #4045

Exceptions writing http to the client #4045

Comments

alexlarsson commented Apr 12, 2018

tardyp commented Apr 12, 2018

alexlarsson commented Apr 12, 2018

alexlarsson commented Apr 12, 2018

marrak commented Jun 14, 2018

tardyp commented Jun 23, 2018

brian-peloton commented Jun 24, 2018

alexlarsson commented Jun 25, 2018

devrajatverma commented Jul 6, 2018

ghost commented Jul 13, 2018

brian-peloton commented Jul 13, 2018

meffie commented Aug 7, 2018

glyph commented Sep 24, 2018

marcin-jedynski commented Dec 6, 2018

yhager commented Jun 7, 2019

glyph commented Aug 3, 2019

tardyp commented May 23, 2020