New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal Exception on DB while marking build requests as completed #3591

Open
Frodox opened this Issue Sep 7, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@Frodox
Member

Frodox commented Sep 7, 2017

2017-09-07 16:21:17+0300 [-] Got fatal Exception on DB
        Traceback (most recent call last):
        Failure: exceptions.KeyError: 

2017-09-07 16:21:17+0300 [-] while marking build requests as completed
        Traceback (most recent call last):
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/buildbot/data/buildrequests.py", line 248, in completeBuildRequests
            yield self.master.data.updates.maybeBuildsetComplete(bsid)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/buildbot/data/buildsets.py", line 230, in maybeBuildsetComplete
            cumulative_results, complete_at=complete_at)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/buildbot/db/pool.py", line 183, in __thd
            rv = callable(arg, *args, **kwargs)
          File "/home/builder/buildbot/sandbox/lib/python2.7/site-packages/buildbot/db/buildsets.py", line 152, in thd
            raise KeyError
        exceptions.KeyError: 

I regularly got this exception. I use postgresql-9.2.18, bb-0.9.9.post2.
GitLab webHook + ForceScheduler

how can I debug/help in fixing it?

@tardyp

This comment has been minimized.

Show comment
Hide comment
@tardyp

tardyp Sep 7, 2017

Member

you can try to print res.rowcount at the place of the error to see if there is 0 rowcount or more thant one rowcount.
Then we can try to understand why we have this unexpected value.

Member

tardyp commented Sep 7, 2017

you can try to print res.rowcount at the place of the error to see if there is 0 rowcount or more thant one rowcount.
Then we can try to understand why we have this unexpected value.

@tardyp

This comment has been minimized.

Show comment
Hide comment
@tardyp

tardyp Sep 24, 2017

Member

@Frodox I found with #3650 that this can happen when several buildrequests of the same buildset finish at the same time. Would that explain your issues?

Member

tardyp commented Sep 24, 2017

@Frodox I found with #3650 that this can happen when several buildrequests of the same buildset finish at the same time. Would that explain your issues?

@Frodox

This comment has been minimized.

Show comment
Hide comment
@Frodox

Frodox Sep 25, 2017

Member

It may be my case, because I have +/- same workers, so they can finish build ~same time.
How can I be sure? I didn't catch this error since your previous comment, when I added debug output into buildbot/db/buildsets.py. May be because of low load, vacations and small amount of workers

Member

Frodox commented Sep 25, 2017

It may be my case, because I have +/- same workers, so they can finish build ~same time.
How can I be sure? I didn't catch this error since your previous comment, when I added debug output into buildbot/db/buildsets.py. May be because of low load, vacations and small amount of workers

@Frodox

This comment has been minimized.

Show comment
Hide comment
@Frodox

Frodox Sep 27, 2017

Member

Just catch it again.

2017-09-27 18:37:17+0300 [-] DEBUG #3591, rowcount: 0
Member

Frodox commented Sep 27, 2017

Just catch it again.

2017-09-27 18:37:17+0300 [-] DEBUG #3591, rowcount: 0
@Frodox

This comment has been minimized.

Show comment
Hide comment
@Frodox

Frodox Oct 4, 2017

Member

Same with bb-0.9.11

Member

Frodox commented Oct 4, 2017

Same with bb-0.9.11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment