New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gerritbot: deployed version crashes sometimes in GerritCL.Footer #27536

Closed
dmitshur opened this Issue Sep 6, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@dmitshur
Member

dmitshur commented Sep 6, 2018

gerritbot has crashed a few times in production. The latest log ended with:

...
2018/08/26 18:54:12 Reloaded data from log *maintner.netMutSource.
2018/08/26 18:54:12 Starting corpus update loop ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x8dbc5c]
goroutine 139 [running]:
golang.org/x/build/maintner.(*GerritCL).Footer(0xc46dc8efd0, 0xacaf9e, 0x14, 0x0, 0x0)
        /go/src/golang.org/x/build/maintner/gerrit.go:418 +0x4c
main.(*bot).checkPullRequests.func1.1(0xc46dc8efd0, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:308 +0x4d
golang.org/x/build/maintner.(*GerritProject).ForeachOpenCL(0xc42a6173b0, 0xc420220d90, 0xc42bc390f4, 0x2)
        /go/src/golang.org/x/build/maintner/gerrit.go:189 +0x246
main.(*bot).checkPullRequests.func1(0xc42a6173b0, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:307 +0xa2
golang.org/x/build/maintner.(*Gerrit).ForeachProjectUnsorted(0xc42f0512c0, 0xc420220e88, 0xc4201f2240, 0xc420220ef0)
        /go/src/golang.org/x/build/maintner/gerrit.go:84 +0xa1
main.(*bot).checkPullRequests(0xc4201f2230)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:302 +0xee
main.(*bot).corpusUpdateLoop(0xc4201f2230, 0xb359e0, 0xc42009a020)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:276 +0x75
created by main.main
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:64 +0x264

It looks like gerritbot was built a while ago (month or two), and there have been changes to maintner since then (that might help resolve the issue).

I'm guessing line 418 in gerrit.go refers to this old commit:

https://github.com/golang/build/blob/dd35151f24e38c0c3b2829571908d18ce9388c1c/maintner/gerrit.go#L418

First step will be to redeploy it with the current version of code, and see if that resolves the issue. There's a good chance it might. If not, the next step is to investigate how that value ended up being nil when it shouldn't be.

/cc @andybons

@dmitshur dmitshur self-assigned this Sep 6, 2018

@gopherbot gopherbot added this to the Unreleased milestone Sep 6, 2018

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 6, 2018

Member

Redeployed now with current version. I'll check back in a week and see if it re-occurs.

Member

dmitshur commented Sep 6, 2018

Redeployed now with current version. I'll check back in a week and see if it re-occurs.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 13, 2018

Member

Haven't seen any crashes since then. I'll consider this resolved until there's evidence otherwise (we can re-open this).

Member

dmitshur commented Sep 13, 2018

Haven't seen any crashes since then. I'll consider this resolved until there's evidence otherwise (we can re-open this).

@dmitshur dmitshur closed this Sep 13, 2018

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 17, 2018

Member

Seeing a few more instances of the same crash. This time it should be easier to debug because the deployed version is known/latest:

2018/09/15 16:16:52 Reloaded data from log *maintner.netMutSource.
2018/09/15 16:16:52 Starting corpus update loop ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x9033e3]

goroutine 123 [running]:
golang.org/x/build/maintner.(*GerritCL).Footer(0xc04aff1130, 0xaeac4b, 0x14, 0x0, 0x0)
        /go/src/golang.org/x/build/maintner/gerrit.go:441 +0x43
main.(*bot).checkPullRequests.func1.1(0xc04aff1130, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:309 +0x4d
golang.org/x/build/maintner.(*GerritProject).ForeachOpenCL(0xc0039aab90, 0xc0001cfd90, 0xc0067d8234, 0x2)
        /go/src/golang.org/x/build/maintner/gerrit.go:189 +0x23c
main.(*bot).checkPullRequests.func1(0xc0039aab90, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:308 +0xa8
golang.org/x/build/maintner.(*Gerrit).ForeachProjectUnsorted(0xc00f7b7c20, 0xc0001cfe88, 0xc0001e8150, 0xc0000a4000)
        /go/src/golang.org/x/build/maintner/gerrit.go:84 +0x9f
main.(*bot).checkPullRequests(0xc0001e8140)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:303 +0xf1
main.(*bot).corpusUpdateLoop(0xc0001e8140, 0xb85a60, 0xc00009a020)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:277 +0x75
created by main.main
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:64 +0x250
Member

dmitshur commented Sep 17, 2018

Seeing a few more instances of the same crash. This time it should be easier to debug because the deployed version is known/latest:

2018/09/15 16:16:52 Reloaded data from log *maintner.netMutSource.
2018/09/15 16:16:52 Starting corpus update loop ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x9033e3]

goroutine 123 [running]:
golang.org/x/build/maintner.(*GerritCL).Footer(0xc04aff1130, 0xaeac4b, 0x14, 0x0, 0x0)
        /go/src/golang.org/x/build/maintner/gerrit.go:441 +0x43
main.(*bot).checkPullRequests.func1.1(0xc04aff1130, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:309 +0x4d
golang.org/x/build/maintner.(*GerritProject).ForeachOpenCL(0xc0039aab90, 0xc0001cfd90, 0xc0067d8234, 0x2)
        /go/src/golang.org/x/build/maintner/gerrit.go:189 +0x23c
main.(*bot).checkPullRequests.func1(0xc0039aab90, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:308 +0xa8
golang.org/x/build/maintner.(*Gerrit).ForeachProjectUnsorted(0xc00f7b7c20, 0xc0001cfe88, 0xc0001e8150, 0xc0000a4000)
        /go/src/golang.org/x/build/maintner/gerrit.go:84 +0x9f
main.(*bot).checkPullRequests(0xc0001e8140)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:303 +0xf1
main.(*bot).corpusUpdateLoop(0xc0001e8140, 0xb85a60, 0xc00009a020)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:277 +0x75
created by main.main
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:64 +0x250

@dmitshur dmitshur reopened this Sep 17, 2018

@dmitshur dmitshur removed their assignment Sep 17, 2018

@dmitshur dmitshur self-assigned this Sep 17, 2018

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 17, 2018

Member

I see a bug in maintner code. It's likely the bug causing this crash, but I don't have full information to say that conclusively.

I'll start by fixing it and see if anything more needs to be done. Will send a CL soon.

Member

dmitshur commented Sep 17, 2018

I see a bug in maintner code. It's likely the bug causing this crash, but I don't have full information to say that conclusively.

I'll start by fixing it and see if anything more needs to be done. Will send a CL soon.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 17, 2018

Change https://golang.org/cl/135677 mentions this issue: maintner: skip incomplete CLs in ForeachOpenCL

gopherbot commented Sep 17, 2018

Change https://golang.org/cl/135677 mentions this issue: maintner: skip incomplete CLs in ForeachOpenCL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment