Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gopherbot: gopherbot is fallible to sleep-error loops #30182

Closed
dmitshur opened this Issue Feb 12, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@dmitshur
Copy link
Member

dmitshur commented Feb 12, 2019

Gopherbot isn't happy right now. From its logs:

2019/02/12 01:01:02 got corpus update after 274.731276ms
freeze
	https://golang.org/issue/23772  x/build/cmd/gerritbot: <fill this in>
2019/02/12 01:01:02 freeze old issues: PUT https://api.github.com/repos/golang/go/issues/23772/lock: 404 Not Found []
2019/02/12 01:01:02 PUT https://api.github.com/repos/golang/go/issues/23772/lock: 404 Not Found []
2019/02/12 01:01:02 gopherbot ran in 110.450416ms
2019/02/12 01:01:02 sleeping 30s after previous error.
2019/02/12 01:01:32 Updating data from log *maintner.netMutSource ...
2019/02/12 01:01:36 Downloading 649 bytes of https://maintner.golang.org/logs/45 ...
2019/02/12 01:01:36 wrote /cache/golang-maintner/0045.growing.mutlog
2019/02/12 01:01:36 gerrit go.googlesource.com/gofrontend: Ref {CLNumber:161963 Version:0} => 3a9fd412dc7603e55cff110c741a02565ae83980
2019/02/12 01:01:36 Reloaded data from log *maintner.netMutSource.
2019/02/12 01:01:36 got corpus update after 3.996453811s
freeze
	https://golang.org/issue/23772  x/build/cmd/gerritbot: <fill this in>
2019/02/12 01:01:36 freeze old issues: PUT https://api.github.com/repos/golang/go/issues/23772/lock: 404 Not Found []
2019/02/12 01:01:36 PUT https://api.github.com/repos/golang/go/issues/23772/lock: 404 Not Found []
2019/02/12 01:01:36 gopherbot ran in 98.493511ms
2019/02/12 01:01:36 sleeping 30s after previous error.
2019/02/12 01:02:06 Updating data from log *maintner.netMutSource ...
...

It's constantly looping with "sleeping 30s after previous error" and not making any progress.

@bradfitz spotted this in #30181 (comment).

@dmitshur

This comment has been minimized.

Copy link
Member Author

dmitshur commented Feb 12, 2019

Issue #23772 was an empty/spam issue that was closed a year ago. It's showing up as 404 now, which gopherbot can't handle.

There are two things we need to fix here:

  1. This specific problem in the "freeze old issues" task: it should skip issues that are 404 and move on to the rest (not treat it as a fatal error).

  2. See if the high-level behavior can be improved such that if one task starts to fail, perhaps it can still perform the other tasks rather than short-circuiting and not attempting other tasks. Maybe it's not possible to make such an improvement in the general case, but it's worth considering.

@dmitshur dmitshur self-assigned this Feb 12, 2019

@dmitshur

This comment has been minimized.

Copy link
Member Author

dmitshur commented Feb 12, 2019

Since this isn't resolved yet, I have to manually say that I've sent a fix for the "soon" part of this issue as CL 161906.

@gopherbot gopherbot added this to the Unreleased milestone Feb 12, 2019

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 12, 2019

Change https://golang.org/cl/161906 mentions this issue: cmd/gopherbot: handle 404 GitHub issues in freezeOldIssues task

@dmitshur

This comment has been minimized.

Copy link
Member Author

dmitshur commented Feb 12, 2019

I've deployed the fix in the aforementioned CL, which is why gopherbot woke up. The CL still needs to be reviewed before it can be merged.

Edit: I'll remove the Soon label, since the immediate problem is resolved.

@dmitshur

This comment has been minimized.

Copy link
Member Author

dmitshur commented Feb 12, 2019

The current fix adds to the plate of "actions gopherbot keeps taking without any effect" (issue #28320):

freeze
	https://golang.org/issue/23772  x/build/cmd/gerritbot: <fill this in>

To fix that, we need to update maintner to detect issues that have gone missing, and set their NotExist field to true. CL 161521 may help with that. /cc @jmdobry

Edit: Opened #30184 for that.

@dmitshur dmitshur removed the Soon label Feb 12, 2019

gopherbot pushed a commit to golang/build that referenced this issue Feb 12, 2019

cmd/gopherbot: handle 404 GitHub issues in freezeOldIssues task
A GitHub issue can become 404. Attempting to lock it will produce a
404 response from the GitHub API. Don't treat it as a fatal error
when it happens.

Add a check for the NotExist field. This will help after golang/go#30184
is resolved.

Updates golang/go#30182

Change-Id: Ia04c59879909b1de00bd681606bfa331fe642cd4
Reviewed-on: https://go-review.googlesource.com/c/161906
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@dmitshur dmitshur changed the title x/build/cmd/gopherbot: gopherbot is in sleep-error loop x/build/cmd/gopherbot: gopherbot is prone to sleep-error loops Feb 12, 2019

@dmitshur dmitshur changed the title x/build/cmd/gopherbot: gopherbot is prone to sleep-error loops x/build/cmd/gopherbot: gopherbot is fallible to sleep-error loops Feb 12, 2019

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2019

Change https://golang.org/cl/164157 mentions this issue: cmd/gopherbot: don't return early on error in doTasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.