x/build/cmd/gopherbot: race condition in auto-closure of scratch CLs causing gopherbot to stall 30s

[andybons]

When closing scratch CLs, the maintner corpus doesn't get updated fast enough and it attempts to close the same CLs again, resulting in a 409 conflict:

2018/10/19 20:19:32 closing scratch CL https://golang.org/cl/140997 ...
2018/10/19 20:19:32 abandon scratch reviews: HTTP status 409 Conflict; change is abandoned
2018/10/19 20:19:32 HTTP status 409 Conflict; change is abandoned
2018/10/19 20:19:32 gopherbot ran in 1.745149115s
2018/10/19 20:19:32 sleeping 30s after previous error.

This error causes gopherbot to sleep for 30s as a result.

This is related to #28226 /cc @dmitshur

[dmitshur]

I think I've already seen code somewhere that uses Gerrit API in addition to maintner to confirm the "latest" state before taking some action. @bradfitz will probably remember where that was.

I suspect the fix for this can use a similar approach.

Note that this is different from #28226 in that the reported state isn't inconsistent, it's just not up to date.

[andybons]

@dmitshur auto-assignment of reviews and gerritbot both use the Gerrit API so that’s likely where you saw it.

[bradfitz]

@dmitshur, can you take this one?