x/build: some Gerrit CLs are getting into a state where they can't be updated

[dmitshur]

From gopherbot logs, there are a number of CLs where a reviewer fails to be added due to the error "Change nnn may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.":

2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/text/+/238119
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/text/+/238119: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:mpvl@golang.org State:}]}
2020/07/10 13:27:29 Could not set review for change "text~238119": HTTP status 409 Conflict; Change 238119 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/212839
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/212839: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~212839": HTTP status 409 Conflict; Change 212839 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/231681
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/231681: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~231681": HTTP status 409 Conflict; Change 231681 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/234239
2020/07/10 13:27:29 Setting review on https://go-review.googlesource.com/c/mobile/+/234239: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:29 Could not set review for change "mobile~234239": HTTP status 409 Conflict; Change 234239 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:29 No reviewers or cc: https://go-review.googlesource.com/c/mobile/+/240009
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/mobile/+/240009: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:hyangah@gmail.com State:}]}
2020/07/10 13:27:30 Could not set review for change "mobile~240009": HTTP status 409 Conflict; Change 240009 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/go/+/223118
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/go/+/223118: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:bradfitz@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "go~223118": HTTP status 409 Conflict; Change 223118 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/223779
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/tools/+/223779: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:dmitshur@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "tools~223779": HTTP status 409 Conflict; Change 223779 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/231464
2020/07/10 13:27:30 Setting review on https://go-review.googlesource.com/c/tools/+/231464: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:dmitshur@golang.org State:}]}
2020/07/10 13:27:30 Could not set review for change "tools~231464": HTTP status 409 Conflict; Change 231464 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:30 No reviewers or cc: https://go-review.googlesource.com/c/tools/+/239498
2020/07/10 13:27:31 Setting review on https://go-review.googlesource.com/c/tools/+/239498: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:matloob@golang.org State:}]}
2020/07/10 13:27:31 Could not set review for change "tools~239498": HTTP status 409 Conflict; Change 239498 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.
2020/07/10 13:27:31 No reviewers or cc: https://go-review.googlesource.com/c/net/+/187478
2020/07/10 13:27:31 Setting review on https://go-review.googlesource.com/c/net/+/187478: {Message: Labels:map[] Comments:map[] Reviewers:[{Reviewer:mikioh.public.networking@gmail.com State:}]}
2020/07/10 13:27:31 Could not set review for change "net~187478": HTTP status 409 Conflict; Change 187478 may not exceed 5000 updates. It may still be abandoned or submitted. To continue working on this change, recreate it with a new Change-Id, then abandon this one.

It's not possible to leave review comments on those CLs.

Need to investigate why this is happening.

From initial observations, it might be CLs that GerritBot imported from PRs, where PRs have had large review comments posted (e.g., golang/tools#215 (review)), which might have caused some sort of fail-and-repeat loops to make an excess of 5000 API calls to the Gerrit CL, causing Gerrit to lock those CLs. But I've only looked very briefly so far, more investigation is needed to confirm this or find another explanation.

/cc @golang/osp-team

[dmitshur]

The relevant code for this error message in Gerrit source code is at https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.2.2/java/com/google/gerrit/server/notedb/OpenRepo.java#181.

[andybons]

I’m going to temporarily disabled assigning reviewers to CLs, as it’s possible that there’s an infinite loop there.

[gopherbot]

Change https://golang.org/cl/247195 mentions this issue: cmd/gopherbot: temporarily disable assigning reviewers

[dmitshur]

We ran into this in https://go.dev/cl/231464 (created in May 2020), reported by @ianthehat recently.