Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gitmirror: build repo is broken due to updateDashboard failing to postCommit on a commit with 2 parents #35828

Open
dmitshur opened this issue Nov 25, 2019 · 6 comments
Milestone

Comments

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Nov 25, 2019

From https://farmer.golang.org/#health right now:

This has happened a few times in the last few days. I've fixed previous instances by restarting gitmirror, but since it keeps recurring, this will need to be investigated and fixed more.

/cc @toothrot @cagedmantis

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 25, 2019

Change https://golang.org/cl/208663 mentions this issue: cmd/gitmirror: log error details when Repo.Loop operations fail

@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Nov 25, 2019

If it helps:

$ for IP in $(kubectl get pods  -l app=gitmirror -o=jsonpath={.items[*].status.podIP}); do curl http://$IP:8585/debug/goroutines; done

.. to look at the goroutine stacks of the active pods.

gopherbot pushed a commit to golang/build that referenced this issue Nov 25, 2019
The error value is not exposed anywhere. Start by logging it,
so that it's possible to look up the error details from logs.

Updates golang/go#35828

Change-Id: I0cfc89685269a3511c8aee767459be7a198860b1
Reviewed-on: https://go-review.googlesource.com/c/build/+/208663
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur dmitshur self-assigned this Nov 25, 2019
@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Nov 25, 2019

If it turns out this is hanging due to updating the dashboard, note that all the code's gone as of pending https://go-review.googlesource.com/c/build/+/208697

@dmitshur

This comment has been minimized.

Copy link
Member Author

@dmitshur dmitshur commented Nov 25, 2019

It is. It happened again, and here are the logs:

[...]
2019/11/25 22:18:37 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:37 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:18:49 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:49 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:19:01 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:19:01 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
[...]

The "parent commit not found" error is likely related to commit golang/build@63986c1 having 2 parents instead of the usual 1.

It sounds like we can let this wait for CL 208697 before trying to make progress here, does that sound good to you @bradfitz?

@dmitshur dmitshur removed their assignment Nov 25, 2019
@dmitshur dmitshur changed the title x/build/cmd/gitmirror: build repo keeps breaking/hanging x/build/cmd/gitmirror: build repo is broken due to updateDashboard failing to postCommit on a commit with 2 parents Nov 25, 2019
@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Nov 25, 2019

Yup. Let's wait.

That code is all scary and it's not worth debugging. Easier to delete it all.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 26, 2019

Change https://golang.org/cl/208697 mentions this issue: app, gitmirror, maintner: use maintner for dashboard, not datastore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.