Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: "no commits" for new branches #21555

Closed
adams-sarah opened this issue Aug 22, 2017 · 8 comments

Comments

Projects
None yet
3 participants
@adams-sarah
Copy link
Contributor

commented Aug 22, 2017

https://build.golang.org/?branch=dev.boringcrypto
said "no commits" until gitmirror restart.

gitmirror should always have an up-to-date set of "known branches" for which it builds commits.

cc @rsc @broady

@gopherbot gopherbot added this to the Unreleased milestone Aug 22, 2017

@gopherbot gopherbot added the Builders label Aug 22, 2017

@adams-sarah

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2017

okie dok i'm fairly certain i've tracked down the issue.

there is a for loop running in a goroutine on gitmirror which runs git fetch every N seconds.
the git fetch (origin being https://go.googlesource.com/blah) is giving a 502 on the rare-ish occasion.
when this happens, something is killing the goroutine. the git fetch loop then never restarts.

@adams-sarah

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2017

logs:

502 => fatal:

2017/08/22 00:11:21 exp: git fetch: exit status 128
fatal: unable to access 'https://go.googlesource.com/exp/': The requested URL returned error: 502

also, a non-502 error producing the same result:

2017/08/22 15:32:59 sys: git fetch: exit status 128
fatal: unable to access 'https://go.googlesource.com/sys/': GnuTLS recv error (-110): The TLS connection was non-properly terminated.

EDIT:
(these each happened on different pods)

@adams-sarah

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2017

nope nvm, doesn't look like that's it.

@gopherbot

This comment has been minimized.

Copy link

commented Sep 14, 2017

Change https://golang.org/cl/63870 mentions this issue: cmd/gitmirror: add debug logs

@adams-sarah

This comment has been minimized.

Copy link
Contributor Author

commented Sep 15, 2017

Ok so I've finally figured out what is happening. I just don't know where in the code it is happening yet.
Here is my update:

Essentially, we are continuously constructing a double-linked list of all the commits on a given branch. When we find a new commit, we link it to its parent based on the parent's SHA, such that parentCommit.child == newCommit && newCommit.parent == parentCommit.
Now, for the first new commit on a given branch (not the base commit from the fork to create the branch) this link is erased somewhere after it is created such that now, parent.child == nil && child.parent == nil.

This makes it so that when we take a parent commit and try to iterate over its children commits to post them each to the dashboard, the parent commit of our new commit has no children. So, the first commit on a branch is never sent to the dashboard.

The second piece of this is that the builders are set up such that if a commit is pushed to the dashboard, and the dashboard does not recognize the new commit's parent commit SHA, it will reject the new commit. This puts us in a place where, if we lose one commit, we can never recover (until we restart gitmirror).

This second bit was also what caused #21849.

Working more on this today. Hoping to have CL(s) up soon.

cc @bradfitz @andybons also on this issue

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 15, 2017

Thanks for looking into this. I've never really understood (or dug into) how this code works. I think @adg wrote it originally, but I've seen moved it around a few times. It's possible I broke it in the process during one of the moves. Or maybe it rotted a bit during the Mercurial to Git or other move.

@adams-sarah

This comment has been minimized.

Copy link
Contributor Author

commented Sep 15, 2017

I found it!! oh man, my eyes are so crossed.
sending cl. 1 line fix.

in short, we insert a copy of the commit into our linked list, not the commit obj w/ the correct address.
so linked list gets all messed up.

phew that was a pain

@gopherbot

This comment has been minimized.

Copy link

commented Sep 15, 2017

Change https://golang.org/cl/64150 mentions this issue: cmd/gitmirror: send new branch commits to dashboard

@golang golang locked and limited conversation to collaborators Sep 18, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.