Buildbot sometimes checks out the wrong revision for try runs

[jdm]

servo/servo#13957 (comment) links to a log with a compilation error that is actually from servo/servo#13956 (comment) . This really confused me!

[aneeshusa]

I'm unable to reproduce - when I click through to the stdio and warnings logs for those comments, they have different contents - different lengths, file paths, headers, etc. (although they both end in the same error). Can you provide direct links to the logs you are seeing or maybe upload some screenshots?
There may be a caching problem somewhere, although I'm doubtful as one is for a windows-dev build and the other is for a linux-dev build.

[jdm]

The windows build in servo/servo#13956 failed with the same error as the linux one. The changes in servo/servo#13957 do not include the code that is reported as causing a windows build error.

[jdm]

However, you're absolutely right that http://build.servo.org/builders/windows-dev/builds/1770/steps/compile/logs/stdio and http://build.servo.org/builders/windows-dev/builds/1772/steps/compile/logs/stdio are different logs with different contents. This may be a problem of the code on the try branch not being updated correctly?

[jdm]

http://build.servo.org/builders/windows-dev/builds/1770/steps/git/logs/stdio and http://build.servo.org/builders/windows-dev/builds/1772/steps/git/logs/stdio show the same hash being used when updating the branch, which would explain why this occurred.

[jdm]

http://build.servo.org/builders/windows-dev/builds/1770 and http://build.servo.org/builders/windows-dev/builds/1772 show that different revisions should have been used.

[aneeshusa]

OK, that makes more sense and matches what I'm seeing, thanks for looking again.
http://build.servo.org/builders/windows-dev/builds/1772 has different values for revision (the revision triggering the build request, in this case a06cb1ca) and got_revision (the revision that was actually used during the build, in this case 59e651b7). Checking the git log, it seems like Buildbot fetched the try branch, tried to check out the commit from revision, did both of those steps again, then gave up and just used the current value of try as the got_revision branch.

I can see two potential problems here:

• Something force-pushing to try such that a06cb1ca was not available - this may be due to multiple restarts of Buildbot soon after each other when fixing #524 yesterday.
• Buildbot accepting whatever commit it found on the try branch (aka 59e651b7) instead of erroring out.

I'll look into some potential solutions.

[aneeshusa]

I made a PR that should fix the second problem, but I'm not sure about a good way to fix the first one; I think just manually retrying if we encounter this again is the most practical solution.

[jdm]

Thanks!

[aneeshusa]

Should be fixed by #531 and #554.