x/build: trybots should rebase when testing

[bradfitz]

Currently with the trybot code, if Go's master is A and we have pending CLs B and C on A, currently we test B-on-A and C-on-A, but then when B is merged, we're still testing C on A, instead of C-on-B.

We should do:

https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#rebase-change

... which will add a patch set to the change, rebased. It could get spammy, though. Shawn says "don't complain when you have patch set 85 on a trivial one line change". So maybe we want to be careful when we do it. Maybe only for auto-submit.

[minux]

We can always do git cherry-pick after git fetch the CL in trybot. Any failures in merging is a build failure (the CL won't submit successfully anyway) Even if rebase the CL on gerrit, each time its author updates the CL, the rebased patch set will be overwritten.

[bradfitz]

We could, except that the trybot doesn't use any git right now. It pulls tar.gz files from Gerrit of a commit. We don't even have the "git" command on the builders.

[minux]

Ahh, that makes the problem harder. Could we add a patch primitive to the buildlet and send a patch for it to apply on latest git master tarball? We might be able to reuse golang.org/x/codereview/patch for that. But we need to add fuzzy support to that package.

[bradfitz]

I wouldn't do anything to the buildlet for this. We could apply the patch using a new service in a Docker container under the coordinator, using real git and/or real patch, and then extract the resulting tarball to push to the buildlets, like normal.

[minux]

Yeah, that's better.

[josharian]

This'll need to respect the branch as well to keep trybot useful for release branch testing.

[bradfitz]

@rsc says in email "Please keep using the exact commit. Thanks."

Russ, can you elaborate on what you'd like to see long-term? Would it be useful to also try the commit cherry-picked on top of tip?

[minux]

Please see https://golang.org/cl/6802/ as an extreme example of why this
issue should be fixed.

All trybots are fine with that CL, but because the CL is not rebased to the
latest master, after submission, it fails all builds.

[minux]

Sorry for the false alert. The problem I mentioned in the last comment is
caused by me running trybot on the wrong patch set. Will file a separate
issue.

Update: the actual reason is not me running on the wrong patch set.
It's because the auto-merge feature of Gerrit failed to merge the
change correctly. How could that be possible? (Note patch set 2 is created
by Gerrit, not by the author of the CL.)

So again, trybot should use rebase to test the patch in order to catch this
problem.

[minux]

Trybot really should rebase before testing, two more cases:

1. CL 7032 removes one argument to gc.Naddr, trybot passes, but
   the build breaks after it's submitted because master contains cmd/7g,
   but that CL is based on a commit before the introduction of 7g, so
   it doesn't change 7g.
2. trybot reports failure for CL 7080 on nacl/amd64p32, however,
   that's because the CL is based on a commit before the nacl build fix.
   The trybot messages asks to see the build dashboard if the failure
   is a new one, and according to the dashboard, it's indeed a new
   failure, but actually it's not.