cmd/dist: "unexpected stale targets" on linux-ppc64le-power9osu builder

[bcmills]

The linux-ppc64le-power9osu builder is frequently failing in cmd/dist.

It's not obvious to me exactly which cmd/dist invocation is failing, though. More logging would probably be helpful.

Curiously, the other *-ppc* builders do not seem to be affected.

2019-11-21T14:55:12-8bbfc51/linux-ppc64le-power9osu
2019-11-19T18:43:52-2ac9f1d/linux-ppc64le-power9osu
2019-10-30T20:46:44-81a74b4/linux-ppc64le-power9osu
2019-10-22T17:34:15-22278ca/linux-ppc64le-power9osu
2019-10-22T10:09:39-16f0f9c/linux-ppc64le-power9osu
2019-10-17T18:21:01-3972f97/linux-ppc64le-power9osu
2019-10-17T17:11:19-4569f1b/linux-ppc64le-power9osu
2019-10-16T23:10:06-4cabf69/linux-ppc64le-power9osu

This issue is forked from #33598 (comment).

CC @laboger @bradfitz @cagedmantis @dmitshur @toothrot

[laboger]

I will try to reproduce the problem. I also agree it is curious that it only happens on the power9 builder, which has happened with other intermittent failures too.

I don't know if this is possible but if somehow the value for objabi.GOPPC64 is not set to power9 before all compiles start, and some are compiled for power8 but rebuilt later for power9 then the hashes wouldn't match.

[laboger]

The errlogs look strange, almost like the build hasn't even started yet. It doesn't display the line that says it is building go_bootstrap or even the one that provides the detail about the test run.

[laboger]

I found that the old power9 builder was still active out in the farm (not running Docker). Not sure but maybe these errors were happening if that buildlet was being used, since the error happened intermittently. Carlos has shut down that buildlet and we will see if that makes a difference.

[laboger]

I think that was the problem. Now that the old buildlet isn't running, I am not seeing the error. When it did happen, it was failing every 5 or 6 runs which would make sense since that was the 6th one in the pool.

[bcmills]

Great, thank you!

I'll close this out; we can always reopen it if the problem isn't actually gone.