cmd/dist: "unexpected stale targets" on linux-riscv64-unmatched since 2022-06-06

[bcmills]

greplogs -l -e '(?ms)\Alinux-riscv64-unmatched.*unexpected stale targets'
2022-06-10T23:30:35-55590f3/linux-riscv64-unmatched
2022-06-10T22:10:29-ff3db8d/linux-riscv64-unmatched
2022-06-09T18:17:39-91019cc/linux-riscv64-unmatched
2022-06-08T21:56:02-13f6be2/linux-riscv64-unmatched
2022-06-07T18:47:38-f3e051a/linux-riscv64-unmatched
2022-06-07T16:07:58-d2630aa/linux-riscv64-unmatched
2022-06-06T19:03:15-95547ae/linux-riscv64-unmatched

(attn @mengzhuo)

[bcmills]

Curiously, all of the above failures are also missing the environment and bootstrapping headers (#51050), but maybe that's just a byproduct of the interaction between the staleness check and the way cmd/coordinator shards the tests?

[bcmills]

With this being a reverse builder, at first I thought it might be related to the recent cmd/go module indexing changes.

However, these failures started well before CL 410821 was merged, so I don't immediately see how that's possible.

[bcmills]

The other likely causes of unexpected stale targets that I can think of are:

• a bug in x/build/cmd/coordinator and/or x/build/cmd/buildlet causing the builder to run sharded tests starting from the wrong snapshot
• filesystem corruption on the builder (causing file hashes to differ for files that should be unchanged)
• some kind of variation in the process environment (which would be masked in the logs by x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050)?

[gopherbot]

Change https://go.dev/cl/412175 mentions this issue: dashboard: add a known issue for linux-riscv64-unmatched

[mengzhuo]

Thanks for reporting this ( I forget to submit this issue ).
After CL 411475 (different builders failed twice in dashboard).
I had cleared all temp files and rebuilt stage0 on unmatched01 and unmatched057.
All builders using /tmp/ (ramdisk) as it's workdir so I think filesystem corruption shouldn't be the problem.

[gopherbot]

Change https://go.dev/cl/413334 mentions this issue: cmd/buildlet/stage0: define env for linux-riscv64-unmatched

[mengzhuo]

The root cause might be my env setting to identify three builders in the pool.
All failed builders are running the bot by setup different WORKDIR and rm -rf $WORKDIR after bot complete their job.

[bcmills]

@mengzhuo, is this fixed? (If so, could you send a CL to remove the KnownIssue in x/build?)

greplogs -l -e '(?ms)\Alinux-riscv64-unmatched.*unexpected stale targets' --since=2022-06-11

(0 matching logs)