x/build: LUCI linux-riscv64-mengzhuo bot has a flaky network connection

[mknyszek]

The linux-riscv64-mengzhuo machine has a flaky network connection. Recent builds have timed out trying to obtain things like the tip-of-tree for a repository and trying to download prebuilt toolchains from the CAS service. The consecutive failures of the latter led the machine to be quarantined, draining the pool of linux-riscv64 machines for some time.

I will note that some of the work cmd/coordinator used to do is now done on the bot itself. This is just a consequence of the switch to LUCI's execution model.

@mengzhuo is there anything that can be done either by you or us to improve the stability of the machine? For instance, if we're OK with making builds slower, we can skip the prebuild on this platform and always run make.bash, even for subrepos. Fetching the tip-of-tree is a bit unavoidable for subrepo builds though, because those builds can only be triggered by one source. Perhaps we can target a different service, or add some other exception.

[mengzhuo]

All my riscv-builders are located in my homelab in Shenzhen, China, which is home ADSL network.
may be I can transfer these riscv development boards/boxes to a server-farm if possible for more stable/better QoS network. (Looking for funding from RISCV companies here.)

This is a ping latency graph of go.googlesoure.com/*.appspot.com in 24 hours, it looks pretty good to me that all packages go around the whole planet (<180ms)

[mengzhuo]

I take a look at some of failures "go.googlesoure.com" resolved to "173.194.174.82" which my firewall didn't allow this ip.
I had allow "173.194.0.0/16" now.

https://chromium-swarm.appspot.com/task?id=679f0bed18bf1910

[mknyszek]

Thanks for looking into that! The other category of issues I've seen is that the cas download step fails (because the attempt to download, and the retries) fail too. I can provide some instructions on how to try to reproduce that if it would be helpful.

Also, it looks like the bot went down recently? https://chromium-swarm.appspot.com/bot?id=linux-riscv64-mengzhuo has status dead and the last task ended with BOT_DIED.