x/build: darwin/amd64 x/tools builder spuriously observed to download dependencies at 0.02 MiB/s

[mknyszek]

https://ci.chromium.org/ui/p/golang/builders/try/x_tools-go1.25-darwin-amd64_14/b8697740926644613329/overview took over 1 hour to run, and the culprit was a very slow download rate before golangbuild ever even started running.

[P7320 13:53:05.348 storage.go:245 I] [fetch  6/15] Fetching: 31.1/46.9 MB ( 66%, 0.02 MB/s)
[P7320 13:53:06.915 storage.go:245 I] [fetch  6/15] Fetching: 31.2/46.9 MB ( 66%, 0.02 MB/s)
[P7320 13:53:06.915 storage.go:245 I] [fetch 12/15] Fetching: 11.0/12.7 MB ( 87%, 0.02 MB/s)
[P7320 13:53:09.451 storage.go:245 I] [fetch  6/15] Fetching: 31.2/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:09.451 storage.go:245 I] [fetch 12/15] Fetching: 11.0/12.7 MB ( 87%, 0.02 MB/s)
[P7320 13:53:10.135 storage.go:245 I] [fetch  6/15] Fetching: 31.2/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:11.179 storage.go:245 I] [fetch  6/15] Fetching: 31.3/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:11.235 storage.go:245 I] [fetch 12/15] Fetching: 11.1/12.7 MB ( 87%, 0.02 MB/s)
[P7320 13:53:14.814 storage.go:245 I] [fetch 12/15] Fetching: 11.1/12.7 MB ( 87%, 0.02 MB/s)
[P7320 13:53:14.814 storage.go:245 I] [fetch  6/15] Fetching: 31.3/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:17.161 storage.go:245 I] [fetch  6/15] Fetching: 31.3/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:17.248 storage.go:245 I] [fetch 12/15] Fetching: 11.1/12.7 MB ( 88%, 0.02 MB/s)
[P7320 13:53:18.203 storage.go:245 I] [fetch 12/15] Fetching: 11.1/12.7 MB ( 88%, 0.02 MB/s)
[P7320 13:53:19.242 storage.go:245 I] [fetch  6/15] Fetching: 31.3/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:19.243 storage.go:245 I] [fetch 12/15] Fetching: 11.2/12.7 MB ( 88%, 0.02 MB/s)
[P7320 13:53:21.031 storage.go:245 I] [fetch 12/15] Fetching: 11.2/12.7 MB ( 88%, 0.02 MB/s)
[P7320 13:53:21.032 storage.go:245 I] [fetch  6/15] Fetching: 31.4/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:21.794 storage.go:245 I] [fetch  6/15] Fetching: 31.4/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:21.892 storage.go:245 I] [fetch 12/15] Fetching: 11.2/12.7 MB ( 88%, 0.02 MB/s)
[P7320 13:53:24.163 storage.go:245 I] [fetch 12/15] Fetching: 11.2/12.7 MB ( 89%, 0.02 MB/s)
[P7320 13:53:24.163 storage.go:245 I] [fetch  6/15] Fetching: 31.4/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:27.834 storage.go:245 I] [fetch 12/15] Fetching: 11.2/12.7 MB ( 89%, 0.02 MB/s)
[P7320 13:53:28.476 storage.go:245 I] [fetch 12/15] Fetching: 11.3/12.7 MB ( 89%, 0.02 MB/s)
[P7320 13:53:30.757 storage.go:245 I] [fetch  6/15] Fetching: 31.4/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:31.779 storage.go:245 I] [fetch  6/15] Fetching: 31.5/46.9 MB ( 67%, 0.02 MB/s)
[P7320 13:53:31.870 storage.go:245 I] [fetch 12/15] Fetching: 11.3/12.7 MB ( 89%, 0.02 MB/s)

(From https://chromium-swarm.appspot.com/task?id=74b6ab5dc38bd810.)

Thankfully, the machine (https://chromium-swarm.appspot.com/bot?id=darwin-amd64-14--f409407c-0460-489e-8d4c-59551c62c446.golang.cf.macservice.goog) appears to have recovered.

If this happens again, we should consider disabling this machine.

[gabyhelp]

Related Issues

• x/build: darwin-amd64-longtest builds are slow #62215 (closed)
• x/build: increase timeouts or reduce parallelism on darwin-arm-mg912baios builder? #32024
• x/build: `darwin-amd64` LUCI builders are sometimes too slow #65040
• x/build: scaleway builders write_snapshot_tar consistently taking > 1h #21839 (closed)
• x/build: scaleway networking is super slow #22749 (closed)
• x/build: Builders takes too long to complete tests #45916 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}