Repository navigation
x/build: solaris-amd64 builder remains offline #81754
Description
Go version
go trunk on solaris/amd64
Output of go env in your module/workspace:
This is just the swarming bot.What did you do?
Nothing changed on my end AFAICT: the builder just remains offline.
What did you see happen?
The Solaris/amd64 builder remains offline since 2026-09-16. The error message isn't helpful at all: first
Failed to install user packages for this build
and since a few days nothing at all.
The logs aren't particularly helpful (at least to me) since they are extremely verbose with no clear indication where to look. One thing stood out that may be the cause of this: run_isolated.log has lots of
11073 2026-09-20 21:26:30.099 W: Retrying request https://storage.googleapis.com/chrome-infra-packages/store/SHA256/7654c8d5df0ebe00e7d88f6fffaa51d3e90cf5ef17c84f888b4ba2c886760b5f, attempt 29/30...
11073 2026-09-20 21:26:30.101 D: Starting new HTTPS connection (150): storage.googleapis.com:443
11073 2026-09-20 21:26:30.119 W: Unable to open url https://storage.googleapis.com/chrome-infra-packages/store/SHA256/7654c8d5df0ebe00e7d88f6fffaa51d3e90cf5ef17c84f888b4ba2c886760b5f?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=chrome-infra-packages%40appspot.gserviceaccount.com%2F20260920%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20260920T204441Z&X-Goog-Expires=7199&X-Goog-Signature=81c0ecaee7f8c2c272dbd0decadb0945b62bbbfb9738c6fc8ac5a569b94b2eca9d09762e186cf8197512ea020fc5c1de79681c5fe8702d0fc2918ec135094fe9d07221912f29fe38bceb916df9f58f2187b7c6263964b878e8f346540643acd4b151150a432afcb047b86623ac61c2ae7720b83bf39c5c685479dc9a4bf13e29bd0413edb5159351cca4862446b4a287ed8a1fbcfdffd0e5f5d300dceb3297346b64016401f7a103b2ce0d7309ce6f94b1586960820e6ab1ca92502a9af83d23bfcd008f824bd7d4f9945cb18663389f03d7b7e7d104e6a3acb504a1f1b9fb3a1e0ff12839eedc317ea3eef8c1df7af2dad3731b416e3ec9b952acdfe5799b7b&X-Goog-SignedHeaders=host&response-content-disposition=attachment%3B+filename%3D%22cipd%22 on attempt 29: ('Connection aborted.', ConnectionResetError(131, 'Connection reset by peer'))
11073 2026-09-20 21:26:30.121 E: internal failure: Could not fetch CIPD client after 5 retries
I already tried removing all of the swarming user's home directory (except for the client certificates), but to no avail.
Also, setting GOPROXY=direct in the builder environment seems to have no effect.
What did you expect to see?
The builder should remain running as it has for years now.
Activity
gabyhelp commentedon Sep 25, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
prattmic commentedon Sep 29, 2026
cc @golang/release
prattmic commentedon Sep 29, 2026
It looks like it's failing to connect to Google Cloud Storage. There is probably some issue on your network? Clearly networking works to some extent, as it received a swarming task and uploaded logs.
I think we've seen cases like this where seemingly random connections dropped when the network's NAT table simply ran out of ports and could not allocate additional connections.
rorth commentedon Sep 29, 2026
I don't think so: things have worked for years just fine, and there were no recent networking changes on our side. E.g. I can just telnet to storage.googleapis.com:443 without issues.
There's no NAT involved at all: the system is (and always has been) using a globally routable IPv4 address.
rorth commentedon Sep 30, 2026
I've just restarted the buildbot yet another time: without changing anything else, the build succeeds now. I don't have the slightest idea what might have caused this, but things seem to be back to normal now.
prattmic commentedon Sep 30, 2026
Huh, OK. Glad its working again. Unfortunately I don't have any more great ideas for debugging if it starts failing again.