-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proxy.golang.org: Intermittent TLS/Network errors with Google's Module Proxy #69826
Comments
Just had it happen again:
|
isn't this more likely to be a network issue in your local network? |
This is happening in GitHub actions primarily. I've also confirmed its happening to someone who is using self-hosted AWS runners. This issue seems similar to #63562 |
I've now added
To my bazelrc, which effectively just changes GOPROXY. So far it seems we've not hit these errors. I'll keep an eye out on this and see if it fixes it. If it does, I suspect some load balancer at Google is struggling. |
I have not seen this issue happen since we’ve changed GOPROXY. I’ll still keep an eye out on it. |
We experienced similar errors at Figma with rules_go until we set up a mirror to use as GOPROXY. #63244 |
We're hitting this issue today, from AWS EC2 hosted CI runners (not from github actions). |
I've switched us over to the following
|
CC @golang/tools-team. |
We suspect this is a transient network issue. Are folks still experiencing this? |
I've for now switched to using multiple proxies, so I dont think I'll be able to know for certain.. |
I just ran into this locally again when using only proxy.golang.org.
|
Thanks @aaomidi |
If I retry it does succeed. I don't see it that often, but that's partially because of heavy caching on my end. I no longer see this in CI after adding goproxy.io to the list of proxies it can retrieve data from. I do think this is specifically an issue with how gazelle interacts with the go module proxy, on a subset of the Google LBs. |
I frequently see this on CI runners hosted on aws ec2. At least multiple times per day this past week. It does succeed when retrying, but sometimes may fail again if we retry |
cc: @samthanawalla |
Since I don't think we'll be able to resolve network issues between aws and Google frontends, it seems like the most tractable solution to this problem would be to incorporate retries into the Go command (#28194). From skimming that issue, it is not clear whether we want to support this. |
This has nothing to do with AWS for what it’s worth. I’ve had this issue on GitHub actions which would be azure. I don’t think it’s the network either. This issue is really mainly prominent in Bazel tooling. |
@aaomidi got it. Absent more detailed traces indicating a problem with the go command or module proxy, this doesn't seem actionable from the perspective of the proxy. |
I’m not sure that’s true though. Especially given Bazel and go are both Google projects. I think realistically this is only really actionable by Google by involving the Bazel team with this bug. Would a repro CI run help with this? |
The errors reported in this issue all seem to be network timeouts. The fact that it mainly shows up in Bazel may just mean that Bazel is the main program contacting the Go proxy. After all, if you are using Bazel there isn't much reason for anything else to contact the Go proxy. And that suggests that there are network problems between wherever people are running Bazel and the Go proxy. I don't see how fixes to either Bazel or the Go proxy can affect that. What might conceivably help is to collect the IP addresses that are failiing, both the IP address where Bazel is running and the IP address that it is using to contact the module proxy. Or, yes, if there is a way to reproduce this reliably that would be helpful. |
Go version
go version go1.23.1 darwin/arm64
Output of
go env
in your module/workspace:What did you do?
Environment:
Ran a
bazel build ${some_target}
Note that I'm not sure if this problem is limited to bazel, or happens with normal go invocations as well.
What did you see happen?
Intermittently, some modules fail to download from Google's Go Module Proxy:
What did you expect to see?
No errors.
The text was updated successfully, but these errors were encountered: