-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proxy.golang.org, cmd/go, net/http: http2 PROTOCOL_ERRORS from the proxy #69544
Comments
#51323 smells similar, but different error codes. |
We have seen four of these in the past week too. |
On September 18th at around 08:20 UTC:
September 19th around 15:00 UTC:
September 20th around 07:40 UTC:
September 20th around 09:40 UTC:
Other failures around the same days which we almost never encounter: September 19th around 16:00 UTC:
September 20th around 11:40 UTC:
|
Same here, found 3-rd party mirror works. |
"received DATA before a HEADERS frame" is the Go HTTP/2 client, and it's complaining about exactly what it sounds like. Either the server is committing a protocol violation, or something is going on that I don't understand (always possible). |
Another one here, I'm currently working around it by setting
|
It'd be very helpful if anyone could get GODEBUG=http2debug=1 output for a time period covering a failure. |
This temporarily adds a GODEBUG flag (as requested at golang/go#69544 (comment)) to a CI step whose output we don't much care about spamming, to help debug a Go proxy issue that's started to affect us sporadically. Signed-off-by: Jonathan Matthews <github@hello.jonathanmatthews.com> Change-Id: I030b06b3991b85da352d2a8bd98e0b0693831267 Dispatch-Trailer: {"type":"trybot","CL":1201575,"patchset":2,"ref":"refs/changes/75/1201575/2","targetBranch":"master"}
Brad mentioned that they saw this on GitHub Actions. Can others report if you also saw this on GitHub Actions, or on some other infrastructure (such as a personal computer or other CI)? |
I believe I can speak for @mvdan: we have seen this only from inside GitHub Actions. |
It turns out we see this in Go's CI as well, which is not GH Actions. |
I experience this on my laptop. |
Thanks everyone for your help. @neild was able to reproduce the problem reliably, and the cause has been found in a recent rollout to Google frontends. The rollback is proceeding. |
I was worried that Go's http2 client was getting confused, but this server response does look wrong, with the DATA for stream=451 coming before its HEADERS..... (from @kevindejong's attachment above)
|
GFE rollout should be completed now, no further errors expected. |
Go version
go1.23.1 Linux (on GitHub Actions CI runs)
What did you do?
A ton of our GitHub CI jobs started failing today with http2 protocol errors from
proxy.golang.org
while fetching modules.e.g.
and
and
and
and a dozen others, all similar.
This doesn't correspond to any change on our side.
Is this a Google HTTP frontend push gone bad? Is it now strict about something that it wasn't before and either its check is wrong or Go's HTTP/2 client is buggy in ways we'd never noticed previously?
/cc @neild
What did you see happen?
The
cmd/go
tool is saying, during module fetches:... and then failing.
The text was updated successfully, but these errors were encountered: