net/http: EOF returned from http.Transport #53472
Comments
|
that's not a raw EOF, it's a |
|
Oops, you’re completely correct. I was too focused on the user-visible text rather than on the type. |
|
At least using the func main() {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
go server(ln)
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("http://%s/", ln.Addr().String()), nil)
if err != nil {
panic(err)
}
res, err := http.DefaultTransport.RoundTrip(req)
fmt.Printf("res=%#v, err=%v (%#v) isEOF=%v\n", res, err, err, err == io.EOF)
}prints |
|
To be specific, my primary concern is about the lack of a clear user-visible error message text. Right now error messages in logs just say “EOF” with little to allow diagnosing the failure further: it doesn’t indicate the relevant component (client / server / proxy / hypothetically something else like a corrupt local (certificate?) file). I can, of course, wrap the So I’d prefer receiving a more descriptive error message from the net/http client. |
seankhliao
changed the title
|
cc @neild |
seankhliao
added
the
NeedsInvestigation
|
I just like to mention that I also struggle here. tr := http.DefaultTransport.(*http.Transport).Clone()
tr.ExpectContinueTimeout = 10 * time.Second
tr.DisableKeepAlives = true
tr.IdleConnTimeout = 10 * time.Second
client := http.Client{
Timeout: 10 * time.Second,
Transport: tr
}In a loop I create a new request and pass it to the client: req, err := http.NewRequest("GET", u, nil)
...
resp, err := c.client.Do(req)Most of the time err is nil - which is expected behaviour because the URL is valid and a valid response should be delivered. I test this in parallel with my browser. But in a few cases I get back an EOF error. Without any further information. In there a It's very likely somehow related to the host. In parallel I do exactly the same with another (local go-) server and I never get that EOF error. Still, I have no idea what's wrong here. I guess I have to dig even more into it. I just like to mention here that it would be really, really nice to get better error messages here, especially because the error chain doesn't pass a stack trace. |
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This resulted in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This resulted in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This resulted in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This resulted in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This results in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This results in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
…error For some reason the stdlib http client reports an unexpected closed connection as an io.EOF with no further information. This resulted in connection problems (like an internal firewall/proxy/connection config blocking connections to a Dynatrace SaaS URL) to just be reported as EOF to users. This fix makes some assumptions about the internals of the http client and may break in the future, but if the client is changed to no longer return io.EOF errors it will hopefully return a more helpful error instead. see also: golang/go#53472
|
I'm also experiencing this issue even on local. It can happen when many connections are made. The error message isn't very helpful because it's a plain "EOF". I read the server side log and it seems that the client closes the connection when this "EOF" happens. I was wondering if it's related to server side closing idle connection when Client side just established the connection. |
|
@G-M-twostay yes, based on the docs: // transportReadFromServerError is used by Transport.readLoop when the
// 1 byte peek read fails and we're actually anticipating a response.
// Usually this is just due to the inherent keep-alive shut down race,
// where the server closed the connection at the same time the client
// wrote. The underlying err field is usually io.EOF or some
// ECONNRESET sort of thing which varies by platform. But it might be
// the user's custom net.Conn.Read error too, so we carry it along for
// them to return from Transport.RoundTrip.
type transportReadFromServerError struct {
err error
}You should make the |
|
Copying over from my issue which got marked as duplicate of this one: I maintain a library that has a portion written in Go. Every once in a while, our users encounter issues caused by HTTP requests failing with just the message "EOF", for instance {"time":"2024-10-09T11:33:30.483338864-04:00","level":"DEBUG","msg":"[ERR] POST ---URL--- request failed: Post "---URL---": EOF"} The outer portion of this message ("[ERR] POST ... failed:") comes from the hashicorp/retryablehttp package. The inner portion is from net/http. The message is unhelpful and does not help me narrow down the issue, other than that it's a protocol problem (which is also not at all obvious). Users who experience this tend to have unique network configurations, and failures are consistent but don't happen for all kinds HTTP requests made by the client (certain requests consistently work, and other requests consistently fail). Without more context, it's very hard to help users debug their configurations. It would be massively helpful if net/http simply used fmt.Errorf("...: %w", err) in a few places. Likewise for x/net/http2 (I don't know how to tell which we're using). It seems this issue has been open a long time. Are there any plans to fix this, or any way to contribute a fix? Also, is there any advice for how to debug this sort of problem? |
|
To add to this, we have a user who partially fixed the EOF by using a build of the library downgraded to Go 1.22. We don't see anything in the Go release notes or the GODEBUG history that looks relevant (the SSL changes are suspicious, but I've seen them result in better error messages such as "tls: failed to parse certificate from server: x509: negative serial number"). If it's a regression in the Go standard library, then without a useful error message, I can't file a useful bug report. |
Patches are certainly welcome. See https://go.dev/doc/contribute. Thanks.
The first post in this issue points to some code that may be at fault. |
|
Did a little bit of digging through the source. Like @mtrmac identified, it looks like the error comes from the 1-byte
Lines 716 to 732 in c93477b which references #16465. It appears that the code unwrapping EDIT: It turned out the Go version was a red herring. The user was experiencing dropped connections because of a misconfigured "maximum transmission size" setting, causing packets beyond a certain size to get dropped. This manifested as slow network speeds and occasional EOF errors due to connections getting closed.
Leaving these notes here in case it helps others debug this error. Unfortunately, wrapping the EOF may be impossible due to backward compatibility requirements. But if it only has a single meaning (an unexpectedly closed connection), this meaning could perhaps be documented on |
|
Another way for an EOF to manifest is described here: 5dd372b
Our requests are not idempotent (we use POST), but we retry EOF errors and see more EOFs, so it's unlikely to be a connection reuse-related race. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Given a HTTP server that reads the request, but (cleanly) closes the connection without producing any response:
What did you expect to see?
An error saying something about an unexpectedly closed connection.
What did you see instead?
i.e. the error is
io.EOF, which seems inconsistent with the official definition of that value:Notes
I appreciate that this might not be possible to change due to the compatibility promise.
The immediate cause is
go/src/net/http/transport.go
Line 2092 in 3fcbfb0
io.EOF, it is wrapped ingo/src/net/http/transport.go
Line 2109 in 3fcbfb0
io.EOFagain, with no logic anywhere to turn it into an “this was unexpected” error.The text was updated successfully, but these errors were encountered: