net/http: readLoopPeekFailLocked: <nil> from client when server sends response immediately
#31259
Comments
logrusorgru
changed the title
logrusorgru
changed the title
bcmills
changed the title
readLoopPeekFailLocked: <nil> from client when server sends response immediately
|
CC @bradfitz |
bcmills
added
the
NeedsInvestigation
Go version
BehaviorRunning the following code with package race
import (
"bytes"
"context"
"net"
h "net/http"
ht "net/http/httptest"
"testing"
"time"
)
func TestDataRace(t *testing.T) {
d := &dialer{content: "HTTP/1.1 502 Bad Gateway\r\n\r\n"}
tr := &h.Transport{
DialContext: d.dial,
}
req := ht.NewRequest(h.MethodGet, "http://example.com", nil)
resp, e := tr.RoundTrip(req)
if e != nil {
t.Error(e)
} else if resp.StatusCode != h.StatusBadGateway {
t.Error("status:", resp.Status)
}
}
type dialer struct {
content string
}
func (d *dialer) dial(ctx context.Context, network,
addr string) (n net.Conn, e error) {
n = newMockConn(d.content)
return
}
type mockConn struct {
read *bytes.Buffer
write *bytes.Buffer
}
func newMockConn(content string) (m *mockConn) {
m = &mockConn{
read: bytes.NewBufferString(content),
write: new(bytes.Buffer),
}
return
}
func (m *mockConn) Read(p []byte) (n int, e error) {
n, e = m.read.Read(p)
return
}
func (m *mockConn) Write(p []byte) (n int, e error) {
n, e = m.write.Write(p)
return
}
func (m *mockConn) Close() (e error) { return }
func (m *mockConn) LocalAddr() (a net.Addr) { return }
func (m *mockConn) RemoteAddr() (a net.Addr) { return }
func (m *mockConn) SetDeadline(t time.Time) (e error) { return }
func (m *mockConn) SetReadDeadline(t time.Time) (e error) {
return
}
func (m *mockConn) SetWriteDeadline(t time.Time) (e error) {
return
}Output when failing |
|
Defining in the previous code type mockConn struct {
read *bytes.Buffer
write *bytes.Buffer
writeOk chan bool
}
func newMockConn(content string) (m *mockConn) {
m = &mockConn{
read: bytes.NewBufferString(content),
write: new(bytes.Buffer),
writeOk: make(chan bool, 0),
}
return
}
func (m *mockConn) Read(p []byte) (n int, e error) {
<-m.writeOk
n, e = m.read.Read(p)
return
}
func (m *mockConn) Write(p []byte) (n int, e error) {
n, e = m.write.Write(p)
m.writeOk <- true
return
} |
|
Maybe related #34066 |
|
I have exactly the same with golang 1.13 and a fake http server like that: |
|
@logrusorgru seeing this happen as well. Can you reproduce it on go 1.11 as well or just 1.12++ ? |
|
@cyx, note that per the release policy, neither Go 1.11 nor 1.12 is supported at this point. (@sebglon's report of a repro using Go 1.13 is helpful, though.) |
|
@cyx, I just have checked out I don't think this issue is a real problem, but anyone is working with HTTP "manually" may run into it. I just use (let's say it's app.go) var isTesting bool
// [...]
// handler function
if isTesting {
time.Sleep(100*time.Microsecond)
}(let's say it's app_test.go) var _ = func() struct{} {
isTesting = true
return struct{}{}
}()To get my tests 100% stable. Because of the 100µs, I think it's about goroutines scheduling. |
|
Thanks! That makes sense. I'm seeing this in production with the clients
primarily being node http clients -- although the interesting thing is that
I only see it on higher throughput environments
|
|
I can report that we're seeing this issue in production at Let's Encrypt, using https://github.com/letsencrypt/boulder/. Specifically, there are some "hasty" HTTP servers out there that will send an HTTP response as soon as the TCP connection completes. When we make multiple concurrent requests to such a server, we get The reason we'd like to see this fixed: In general we try to route all log output through our logging system, and this is the one place in The output is like below (the number of errors varies from run to run, but there's almost always at least one error): Expected behavior: No errors output. Additional detail: This test case also reproduces when I set I've tried to figure out whether hasty responses are allowed or disallowed by RFC 7230. The closest language I can find is:
Which seems to implicitly allow them. At any rate it's not clear that it would be possible to consistently reject hasty responses, since there's an unavoidable network race between writing a request and reading a response. Given that, I think it probably makes sense to make the current behavior (accepting hasty responses) more consistent. |
|
I can confirm that this patch, which starts persistConns with numExpectedResponses = 1, causes the repro code above to run without errors. 0001-Start-persistConns-with-numExpectedResponses-1.patch.txt |
|
move --- net/http/transport.go.bk 2020-04-24 10:06:11.253338231 +0800
+++ net/http/transport.go 2020-04-24 10:07:03.676671394 +0800
@@ -1893,6 +1893,7 @@
alive := true
for alive {
+ rc := <-pc.reqch
pc.readLimit = pc.maxHeaderResponseSize()
_, err := pc.br.Peek(1)
@@ -1904,7 +1905,6 @@
}
pc.mu.Unlock()
- rc := <-pc.reqch
trace := httptrace.ContextClientTrace(rc.req.Context())
var resp *Response
|
|
hi i'm receiving the same error. is there a way I can try this patch out without having to rebuild a fork fo go? |
|
Yep, it's still actual with |
@jsha Thanks for your repro code! As far as I understand, I think the race you mentioned above may also happen for "non-hasty" HTTP servers, in which case However, I can not reproduce the same error by replacing package main
import (
"fmt"
"log"
"net"
"net/http"
"sync"
"time"
)
const port = ":8081"
var wg sync.WaitGroup
func init() {
tr := http.DefaultTransport.(*http.Transport)
tr.DisableKeepAlives = true
}
func main() {
//go hastyHTTPServer()
go normalHTTPServer()
time.Sleep(100 * time.Millisecond)
for i := 0; i < 2; i++ {
wg.Add(1)
go fetchLoop()
}
wg.Wait()
}
func fetchLoop() {
for i := 0; i < 1e3; i++ {
resp, err := http.Get("http://localhost" + port)
if err != nil {
log.Println(err)
continue
}
resp.Body.Close()
}
wg.Done()
}
// hastyHTTPServer runs an HTTP server that sends a response and closes the
// connection as soon as a client makes a TCP connection, rather than waiting
// for the client to actually make an HTTP request.
// A real-world example of this type of server can be seen at 8849777.com
func hastyHTTPServer() {
ln, err := net.Listen("tcp", port)
if err != nil {
log.Fatal(err)
}
for {
conn, err := ln.Accept()
if err != nil {
log.Fatal(err)
}
go func() {
conn.Write([]byte("HTTP/1.0 200 OK\r\n\r\nQuick response!"))
conn.Close()
}()
}
}
func normalHTTPServer() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "HTTP/1.0 200 OK\r\n\r\nQuick response!")
})
log.Fatal(http.ListenAndServe(port, nil))
}UPDATE: Sorry for my misunderstanding! In fact, the following line will block until the HTTP response is available (or until the Client.Timeout is exceeded): Line 2087 in f5bdbf3 Therefore, in a typical HTTP interaction (i.e. the response follows the request), |
|
For those who are interested, I have finally figured out the reason for the TL;DR: An abnormal 204 response, which includes a body, will cause the BackgroundWe have deployed two types of backends behind Caddy (a web server in Go):
But only HTTP requests in the 2nd case (i.e. Caddy <-> Erlang Backend) are encountering the Root CauseThe Erlang Backend is using Cowboy 2.7.0, which unfortunately allows developers to construct a 204 response with a body. And the fact is that our Erlang developer does do this! After correcting these abnormal 204 responses, all NOTE: Cowboy 2.8.0 has fixed this issue (commit). Reproducing Codepackage main
import (
"log"
"net/http"
"time"
)
const port = ":8081"
func main() {
go abnormalHTTPServer()
time.Sleep(100 * time.Millisecond)
fetchLoop()
}
func fetchLoop() {
client := &http.Client{Timeout: 2 * time.Second}
for i := 0; i < 2; i++ {
resp, err := client.Get("http://localhost" + port)
if err != nil {
log.Println(err)
continue
}
resp.Body.Close()
}
}
// abnormalHTTPServer runs an HTTP server that sends a 204 response with a body.
func abnormalHTTPServer() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
hj, ok := w.(http.Hijacker)
if !ok {
panic("impossible for HTTP/1.x")
}
conn, _, err := hj.Hijack()
if err != nil {
panic(err)
}
// Mimic the abnormal 204 response from Cowboy 2.7.0.
conn.Write([]byte("HTTP/1.1 204 No Content\r\n\r\n{}"))
//conn.Close()
})
log.Fatal(http.ListenAndServe(port, nil))
} |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yep. 1.14.1
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
100% reproducible on Linux x64, and play.golang.org. The behavior is unpredictable and the code (see below) to reproduce should be executed few times to got the error and the log.
The error is:
The log is:
What happens?
HTTP server sends response without reading request and without a delay. This approach is good for proxy servers where a client determined by his IP (i.e. using white-list). Thus, we haven't to read request, if the user is not white-listed.
Reproducing
(try few times to got error)
play.golang.org
Also, playing with the N constant I've got this error for N=10, N=100 and N=1000. For N=1000 this error occurs every time. For N=10 rarely. Seems, it depends on goroutines scheduling.
What did you expect to see?
Got this error every time or never.
What did you see instead?
It's unpredictable. That makes testing a bit harder.
Also
Probably related issues #15253, #15446
The text was updated successfully, but these errors were encountered: