Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: `readLoopPeekFailLocked: <nil>` from client when server sends response immediately #31259

Open
logrusorgru opened this issue Apr 4, 2019 · 4 comments
Milestone

Comments

@logrusorgru
Copy link

@logrusorgru logrusorgru commented Apr 4, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

Yep. 1.12.1

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GOPROXY=""
GORACE=""
GOROOT="/opt/go"
GOTMPDIR=""
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build705707104=/tmp/go-build -gno-record-gcc-switches"

What did you do?

100% reproducible on Linux X64, and play.golang.org. The behavior is unpredictable and the code (see below) to reproduce should be executed few times to got the error and the log.

The error is:

Get http://127.0.0.1:9101/path: readLoopPeekFailLocked: <nil>

The log is:

transport.go:1829: Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 502 Bad Gateway\r\n\r\n"; err=<nil>

What happens?

HTTP server sends response without reading request and without a delay. This approach is good for proxy servers where a client determined by his IP (i.e. using white-list). Thus, we haven't to read request, if the user is not white-listed.

Reproducing

(try few times to got error)

play.golang.org

package main

import (
	"fmt"
	"log"
	"net"
	"net/http"
	"net/url"
	"sync"
	"time"
)

func mustParseURL(uri string) *url.URL {
	x, err := url.Parse(uri)
	if err != nil {
		panic(err)
	}
	return x
}

func main() {

	l, err := net.Listen("tcp", "127.0.0.1:9100")
	if err != nil {
		log.Fatal(err)
	}
	go func(l net.Listener) {
		for {
			c, err := l.Accept()
			if err != nil {
				log.Fatal(err)
			}
			go func(c net.Conn) {
				defer c.Close()
				fmt.Fprint(c, "HTTP/1.1 502 Bad Gateway\r\n\r\n")
			}(c)
		}
	}(l)
	time.Sleep(100 * time.Millisecond)

	var cl http.Client
	cl.Transport = &http.Transport{
		Proxy:           http.ProxyURL(mustParseURL("http://127.0.0.1:9100/")),
		MaxConnsPerHost: 100,
	}

	const N = 100

	var wg sync.WaitGroup
	wg.Add(N)
	for i := 0; i < N; i++ {
		go func() {
			defer wg.Done()
			resp, err := cl.Get("http://127.0.0.1:9101/path")
			if err != nil {
				log.Print("error:", err)
				return
			}
			if resp.StatusCode != 502 {
				log.Println("status:", resp.StatusCode)
			}
			resp.Body.Close()
		}()
	}

	wg.Wait()

}

Also, playing with the N constant I've got this error for N=10, N=100 and N=1000. For N=1000 this error occurs every time. For N=10 rarely. Seems, it depends on goroutines scheduling.

What did you expect to see?

Got this error every time or never.

What did you see instead?

It's unpredictable. That makes testing a bit harder.

Also

Probably related issues #15253, #15446

@logrusorgru logrusorgru changed the title readLoopPeekFailLocked: <nil> or unpredictable behaviour of http client wherer server sends response immediately readLoopPeekFailLocked: <nil> or unpredictable behaviour of http client where server sends response immediately Apr 4, 2019
@logrusorgru logrusorgru changed the title readLoopPeekFailLocked: <nil> or unpredictable behaviour of http client where server sends response immediately readLoopPeekFailLocked: <nil> or, other words, unpredictable behaviour of http client where server sends response immediately Apr 4, 2019
@bcmills bcmills changed the title readLoopPeekFailLocked: <nil> or, other words, unpredictable behaviour of http client where server sends response immediately net/http: `readLoopPeekFailLocked: <nil>` from client when server sends response immediately Apr 12, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Apr 12, 2019

@bcmills bcmills added this to the Go1.13 milestone Apr 12, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@lamg

This comment has been minimized.

Copy link

@lamg lamg commented Oct 11, 2019

Go version

go version go1.13 linux/amd64

Behavior

Running the following code with go test -race fails most of the times, but not always. I've never seen it fail with go test.

package race

import (
	"bytes"
	"context"
	"net"
	h "net/http"
	ht "net/http/httptest"
	"testing"
	"time"
)

func TestDataRace(t *testing.T) {
	d := &dialer{content: "HTTP/1.1 502 Bad Gateway\r\n\r\n"}
	tr := &h.Transport{
		DialContext: d.dial,
	}
	req := ht.NewRequest(h.MethodGet, "http://example.com", nil)
	resp, e := tr.RoundTrip(req)
	if e != nil {
		t.Error(e)
	} else if resp.StatusCode != h.StatusBadGateway {
		t.Error("status:", resp.Status)
	}
}

type dialer struct {
	content string
}

func (d *dialer) dial(ctx context.Context, network,
	addr string) (n net.Conn, e error) {
	n = newMockConn(d.content)
	return
}

type mockConn struct {
	read  *bytes.Buffer
	write *bytes.Buffer
}

func newMockConn(content string) (m *mockConn) {
	m = &mockConn{
		read:  bytes.NewBufferString(content),
		write: new(bytes.Buffer),
	}
	return
}

func (m *mockConn) Read(p []byte) (n int, e error) {
	n, e = m.read.Read(p)
	return
}

func (m *mockConn) Write(p []byte) (n int, e error) {
	n, e = m.write.Write(p)
	return
}

func (m *mockConn) Close() (e error) { return }

func (m *mockConn) LocalAddr() (a net.Addr) { return }

func (m *mockConn) RemoteAddr() (a net.Addr) { return }

func (m *mockConn) SetDeadline(t time.Time) (e error) { return }

func (m *mockConn) SetReadDeadline(t time.Time) (e error) {
	return
}

func (m *mockConn) SetWriteDeadline(t time.Time) (e error) {
	return
}

Output when failing

2019/10/11 17:19:50 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 502 Bad Gateway\r\n\r\n"; err=<nil>
--- FAIL: TestDataRace (0.00s)
    race_test.go:21: readLoopPeekFailLocked: <nil>
FAIL
exit status 1
@lamg

This comment has been minimized.

Copy link

@lamg lamg commented Oct 11, 2019

Defining in the previous code mockConn, newMockConn, mockConn.Read and mockConn.Write the following way, the error never appeared so far, with go test or go test -race:

type mockConn struct {
	read    *bytes.Buffer
	write   *bytes.Buffer
	writeOk chan bool
}

func newMockConn(content string) (m *mockConn) {
	m = &mockConn{
		read:    bytes.NewBufferString(content),
		write:   new(bytes.Buffer),
		writeOk: make(chan bool, 0),
	}
	return
}

func (m *mockConn) Read(p []byte) (n int, e error) {
	<-m.writeOk
	n, e = m.read.Read(p)
	return
}

func (m *mockConn) Write(p []byte) (n int, e error) {
	n, e = m.write.Write(p)
	m.writeOk <- true
	return
}
@cuonglm

This comment has been minimized.

Copy link
Contributor

@cuonglm cuonglm commented Oct 12, 2019

Maybe related #34066

lamg added a commit to lamg/proxy that referenced this issue Oct 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.