net/http: (*http2pipe).closeWithError” Crashed

[wgplt]

What version of Go are you using (go version)?

$ go version 
1.15.4-1.15.7

Does this issue reproduce with the latest release?

1.15.7

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/test/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/test/go/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-I/home/db2inst2/sqllib/include"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-L/home/db2inst2/sqllib/lib"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build439006933=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I used a common "http.Transport" object for every “http.Client”，bug after running for a while ，the program crashes and the output error message is as follows。

panic: runtime error: invalid memory address or nil pointer dereference
        panic: err must be non-nil
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x6ae0ea]

goroutine 4011 [running]:
net/http.(*http2pipe).closeWithError(0xc00084fe68, 0xc00084fec0, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3558 +0x247
net/http.(*http2pipe).CloseWithError(...)
        /usr/lib/go/src/net/http/h2_bundle.go:3545
net/http.(*http2clientConnReadLoop).cleanup(0xc000a8cfa8)
        /usr/lib/go/src/net/http/h2_bundle.go:8233 +0x294
panic(0x80de40, 0xd4f050)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
net/http.(*http2pipe).Write(0xc000a80028, 0xc000a18300, 0x1c, 0x55, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3537 +0x10a
net/http.(*http2clientConnReadLoop).processData(0xc001199fa8, 0xc000634720, 0xc000634720, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:8706 +0x325
net/http.(*http2clientConnReadLoop).run(0xc001199fa8, 0xc0010f67b0, 0x6ddaa5)
        /usr/lib/go/src/net/http/h2_bundle.go:8286 +0x52a
net/http.(*http2ClientConn).readLoop(0xc000b54780)
        /usr/lib/go/src/net/http/h2_bundle.go:8179 +0x6f
created by net/http.(*http2Transport).newClientConn
        /usr/lib/go/src/net/http/h2_bundle.go:7175 +0x685

my codes like this:

common.HttpTransport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
MaxIdleConns: 100,
MaxIdleConnsPerHost: 100,
IdleConnTimeout: 60 * time.Second,
DisableKeepAlives: false,

Dial: func(netw, addr string) (net.Conn, error) {
	deadline := time.Now().Add(90 * time.Second)
	c, err := net.DialTimeout(netw, addr, time.Second*45)
	if err != nil {
		return nil, err
	}
	c.SetDeadline(deadline) 
	return c, nil
}, 
ForceAttemptHTTP2:     true,
TLSHandshakeTimeout:   12 * time.Second,
ResponseHeaderTimeout: 12 * time.Second, 
ExpectContinueTimeout: 1 * time.Second,
DisableCompression:    true, 
}

var (
	client      *http.Client
)
client = &http.Client{
	Transport: common.HttpTransport,
}
client.Timeout = time.Duration(50) * time.Second
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
	return false
}
req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
resp, err := client.Do(req)
if err != nil {
	if resp != nil {
		if resp.Body != nil {
			io.Copy(ioutil.Discard, resp.Body) 
			resp.Body.Close()
		}
	}
	return false
}
defer func() {
	io.Copy(ioutil.Discard, resp.Body)
	resp.Body.Close()
}()

What did you expect to see?

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
        panic: err must be non-nil
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x6ae0ea]

goroutine 4011 [running]:
net/http.(*http2pipe).closeWithError(0xc00084fe68, 0xc00084fec0, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3558 +0x247
net/http.(*http2pipe).CloseWithError(...)
        /usr/lib/go/src/net/http/h2_bundle.go:3545
net/http.(*http2clientConnReadLoop).cleanup(0xc000a8cfa8)
        /usr/lib/go/src/net/http/h2_bundle.go:8233 +0x294
panic(0x80de40, 0xd4f050)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
net/http.(*http2pipe).Write(0xc000a80028, 0xc000a18300, 0x1c, 0x55, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3537 +0x10a
net/http.(*http2clientConnReadLoop).processData(0xc001199fa8, 0xc000634720, 0xc000634720, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:8706 +0x325
net/http.(*http2clientConnReadLoop).run(0xc001199fa8, 0xc0010f67b0, 0x6ddaa5)
        /usr/lib/go/src/net/http/h2_bundle.go:8286 +0x52a
net/http.(*http2ClientConn).readLoop(0xc000b54780)
        /usr/lib/go/src/net/http/h2_bundle.go:8179 +0x6f
created by net/http.(*http2Transport).newClientConn
        /usr/lib/go/src/net/http/h2_bundle.go:7175 +0x685

[bcmills]

The err must be non-nil panic is on this line:

go/src/net/http/h2_bundle.go

Line 3558 in 2117ea9

panic("err must be non-nil")

The question is, where does the panic: runtime error: invalid memory address or nil pointer dereference come from? I think that's the one at `h2_bundle.go:3537':

go/src/net/http/h2_bundle.go

Line 3537 in 2117ea9

return p.b.Write(d)

At that point we know that p is non-nil, or else p.mu.Lock() would have panicked at line 3524. So it must be the case that p.b is nil.

The comment for that field says:

go/src/net/http/h2_bundle.go

Line 3469 in 2117ea9

b http2pipeBuffer // nil when done reading

That behavior is implemented here:

go/src/net/http/h2_bundle.go

Lines 3507 to 3514 in 2117ea9

	if p.err != nil {
	if p.readFn != nil {
	p.readFn() // e.g. copy trailers
	p.readFn = nil // not sticky like p.err
	}
	p.b = nil
	return 0, p.err
	}

So the question is: what is calling Read on the *http2pipe, and how is that intended to synchronize with the Write call?

(CC @bradfitz @tombergan @neild @empijei @fraenkel)

[fraenkel]

@bcmills Why does it matter? There are only 2 places where p.b = nil occurs. In Read(), but the guarantee is p.err is non nil or during closeWithError() which will set p.breakErr non nil.
In either case a Write() will never reach p.b.Write() if either err or breakErr is set.
Unless there is memory corruption, I don't see how this is possible. Or am I missing something more obvious?

@wgplt Have you run your program with race detection enabled?

[wgplt]

@bcmills Why does it matter? There are only 2 places where p.b = nil occurs. In Read(), but the guarantee is p.err is non nil or during closeWithError() which will set p.breakErr non nil.
In either case a Write() will never reach p.b.Write() if either err or breakErr is set.
Unless there is memory corruption, I don't see how this is possible. Or am I missing something more obvious?

@wgplt Have you run your program with race detection enabled?

I run my program with race detection enabled，sometimes it can reappear, sometimes it can't!

[davecheney]

@wgplt can you provide more details?

[wgplt]

@wgplt can you provide more details?

@davecheney @fraenkel @bcmills @seankhliao

I found out why the program crashed because the website I requested returned the following response:

`HTTP/1.1 100001
Server: nginx/1.12.2
Date: Mon, 22 Feb 2021 03:27:06 GMT
Content-Type: application/json; charset=utf-8
Connection: close
X-Powered-By: PHP/5.5.38
Set-Cookie: PHPSESSID=e9bthjpj2gu8cgqohhfeo2j8r2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

`
This site does not return a standard HTTP response header and has no response body.

[fraenkel]

There has to be more than this. The fact that your panic was in processData means we saw a data frame. There is no way for the http2 stack to process the above as something remotely valid.
I am just trying to understand how the above causes the panic given there was an http2 negotiation and we received enough detail that a DATA frame is coming.

Do you have any more of the actual wire flow?

[wgplt]

There has to be more than this. The fact that your panic was in processData means we saw a data frame. There is no way for the http2 stack to process the above as something remotely valid.
I am just trying to understand how the above causes the panic given there was an http2 negotiation and we received enough detail that a DATA frame is coming.

Do you have any more of the actual wire flow?

The test code is as follows：

`// test project main.go
package main

import (
"crypto/tls"
"net/http"
)

func main() {
GET("https://xxxx.com/")
}

func GET(URL string) {
var (
client *http.Client
)
client = &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
ForceAttemptHTTP2: true, //The crash caused by this parameter
},
}
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
return
}
client.Do(req)
}
`