Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: (*http2pipe).closeWithError” Crashed #43965

Open
wgplt opened this issue Jan 28, 2021 · 7 comments
Open

net/http: (*http2pipe).closeWithError” Crashed #43965

wgplt opened this issue Jan 28, 2021 · 7 comments
Milestone

Comments

@wgplt
Copy link

@wgplt wgplt commented Jan 28, 2021

What version of Go are you using (go version)?

$ go version 
1.15.4-1.15.7

Does this issue reproduce with the latest release?

1.15.7

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/test/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/test/go/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-I/home/db2inst2/sqllib/include"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-L/home/db2inst2/sqllib/lib"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build439006933=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I used a common "http.Transport" object for every “http.Client”,bug after running for a while ,the program crashes and the output error message is as follows。

panic: runtime error: invalid memory address or nil pointer dereference
        panic: err must be non-nil
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x6ae0ea]

goroutine 4011 [running]:
net/http.(*http2pipe).closeWithError(0xc00084fe68, 0xc00084fec0, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3558 +0x247
net/http.(*http2pipe).CloseWithError(...)
        /usr/lib/go/src/net/http/h2_bundle.go:3545
net/http.(*http2clientConnReadLoop).cleanup(0xc000a8cfa8)
        /usr/lib/go/src/net/http/h2_bundle.go:8233 +0x294
panic(0x80de40, 0xd4f050)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
net/http.(*http2pipe).Write(0xc000a80028, 0xc000a18300, 0x1c, 0x55, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3537 +0x10a
net/http.(*http2clientConnReadLoop).processData(0xc001199fa8, 0xc000634720, 0xc000634720, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:8706 +0x325
net/http.(*http2clientConnReadLoop).run(0xc001199fa8, 0xc0010f67b0, 0x6ddaa5)
        /usr/lib/go/src/net/http/h2_bundle.go:8286 +0x52a
net/http.(*http2ClientConn).readLoop(0xc000b54780)
        /usr/lib/go/src/net/http/h2_bundle.go:8179 +0x6f
created by net/http.(*http2Transport).newClientConn
        /usr/lib/go/src/net/http/h2_bundle.go:7175 +0x685

my codes like this:

common.HttpTransport = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
MaxIdleConns: 100,
MaxIdleConnsPerHost: 100,
IdleConnTimeout: 60 * time.Second,
DisableKeepAlives: false,

Dial: func(netw, addr string) (net.Conn, error) {
	deadline := time.Now().Add(90 * time.Second)
	c, err := net.DialTimeout(netw, addr, time.Second*45)
	if err != nil {
		return nil, err
	}
	c.SetDeadline(deadline) 
	return c, nil
}, 
ForceAttemptHTTP2:     true,
TLSHandshakeTimeout:   12 * time.Second,
ResponseHeaderTimeout: 12 * time.Second, 
ExpectContinueTimeout: 1 * time.Second,
DisableCompression:    true, 
}

var (
	client      *http.Client
)
client = &http.Client{
	Transport: common.HttpTransport,
}
client.Timeout = time.Duration(50) * time.Second
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
	return false
}
req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
resp, err := client.Do(req)
if err != nil {
	if resp != nil {
		if resp.Body != nil {
			io.Copy(ioutil.Discard, resp.Body) 
			resp.Body.Close()
		}
	}
	return false
}
defer func() {
	io.Copy(ioutil.Discard, resp.Body)
	resp.Body.Close()
}()

What did you expect to see?

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
        panic: err must be non-nil
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x6ae0ea]

goroutine 4011 [running]:
net/http.(*http2pipe).closeWithError(0xc00084fe68, 0xc00084fec0, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3558 +0x247
net/http.(*http2pipe).CloseWithError(...)
        /usr/lib/go/src/net/http/h2_bundle.go:3545
net/http.(*http2clientConnReadLoop).cleanup(0xc000a8cfa8)
        /usr/lib/go/src/net/http/h2_bundle.go:8233 +0x294
panic(0x80de40, 0xd4f050)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
net/http.(*http2pipe).Write(0xc000a80028, 0xc000a18300, 0x1c, 0x55, 0x0, 0x0, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:3537 +0x10a
net/http.(*http2clientConnReadLoop).processData(0xc001199fa8, 0xc000634720, 0xc000634720, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:8706 +0x325
net/http.(*http2clientConnReadLoop).run(0xc001199fa8, 0xc0010f67b0, 0x6ddaa5)
        /usr/lib/go/src/net/http/h2_bundle.go:8286 +0x52a
net/http.(*http2ClientConn).readLoop(0xc000b54780)
        /usr/lib/go/src/net/http/h2_bundle.go:8179 +0x6f
created by net/http.(*http2Transport).newClientConn
        /usr/lib/go/src/net/http/h2_bundle.go:7175 +0x685
@seankhliao seankhliao changed the title HTTP Packet “net/http.(*http2pipe).closeWithError” Crashed net/http: (*http2pipe).closeWithError” Crashed Jan 28, 2021
@bcmills
Copy link
Member

@bcmills bcmills commented Jan 28, 2021

The err must be non-nil panic is on this line:

panic("err must be non-nil")

The question is, where does the panic: runtime error: invalid memory address or nil pointer dereference come from? I think that's the one at `h2_bundle.go:3537':

return p.b.Write(d)

At that point we know that p is non-nil, or else p.mu.Lock() would have panicked at line 3524. So it must be the case that p.b is nil.

The comment for that field says:

b http2pipeBuffer // nil when done reading

That behavior is implemented here:

go/src/net/http/h2_bundle.go

Lines 3507 to 3514 in 2117ea9

if p.err != nil {
if p.readFn != nil {
p.readFn() // e.g. copy trailers
p.readFn = nil // not sticky like p.err
}
p.b = nil
return 0, p.err
}

So the question is: what is calling Read on the *http2pipe, and how is that intended to synchronize with the Write call?

(CC @bradfitz @tombergan @neild @empijei @fraenkel)

@bcmills bcmills added this to the Backlog milestone Jan 28, 2021
@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Jan 28, 2021

@bcmills Why does it matter? There are only 2 places where p.b = nil occurs. In Read(), but the guarantee is p.err is non nil or during closeWithError() which will set p.breakErr non nil.
In either case a Write() will never reach p.b.Write() if either err or breakErr is set.
Unless there is memory corruption, I don't see how this is possible. Or am I missing something more obvious?

@wgplt Have you run your program with race detection enabled?

@wgplt
Copy link
Author

@wgplt wgplt commented Feb 1, 2021

@bcmills Why does it matter? There are only 2 places where p.b = nil occurs. In Read(), but the guarantee is p.err is non nil or during closeWithError() which will set p.breakErr non nil.
In either case a Write() will never reach p.b.Write() if either err or breakErr is set.
Unless there is memory corruption, I don't see how this is possible. Or am I missing something more obvious?

@wgplt Have you run your program with race detection enabled?

I run my program with race detection enabled,sometimes it can reappear, sometimes it can't!

@davecheney
Copy link
Contributor

@davecheney davecheney commented Feb 1, 2021

@wgplt can you provide more details?

@wgplt
Copy link
Author

@wgplt wgplt commented Feb 22, 2021

@wgplt can you provide more details?

@davecheney @fraenkel @bcmills @seankhliao

I found out why the program crashed because the website I requested returned the following response:

`HTTP/1.1 100001
Server: nginx/1.12.2
Date: Mon, 22 Feb 2021 03:27:06 GMT
Content-Type: application/json; charset=utf-8
Connection: close
X-Powered-By: PHP/5.5.38
Set-Cookie: PHPSESSID=e9bthjpj2gu8cgqohhfeo2j8r2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

`
This site does not return a standard HTTP response header and has no response body.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Feb 22, 2021

There has to be more than this. The fact that your panic was in processData means we saw a data frame. There is no way for the http2 stack to process the above as something remotely valid.
I am just trying to understand how the above causes the panic given there was an http2 negotiation and we received enough detail that a DATA frame is coming.

Do you have any more of the actual wire flow?

@wgplt
Copy link
Author

@wgplt wgplt commented Feb 23, 2021

There has to be more than this. The fact that your panic was in processData means we saw a data frame. There is no way for the http2 stack to process the above as something remotely valid.
I am just trying to understand how the above causes the panic given there was an http2 negotiation and we received enough detail that a DATA frame is coming.

Do you have any more of the actual wire flow?

The test code is as follows:

`// test project main.go
package main

import (
"crypto/tls"
"net/http"
)

func main() {
GET("https://xxxx.com/")
}

func GET(URL string) {
var (
client *http.Client
)
client = &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
ForceAttemptHTTP2: true, //The crash caused by this parameter
},
}
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
return
}
client.Do(req)
}
`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants