x/net/http2: panic: interface conversion: http.http2Frame is *http.http2UnknownFrame, not *http.http2HeadersFrame

[zhangriyueming]

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build000458693=/tmp/go-build -gno-record-gcc-switches"

What did you do?

It is randomly occurred. Sometimes works fine without error, sometimes panics.

Maybe it associated with my Transport.

type Transport struct {
	upstream  http.RoundTripper
	jar       http.CookieJar
	delay     int
	userAgent string
}

func (t *Transport) RoundTrip(r *http.Request) (*http.Response, error) {
	resp, err := t.upstream.RoundTrip(r)
	if err != nil {
		return nil, err
	}

	if some_condition(resp) {
		
		req, err = http.NewRequest("GET", u.String(), nil)
		if err != nil {
			return nil, err
		}

		client := http.Client{
			Transport: t,
			Jar:       t.jar,
			CheckRedirect: func(req *http.Request, via []*http.Request) error {
				return http.ErrUseLastResponse
			},
		}

		resp, err = client.Do(req)
		if err != nil {
			return nil, err
		}

		redirectUrl := resp.Header.Get("Location")
		redirectLocation, err := url.Parse(redirectUrl)
		if err != nil {
			return nil, err
		}
		if redirectLocation.Host == "" {
			redirectUrl = fmt.Sprintf("%s://%s%s",
				resp.Request.URL.Scheme,
				resp.Request.URL.Host,
				redirectUrl)
		}
		
		req, err = http.NewRequest("GET", redirectUrl, nil)
		if err != nil {
			return nil, err
		}

		client = http.Client{
			Transport: t,
			Jar:       t.jar,
		}
		resp, err = client.Do(req)
		return resp, err
	}

	return resp, err
}

What did you expect to see?

Works without panics.

What did you see instead?

panic: 
api_1  | 15:58:21 app         | interface conversion: http.http2Frame is *http.http2UnknownFrame, not *http.http2HeadersFrame
api_1  | 15:58:21 app         | 
api_1  | 15:58:21 app         | 	
api_1  | 15:58:21 app         | panic: 
api_1  | 15:58:21 app         | err must be non-nil
api_1  | 15:58:21 app         | 
api_1  | 15:58:21 app         | 
api_1  | 15:58:21 app         | goroutine 
api_1  | 15:58:21 app         | 32
api_1  | 15:58:21 app         |  [
api_1  | 15:58:21 app         | running
api_1  | 15:58:21 app         | ]:
api_1  | 15:58:21 app         | panic
api_1  | 15:58:21 app         | (
api_1  | 15:58:21 app         | 0x9e0580
api_1  | 15:58:21 app         | , 
api_1  | 15:58:21 app         | 0xbb6070
api_1  | 15:58:21 app         | )
api_1  | 15:58:21 app         | 	
api_1  | 15:58:21 app         | /usr/local/go/src/runtime/panic.go
api_1  | 15:58:21 app         | :
api_1  | 15:58:21 app         | 565
api_1  | 15:58:21 app         |  +
api_1  | 15:58:21 app         | 0x2c5
api_1  | 15:58:21 app         |  fp=
api_1  | 15:58:21 app         | 0xc000407b68
api_1  | 15:58:21 app         |  sp=
api_1  | 15:58:21 app         | 0xc000407ad8
api_1  | 15:58:21 app         |  pc=
api_1  | 15:58:21 app         | 0x42bca5
api_1  | 15:58:21 app         | 
api_1  | 15:58:21 app         | net/http.(*http2pipe).closeWithError
api_1  | 15:58:21 app         | (
api_1  | 15:58:21 app         | 0xc0002d76a8
api_1  | 15:58:21 app         | , 
api_1  | 15:58:21 app         | 0xc0002d76f8
api_1  | 15:58:21 app         | , 
api_1  | 15:58:21 app         | 0x0, 0x0, 0x0)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:3553 +0x1cd fp=0xc000407b90 sp=0xc000407b68 pc=0x67d70d
api_1  | 15:58:21 app         | net/http.(*http2pipe).CloseWithError
api_1  | 15:58:21 app         | (...)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:3540
api_1  | net/http.(*http2clientConnReadLoop).cleanup(0xc000407fb8)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:8084 +0x26c fp=0xc000407c90 sp=0xc000407b90 pc=0x694cac
api_1  | 15:58:21 app         | runtime.call32(0x0, 0xae22d8, 0xc00005a0b0, 0x800000008)
api_1  | 	/usr/local/go/src/runtime/asm_amd64.s:519 +0x3b fp=0xc000407cc0 sp=0xc000407c90 pc=0x45605b
api_1  | panic(0xa18c40, 0xc0005de240)
api_1  | 	/usr/local/go/src/runtime/panic.go:522 +0x1b5 fp=0xc000407d50 sp=0xc000407cc0 pc=0x42bb95
api_1  | 15:58:21 app         | runtime.panicdottypeE
api_1  | 15:58:21 app         | (...)
api_1  | 	/usr/local/go/src/runtime/iface.go:248
api_1  | runtime.panicdottypeI(0xbc7d40, 0xa6c9a0, 0xa20f80)
api_1  | 	/usr/local/go/src/runtime/iface.go:258 +0xf5 fp=0xc000407d78 sp=0xc000407d50 pc=0x409455
api_1  | net/http.(*http2Framer).ReadFrame(0xc00058f420, 0xc0002830e0, 0x0, 0x0, 0x0)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:1759 +0x65d fp=0xc000407e38 sp=0xc000407d78 pc=0x67559d
api_1  | net/http.(*http2clientConnReadLoop).run(0xc000407fb8, 0xae22d8, 0xc0003b3fb8)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:8102 +0x8f fp=0xc000407f70 sp=0xc000407e38 pc=0x694f7f
api_1  | net/http.(*http2ClientConn).readLoop(0xc00006b080)
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:8030 +0x76 fp=0xc000407fd8 sp=0xc000407f70 pc=0x694836
api_1  | runtime.goexit()
api_1  | 	/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000407fe0 sp=0xc000407fd8 pc=0x457bf1
api_1  | created by net/http.(*http2Transport).newClientConn
api_1  | 	/usr/local/go/src/net/http/h2_bundle.go:7093 +0x637

[agnivade]

@zhangriyueming - Can you give us a complete code sample that reproduces this error, and not a snippet of it ?

[zhangriyueming]

@agnivade - Thanks for your response. But I don't have something to "reproduce this error". It's stochastic. The whole program can either works for hundred hours just fine, or throw that panics at just beginning or in one hour.

It might be related to the network environment or something. There was another panic. Located to ProxyFromEnvironment function which in net/http/transport.go. And it says nil pointer or invalid memory. Then I put something like this http.DefaultTransport.(*http.Transport).Proxy = nil to avoid that panic. Since this program doesn't need any proxy.

But now I have no idea why this http.http2Frame is *http.http2UnknownFrame, not *http.http2HeadersFrame happens. I thought maybe there were some problems in the snippet I posted above.

The snippet I posted contains all and only things related to http/transport stuff. Except this, there are just very simple http.NewRequest(), Client.Do(req) things.

[agnivade]

Yes, I understand it is not deterministic. I just wanted the entire program rather than a part of it so it becomes easy for us to debug it. If you cannot share the entire program, perhaps you can cut out just this part from it and make an independent program ?

[zhangriyueming]

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
	"net/http/cookiejar"
	"net/url"
	"strings"
	"time"
)

type Transport struct {
	upstream  http.RoundTripper
	jar       http.CookieJar
	delay     int
	userAgent string
}

func NewTransport(upstream http.RoundTripper) (*Transport, error) {
	jar, err := cookiejar.New(nil)
	if err != nil {
		return nil, err
	}
	userAgent := "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
	return &Transport{
		upstream:  upstream,
		jar:       jar,
		delay:     8,
		userAgent: userAgent}, nil
}

func (t *Transport) RoundTrip(r *http.Request) (*http.Response, error) {
	resp, err := t.upstream.RoundTrip(r)
	if err != nil {
		return nil, err
	}

	if resp.StatusCode == 503 &&
		strings.HasPrefix(resp.Header.Get("Server"), "cloudflare") {

		req, err := http.NewRequest("GET", "https://www.google.co.jp/", nil)
		if err != nil {
			return nil, err
		}

		client := http.Client{
			Transport: t,
			Jar:       t.jar,
			CheckRedirect: func(req *http.Request, via []*http.Request) error {
				return http.ErrUseLastResponse
			},
		}

		resp, err = client.Do(req)
		if err != nil {
			return nil, err
		}

		redirectUrl := resp.Header.Get("Location")
		redirectLocation, err := url.Parse(redirectUrl)
		if err != nil {
			return nil, err
		}
		if redirectLocation.Host == "" {
			redirectUrl = fmt.Sprintf("%s://%s%s",
				resp.Request.URL.Scheme,
				resp.Request.URL.Host,
				redirectUrl)
		}

		req, err = http.NewRequest("GET", redirectUrl, nil)
		if err != nil {
			return nil, err
		}

		client = http.Client{
			Transport: t,
			Jar:       t.jar,
		}
		resp, err = client.Do(req)
		return resp, err
	}

	return resp, err
}

type Job struct {
	baseUrl             string
	transport           *Transport
	client              *http.Client
	clientWithTransport *http.Client
}

type Response struct {
	Key1 string `json:key1`
	Key2 string `json:key2`
	Key3 string `json:key3`
}

func (job *Job) syncPart1(v interface{}) error {
	req, err := http.NewRequest("GET", job.baseUrl+"part1", nil)
	resp, err := job.client.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	decErr := json.NewDecoder(resp.Body).Decode(v)
	if decErr == io.EOF {
		decErr = nil // ignore EOF errors caused by empty response body
	}
	if decErr != nil {
		err = decErr
	}

	return err
}

func (job *Job) syncPart2() ([]byte, error) {
	req, err := http.NewRequest("GET", job.baseUrl+"part2", nil)
	if err != nil {
		return nil, err
	}
	resp, err := job.clientWithTransport.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return nil, err
	}
	return body, nil
}

func (job *Job) Sync() {
	var r Response
	job.syncPart1(&r)
	job.syncPart2()
}

func NewJob() *Job {
	http.DefaultTransport.(*http.Transport).Proxy = nil
	transport, _ := NewTransport(http.DefaultTransport)
	return &Job{
		baseUrl:             "https://www.google.com/",
		transport:           transport,
		client:              &http.Client{},
		clientWithTransport: &http.Client{Transport: transport},
	}
}

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Greeting! %s", r.URL.Path[1:])
}

func main() {

	job := NewJob()

	go func() {
		for {
			job.Sync()
			time.Sleep(20 * time.Second)
		}
	}()

	http.HandleFunc("/", handler)
	http.ListenAndServe(":8080", nil)
}

@agnivade - If this is what you meant.

[zhangriyueming]

@agnivade - This panic is very rarely. It often jumped out after dozens days running. So it makes harder to catch it. And I still think it associated to specific environment. And if we know which situation will cause this error message would be very helpful.

[agnivade]

Thanks.

@bradfitz @fraenkel

[bradfitz]

I don't see how this can happen (the return fr.readMetaFrame(f.(*HeadersFrame) line crashing in the interface type assertion on f).

@zhangriyueming, can you run your program under the race detector? https://blog.golang.org/race-detector

@fraenkel, @tombergan, you see anything I'm missing?

[zhangriyueming]

@bradfitz, I'm looking around an approach to run a reduced or minimal version under the race detector since it will cause 10 times CPU/Memory expensive. At the mean time, I got another panic #32004, I think these may be linked and related.

And the other one I mentioned above, there was a panic nil pointer or invalid memory located to ProxyFromEnvironment function which in net/http/transport.go. Should I create an issue to it either?

[zhangriyueming]

Btw, the program is running in a docker container, if it mights.

[bradfitz]

I don't see how either of these are possible, so I suspect some sort of memory corruption. Please post a full repro and/or run with the race detector and see if you have any data races.