net/http: cannot set custom/special pseudo-headers when using HTTP/2

[jaddr2line]

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jadenw/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/jadenw/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build648789117=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I attempted to implement websockets over HTTP/2, AKA RFC 8441, with net/http.

What did you expect to see?

Some mechanism of setting :protocol as defined in section 5 of RFC 8441.

What did you see instead?

There is no way to specify extra pseudo-headers in a request with net/http, at least that I can find in the header encoding code.

I am really not sure what the best way to solve this would be, or whether it is fully worthwhile. However, I would really like to be able to use a user-provided *http.Client for this, and this is the only thing I have not yet found a workaround for.

Clarification: pseudo-headers do not satisfy httpguts.ValidHeaderFieldName and therefore cannot be passed in with the regular headers

[jaddr2line]

Also, a link to what I have currently of the client side of the HTTP/2 handshake: https://github.com/jadr2ddude/exp/blob/1ed5641ad76ca5761af3a652406bf49b7cac2ce3/ws/handshake.go#L248

[andybons]

@bradfitz @mikioh

[mikedanese]

This is somewhat related to #27244

[nhooyr]

@bradfitz how about we automatically convert HTTP/1.1 Upgrade requests into HTTP/2 if the peer supports it in *http.Transport? We'd have to perform an initial HTTP/2 request on the endpoint to check if it supports RFC 8441, cache that and then from there we can automatically decide whether or not to do HTTP/1.1 or HTTP/2 for subsequent upgrades.

Would mean my WebSocket library basically gets client side HTTP/2 WebSockets for free and httputil.ReverseProxy can proxy RFC 8441 connect requests.

[nhooyr]

For the server side

1. We can just reuse http.Hijacker and return a net.Conn that tries its best to wrap around a HTTP/2 stream, unfortunately we'd have to allocate a buffo.ReadWriter every time but that shouldn't be too big a deal, most apps will want a bufio.ReadWriter when reading/writing to the stream anyway. We could add the Connection and Upgrade headers to RFC 8441 protocol connect requests too so that net/http hijacking servers all get this for free.
2. Then once the above is implemented, we just need to always set the SETTINGS_ENABLE_CONNECT_PROTOCOL HTTP/2 setting parameter to indicate we support RFC 8441

[nhooyr]

There was previous discussion of this at #26937 (comment).

[odeke-em]

@jadr2ddude, thank you for filing this issue! As @nhooyr mentioned, so if you are implementing Websockets I believe the first step that you want to do is to hijack the connection and then if you know you are using HTTP/2, you can use the golang.org/x/net/http2 package to create the framer and send over the respective headers by yourself. IMHO we shouldn't be changing net/http to make that change here. Please feel free to reopen though if I've missed something, thank you!

[mikedanese]

@odeke-em how would you use the framer without having to rewrite or fork most of net/http? This needs both client and server support.

[nhooyr]

You can’t hijack the connection with http/2. So this needs to be reopened.

[odeke-em]

My apologies @jadr2ddude @mikedanese @nhooyr for the brainless close and non-responsiveness for the correction, I posted my response right before a long dinner. Thanks for reopening it, Brad!

[nhooyr]

@bradfitz I'd like to prepare some CLs for this in the coming weeks, could you please review my proposal above?