net/http: cannot set custom/special pseudo-headers when using HTTP/2

[niaow]

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jadenw/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/jadenw/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build648789117=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I attempted to implement websockets over HTTP/2, AKA RFC 8441, with net/http.

What did you expect to see?

Some mechanism of setting :protocol as defined in section 5 of RFC 8441.

What did you see instead?

There is no way to specify extra pseudo-headers in a request with net/http, at least that I can find in the header encoding code.

I am really not sure what the best way to solve this would be, or whether it is fully worthwhile. However, I would really like to be able to use a user-provided *http.Client for this, and this is the only thing I have not yet found a workaround for.

Clarification: pseudo-headers do not satisfy httpguts.ValidHeaderFieldName and therefore cannot be passed in with the regular headers

[niaow]

Also, a link to what I have currently of the client side of the HTTP/2 handshake: https://github.com/jadr2ddude/exp/blob/1ed5641ad76ca5761af3a652406bf49b7cac2ce3/ws/handshake.go#L248

[andybons]

@bradfitz @mikioh

[mikedanese]

This is somewhat related to #27244

[nhooyr]

@bradfitz how about we automatically convert HTTP/1.1 Upgrade requests into HTTP/2 if the peer supports it in *http.Transport? We'd have to perform an initial HTTP/2 request on the endpoint to check if it supports RFC 8441, cache that and then from there we can automatically decide whether or not to do HTTP/1.1 or HTTP/2 for subsequent upgrades.

Would mean my WebSocket library basically gets client side HTTP/2 WebSockets for free and httputil.ReverseProxy can proxy RFC 8441 connect requests.

[nhooyr]

For the server side

1. We can just reuse http.Hijacker and return a net.Conn that tries its best to wrap around a HTTP/2 stream, unfortunately we'd have to allocate a buffo.ReadWriter every time but that shouldn't be too big a deal, most apps will want a bufio.ReadWriter when reading/writing to the stream anyway. We could add the Connection and Upgrade headers to RFC 8441 protocol connect requests too so that net/http hijacking servers all get this for free.
2. Then once the above is implemented, we just need to always set the SETTINGS_ENABLE_CONNECT_PROTOCOL HTTP/2 setting parameter to indicate we support RFC 8441

[nhooyr]

There was previous discussion of this at #26937 (comment).

[odeke-em]

@jadr2ddude, thank you for filing this issue! As @nhooyr mentioned, so if you are implementing Websockets I believe the first step that you want to do is to hijack the connection and then if you know you are using HTTP/2, you can use the golang.org/x/net/http2 package to create the framer and send over the respective headers by yourself. IMHO we shouldn't be changing net/http to make that change here. Please feel free to reopen though if I've missed something, thank you!

[mikedanese]

@odeke-em how would you use the framer without having to rewrite or fork most of net/http? This needs both client and server support.

[nhooyr]

You can’t hijack the connection with http/2. So this needs to be reopened.

[odeke-em]

My apologies @jadr2ddude @mikedanese @nhooyr for the brainless close and non-responsiveness for the correction, I posted my response right before a long dinner. Thanks for reopening it, Brad!

[nhooyr]

@bradfitz I'd like to prepare some CLs for this in the coming weeks, could you please review my proposal above?

[nhooyr]

With @bradfitz leaving Google, who is going to be maintaining net/http? Who do I have to convince re my proposal? @andybons

[andybons]

@fraenkel have any thoughts about this?

[fraenkel]

Currently I see a couple of outstanding issues. Some we can fix, one I am not so sure about.
We currently have no way of configuring the http2 transport with any additional details which is starting to become more of an issue as we add specific http2 features that may need to be configured for a client, e.g., #31643.

The part I don't see how to handle is hijacking the http2 stream on the client side. Things should work correctly on the server side using the request.Body and ResponseWriter.