New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http/httputil: add WebSocket support to ReverseProxy #26937

Open
bradfitz opened this Issue Aug 12, 2018 · 23 comments

Comments

Projects
None yet
4 participants
@bradfitz
Member

bradfitz commented Aug 12, 2018

Add WebSocket support to ReverseProxy.

I have code for this that I use myself in various projects, but I keep forgetting to add it to Go.

@bradfitz bradfitz added the NeedsFix label Aug 12, 2018

@bradfitz bradfitz added this to the Go1.12 milestone Aug 12, 2018

@bradfitz bradfitz self-assigned this Aug 12, 2018

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Aug 12, 2018

Contributor

This would need to be off by default in case current clients rely on the lack of support to prevent websocket connections.

Contributor

nhooyr commented Aug 12, 2018

This would need to be off by default in case current clients rely on the lack of support to prevent websocket connections.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 12, 2018

Member

@nhooyr, that seems like a stretch. We'd document it in the release notes and say how to disable it if people wanted to. But if the two sides negotiate it and the Go user expressed their intent to wire up the two sides with a ReverseProxy, I don't think it's crazy to say we'd wire it up all the way.

Member

bradfitz commented Aug 12, 2018

@nhooyr, that seems like a stretch. We'd document it in the release notes and say how to disable it if people wanted to. But if the two sides negotiate it and the Go user expressed their intent to wire up the two sides with a ReverseProxy, I don't think it's crazy to say we'd wire it up all the way.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Aug 12, 2018

Contributor

@bradfitz Fair enough. Its extremely unlikely anyone would not want WebSockets support anyway.

Contributor

nhooyr commented Aug 12, 2018

@bradfitz Fair enough. Its extremely unlikely anyone would not want WebSockets support anyway.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Aug 12, 2018

Contributor

@bradfitz Could you post the code for this, I've written something similar myself, want to make sure I'm doing it right.

Contributor

nhooyr commented Aug 12, 2018

@bradfitz Could you post the code for this, I've written something similar myself, want to make sure I'm doing it right.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Aug 12, 2018

Contributor

We should also support arbitrary upgrades instead of just WebSockets.

Contributor

nhooyr commented Aug 12, 2018

We should also support arbitrary upgrades instead of just WebSockets.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Aug 12, 2018

Contributor

@bradfitz In fact, if you post it, I'd love to work on the CL for this.

Contributor

nhooyr commented Aug 12, 2018

@bradfitz In fact, if you post it, I'd love to work on the CL for this.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 12, 2018

Member

@nhooyr, I'll send it myself, as I already have the code. It won't save me any time to hand it to you to hand back to me. You can review it once it's on Gerrit, though.

Member

bradfitz commented Aug 12, 2018

@nhooyr, I'll send it myself, as I already have the code. It won't save me any time to hand it to you to hand back to me. You can review it once it's on Gerrit, though.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Aug 24, 2018

Change https://golang.org/cl/131279 mentions this issue: net/http: make Transport return Writable Response.Body on WebSocket upgrade

gopherbot commented Aug 24, 2018

Change https://golang.org/cl/131279 mentions this issue: net/http: make Transport return Writable Response.Body on WebSocket upgrade

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 24, 2018

Member

I decided to implement this a different way, not using my existing code. Instead, I sent https://golang.org/cl/131279 to add Transport support for WebSockets, so the ReverseProxy code won't need to separately dial the backend.

Member

bradfitz commented Aug 24, 2018

I decided to implement this a different way, not using my existing code. Instead, I sent https://golang.org/cl/131279 to add Transport support for WebSockets, so the ReverseProxy code won't need to separately dial the backend.

gopherbot pushed a commit that referenced this issue Aug 25, 2018

net/http: make Transport return Writable Response.Body on protocol sw…
…itch

Updates #26937
Updates #17227

Change-Id: I79865938b05c219e1947822e60e4f52bb2604b70
Reviewed-on: https://go-review.googlesource.com/131279
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 6, 2018

Contributor

Given its not going to use your existing code @bradfitz, can I work on the CL?

Contributor

nhooyr commented Sep 6, 2018

Given its not going to use your existing code @bradfitz, can I work on the CL?

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 7, 2018

Member

@bradfitz, I thought ReverseProxy already supported WebSocket connections as long as FlushInterval is set to a non-zero value. How does your proposed solution compare?

Member

dmitshur commented Sep 7, 2018

@bradfitz, I thought ReverseProxy already supported WebSocket connections as long as FlushInterval is set to a non-zero value. How does your proposed solution compare?

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 7, 2018

Contributor

@dmitshur

  1. Will be faster and more responsive because data will be sent directly instead of waiting on the flush interval.
  2. ReverseProxy does not actually work. It strips the upgrade header in compliance with the RFC and so backend's will not get the upgrade header and not consider the request to be a HTTP upgrade. Furthermore, I don't it would play well with the bidirectional websocket stream. The reverse proxy first writes the entire request and then reads the entire response, it doesn't copy between the two connections concurrently. Additionally, even if it did, I think this could cause a conflict with the HTTP server, it may send the body to the client as a chunked stream as the reverse proxy does not presently hijack it.
Contributor

nhooyr commented Sep 7, 2018

@dmitshur

  1. Will be faster and more responsive because data will be sent directly instead of waiting on the flush interval.
  2. ReverseProxy does not actually work. It strips the upgrade header in compliance with the RFC and so backend's will not get the upgrade header and not consider the request to be a HTTP upgrade. Furthermore, I don't it would play well with the bidirectional websocket stream. The reverse proxy first writes the entire request and then reads the entire response, it doesn't copy between the two connections concurrently. Additionally, even if it did, I think this could cause a conflict with the HTTP server, it may send the body to the client as a chunked stream as the reverse proxy does not presently hijack it.
@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 7, 2018

Member

@nhooyr First point makes sense.

ReverseProxy does not actually work.

This is surprising to me, because I'm using httputil.ReverseProxy to proxy websocket connections (that send data bidirectionally) and I didn't notice any issues. Perhaps there are issues, but not noticeable enough. I don't have the bandwidth to investigate this further for now.

Member

dmitshur commented Sep 7, 2018

@nhooyr First point makes sense.

ReverseProxy does not actually work.

This is surprising to me, because I'm using httputil.ReverseProxy to proxy websocket connections (that send data bidirectionally) and I didn't notice any issues. Perhaps there are issues, but not noticeable enough. I don't have the bandwidth to investigate this further for now.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 7, 2018

Member

I can't imagine how it would be working today.

I'd be surprised. (But probably not pleasantly as it'd probably be some scary accidental bug.)

Member

bradfitz commented Sep 7, 2018

I can't imagine how it would be working today.

I'd be surprised. (But probably not pleasantly as it'd probably be some scary accidental bug.)

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 7, 2018

Contributor

According to my testing this does not work.

I ran a reverse proxy into a gorilla websocket server and made sure to add back the upgrade headers stripped off by the reverse proxy. The handshake succeeded successfully and then I wrote a message to the websocket server but it never got through to the server.

Contributor

nhooyr commented Sep 7, 2018

According to my testing this does not work.

I ran a reverse proxy into a gorilla websocket server and made sure to add back the upgrade headers stripped off by the reverse proxy. The handshake succeeded successfully and then I wrote a message to the websocket server but it never got through to the server.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Sep 8, 2018

Member

Mea culpa. I've figured out where I made a mistake.

I was confused how it was that I had it working, while you were saying it doesn't work. So I tried to reproduce it, and it didn't work. I got WebSocket connection to 'ws://localhost:8090/rp' failed: Error during WebSocket handshake: Unexpected response code: 502.

It turns out I misremembered and never actually sent my websocket connection via a reverse proxy. I had that connecting directly from client to the server.

The thing I did send over a reverse proxy and required a non-zero FlushInterval was a Server-Sent Events connection, not WebSocket. >.< This was the relevant commit. In retrospect, that makes a lot of sense. Sorry about the noise.

Failed Repro Attempt Code
/*
Trying out httputil.ReverseProxy on a WebSocket connection.

Frontend code:

	ws = new WebSocket("ws://localhost:8080/ws");
	ws.onmessage = function(m) { console.log(m); }
	ws.send("hello\n");

	ws = new WebSocket("ws://localhost:8090/rp");
	ws.onmessage = function(m) { console.log(m); }
	ws.send("hello\n");
*/
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httputil"
	"os"
	"time"

	"golang.org/x/net/websocket"
)

func main() {
	errCh := make(chan error)

	// Start a WebSocket server at localhost:8080/ws.
	go func() {
		mux := http.NewServeMux()
		mux.Handle("/ws", websocket.Handler(func(ws *websocket.Conn) {
			go io.Copy(os.Stdout, ws)
			for now := range time.Tick(10 * time.Second) {
				fmt.Fprintln(ws, "tick", now.Unix())
			}
		}))
		errCh <- http.ListenAndServe("localhost:8080", mux)
	}()

	// Start a reverse proxy server at localhost:8090/rp.
	go func() {
		mux := http.NewServeMux()
		rp := &httputil.ReverseProxy{
			Director: func(req *http.Request) {
				req.URL.Host = "localhost:8080"
				req.URL.Path = "/ws"
			},
			FlushInterval: 1 * time.Second,
		}
		mux.Handle("/rp", rp)
		errCh <- http.ListenAndServe("localhost:8090", mux)
	}()

	log.Fatalln(<-errCh)
}
Member

dmitshur commented Sep 8, 2018

Mea culpa. I've figured out where I made a mistake.

I was confused how it was that I had it working, while you were saying it doesn't work. So I tried to reproduce it, and it didn't work. I got WebSocket connection to 'ws://localhost:8090/rp' failed: Error during WebSocket handshake: Unexpected response code: 502.

It turns out I misremembered and never actually sent my websocket connection via a reverse proxy. I had that connecting directly from client to the server.

The thing I did send over a reverse proxy and required a non-zero FlushInterval was a Server-Sent Events connection, not WebSocket. >.< This was the relevant commit. In retrospect, that makes a lot of sense. Sorry about the noise.

Failed Repro Attempt Code
/*
Trying out httputil.ReverseProxy on a WebSocket connection.

Frontend code:

	ws = new WebSocket("ws://localhost:8080/ws");
	ws.onmessage = function(m) { console.log(m); }
	ws.send("hello\n");

	ws = new WebSocket("ws://localhost:8090/rp");
	ws.onmessage = function(m) { console.log(m); }
	ws.send("hello\n");
*/
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httputil"
	"os"
	"time"

	"golang.org/x/net/websocket"
)

func main() {
	errCh := make(chan error)

	// Start a WebSocket server at localhost:8080/ws.
	go func() {
		mux := http.NewServeMux()
		mux.Handle("/ws", websocket.Handler(func(ws *websocket.Conn) {
			go io.Copy(os.Stdout, ws)
			for now := range time.Tick(10 * time.Second) {
				fmt.Fprintln(ws, "tick", now.Unix())
			}
		}))
		errCh <- http.ListenAndServe("localhost:8080", mux)
	}()

	// Start a reverse proxy server at localhost:8090/rp.
	go func() {
		mux := http.NewServeMux()
		rp := &httputil.ReverseProxy{
			Director: func(req *http.Request) {
				req.URL.Host = "localhost:8080"
				req.URL.Path = "/ws"
			},
			FlushInterval: 1 * time.Second,
		}
		mux.Handle("/rp", rp)
		errCh <- http.ListenAndServe("localhost:8090", mux)
	}()

	log.Fatalln(<-errCh)
}
@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 8, 2018

Contributor

@bradfitz so can I work on the CL? Want to avoid duplicate work in case you've already begun.

Contributor

nhooyr commented Sep 8, 2018

@bradfitz so can I work on the CL? Want to avoid duplicate work in case you've already begun.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 9, 2018

Member

Go for it. Just don't add any new API. I think there are existing hooks users can use already.

Member

bradfitz commented Sep 9, 2018

Go for it. Just don't add any new API. I think there are existing hooks users can use already.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 14, 2018

Contributor

@bradfitz re https://golang.org/cl/131279 , what do you think of reusing http.Hijacker for the response body? Would let callers use a *http.Client to do a websocket handshake in general as then they would have access to the raw net.Conn to set deadlines or enable/disable TCP keep alives etc and also be able to reuse the bufio read/writer.

Contributor

nhooyr commented Sep 14, 2018

@bradfitz re https://golang.org/cl/131279 , what do you think of reusing http.Hijacker for the response body? Would let callers use a *http.Client to do a websocket handshake in general as then they would have access to the raw net.Conn to set deadlines or enable/disable TCP keep alives etc and also be able to reuse the bufio read/writer.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 15, 2018

Contributor

Also we need to always use HTTP 1.1 for requests with Connection: Upgrade and a Upgrade header set as we cannot upgrade over HTTP 2.

Contributor

nhooyr commented Sep 15, 2018

Also we need to always use HTTP 1.1 for requests with Connection: Upgrade and a Upgrade header set as we cannot upgrade over HTTP 2.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 15, 2018

Member

Yes, you'd need to use Hijacker.

As for using HTTP/1.1 for outgoing (Transport) Upgrade requests, I'd prefer that logic be in the Transport itself, not in ReverseProxy. You might need to do that CL first.

Member

bradfitz commented Sep 15, 2018

Yes, you'd need to use Hijacker.

As for using HTTP/1.1 for outgoing (Transport) Upgrade requests, I'd prefer that logic be in the Transport itself, not in ReverseProxy. You might need to do that CL first.

@nhooyr

This comment has been minimized.

Show comment
Hide comment
@nhooyr

nhooyr Sep 15, 2018

Contributor

Yes, you'd need to use Hijacker.

Thats for the server, I mean we reuse the Hijacker interface for the client to allow access to the buffered read/writer and the net.Conn directly (setting deadlines, tcp options etc). The response body would implement http.Hijacker.

As for using HTTP/1.1 for outgoing (Transport) Upgrade requests, I'd prefer that logic be in the Transport itself, not in ReverseProxy. You might need to do that CL first.

Got it.

Contributor

nhooyr commented Sep 15, 2018

Yes, you'd need to use Hijacker.

Thats for the server, I mean we reuse the Hijacker interface for the client to allow access to the buffered read/writer and the net.Conn directly (setting deadlines, tcp options etc). The response body would implement http.Hijacker.

As for using HTTP/1.1 for outgoing (Transport) Upgrade requests, I'd prefer that logic be in the Transport itself, not in ReverseProxy. You might need to do that CL first.

Got it.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 16, 2018

Member

Thats for the server, I mean we reuse the Hijacker interface for the client

No, I don't see why we need any new API, as I implied above (#26937 (comment)).

I think TCP keep-alives will take care of cleanup of disappearing connections to the backend and the first failing io.Copy (of ReverseProxy's two client->server and server->client) will tear the whole world down. I think that's sufficient, at least for round one.

Member

bradfitz commented Sep 16, 2018

Thats for the server, I mean we reuse the Hijacker interface for the client

No, I don't see why we need any new API, as I implied above (#26937 (comment)).

I think TCP keep-alives will take care of cleanup of disappearing connections to the backend and the first failing io.Copy (of ReverseProxy's two client->server and server->client) will tear the whole world down. I think that's sufficient, at least for round one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment