net/http/httputil: ReverseProxy closing upgraded tcp connection #70754
Description
Go version
go version go1.23.2 linux/amd64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/m/.cache/go-build'
GOENV='/home/m/.config/go/env'
GOEXE=''
GOEXPERIMENT='nocoverageredesign'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/m/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/m/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/m/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/m/go-proxy-bug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1314557167=/tmp/go-build -gno-record-gcc-switches'What did you do?
Impacted by bug in rancher-desktop: rancher-sandbox/rancher-desktop#3239
Rancher-desktop uses net/http/httputil.ReverseProxy to proxy requests from the docker cli on linux to another WSL container via unix sockets.
The Docker API uses a perhaps non-standard HTTP/1.1 websocket request to stream input/output from the containers (docker documentation: https://docs.docker.com/reference/api/engine/version/v1.47/#tag/Container/operation/ContainerAttach).
The bug is that ReverseProxy is closing the connection when the client closes the outbound socket.
I was able to reproduce it, repo: https://github.com/matejkramny/go-proxy-bug
The long and short of the bug reproduction is:
- A websocket server
- Proxy using
httputil.ReverseProxy - Client that disconnects the writing side of the tcp connection
Demonstrated here:
https://github.com/matejkramny/go-proxy-bug/blob/main/client/client_raw.go#L84
net.TCPConn has a CloseWrite function that simulates what the docker client is doing to the underlying connection
I couldn't find much about the Upgrade: tcp header but it's not complying with the websocket standard (RFC: https://datatracker.ietf.org/doc/html/rfc6455). Maybe what Docker is doing seems like non-standard behaviour and abusing the http protocol in some ways which may be unsupported by the ReverseProxy?
What did you see happen?
ReverseProxy is closing the entire connection when a client closes the output.
The server can still write but the client is unable to receive from the server. The proxy might be closing the client connection by mistake because the client closed one of it's streams.
What did you expect to see?
The proxy should still be able to forward data from the server even when the client closed one of the streams.