Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net/http: allow customizing proxy CONNECT url #45460

Open
liuxingbaoyu opened this issue Apr 8, 2021 · 15 comments
Open

proposal: net/http: allow customizing proxy CONNECT url #45460

liuxingbaoyu opened this issue Apr 8, 2021 · 15 comments

Comments

@liuxingbaoyu
Copy link

@liuxingbaoyu liuxingbaoyu commented Apr 8, 2021

Update May 5 2021: The current proposed API is in #45460 (comment) - rsc


What version of Go are you using (go version)?

go version go1.16 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\流星暴雨\AppData\Local\go-build
set GOENV=C:\Users\流星暴雨\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=E:\Eternal\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=E:\Eternal\go;
set GOPRIVATE=
set GOPROXY=https://goproxy.io,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ 流星暴雨\AppData\Local\Temp\go-build1194816388=/tmp/go-build -gno-record-gcc-switches

What did you do?

I want to use a proxy to access HTTPS websites and customize the backend IP address

What did you expect to see?

Is there some way to customize the URL of the CONNECT request

What did you see instead?

Both CONNECT request and SSL handshake use cm.targetAddr

go/src/net/http/transport.go

Lines 1667 to 1672 in 2ebe77a

connectReq := &Request{
Method: "CONNECT",
URL: &url.URL{Opaque: cm.targetAddr},
Host: cm.targetAddr,
Header: hdr,
}

go/src/net/http/transport.go

Lines 1725 to 1729 in 2ebe77a

if cm.proxyURL != nil && cm.targetScheme == "https" {
if err := pconn.addTLS(ctx, cm.tlsHost(), trace); err != nil {
return nil, err
}
}

go/src/net/http/transport.go

Lines 1838 to 1846 in 2ebe77a

// tlsHost returns the host name to match against the peer's
// TLS certificate.
func (cm *connectMethod) tlsHost() string {
h := cm.targetAddr
if hasPort(h) {
h = h[:strings.LastIndex(h, ":")]
}
return h
}

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 9, 2021

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

@seankhliao seankhliao closed this Apr 9, 2021
@liuxingbaoyu
Copy link
Author

@liuxingbaoyu liuxingbaoyu commented Apr 9, 2021

@seankhliao Oh no, I'm sorry that I didn't express it clearly. I read the relevant code and did not find a way to customize the CONNECT request. Perhaps this requires a new API.

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 9, 2021

what exactly do you want to do? change the address the of the proxy while keeping the hostname?

@liuxingbaoyu
Copy link
Author

@liuxingbaoyu liuxingbaoyu commented Apr 9, 2021

When an HTTPS request is sent through an HTTP proxy, it will first send a CONNECT request to establish a TCP connection, and then it will start a handshake with the SSL of the website. If a website has multiple resolution addresses, such as using a CDN, you can use the specified backend IP to access the website by specifying the CONNECT URL. I hope there is an API that can customize the CONNECT URL or customize the entire CONNECT request.
Similar to #13290

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 9, 2021

Right, different values for CONNECT xxx (for connecting) and Host: yyy (for TLS)

@seankhliao seankhliao reopened this Apr 9, 2021
@seankhliao seankhliao changed the title net/http: use custom backend IP address in https request with proxy proposal: net/http: allow customizing proxy CONNECT url Apr 9, 2021
@gopherbot gopherbot added this to the Proposal milestone Apr 9, 2021
@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 9, 2021

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Apr 14, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Apr 21, 2021

The code that sets up the CONNECT does:

connectReq := &Request{
	Method: "CONNECT",
	URL:    &url.URL{Opaque: cm.targetAddr},
	Host:   cm.targetAddr,
	Header: hdr,
}

If hdr["Host"] is set, then it sounds like we should use it instead of cm.targetAddr to set up connectReq.Host. That is, add:

if host, ok := hdr["Host"]; ok {
    if len(host) >= 1 {
        connectReq.Host = host[0]
    } else {
        connectReq.Host = ""
    }
}

Does anyone have any objections to this?

@rsc rsc moved this from Incoming to Active in Proposals Apr 21, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Apr 21, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Apr 21, 2021

Sorry but I am confused on what we are trying to solve.

We have a request to some target site, www.target.org, and we are going through some Proxy. The actual request has a URL/host of www.target.org. It really doesn't matter what the host header is. The Request will ignore the Host header and generate its own based, https://github.com/golang/go/blob/master/src/net/http/request.go#L570

A proxy is required to ignore the host header if it receives an absolute uri, https://tools.ietf.org/html/rfc7230#section-5.4

When a proxy receives a request with an absolute-form of
   request-target, the proxy MUST ignore the received Host header field
   (if any) and instead replace it with the host information of the
   request-target.
@rsc
Copy link
Contributor

@rsc rsc commented Apr 28, 2021

@fraenkel, that's for a standard HTTP proxy, not for some custom thing serving CONNECT requests.
Do you object to providing this functionality? It seems unobjectionable to us.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Apr 28, 2021

@rsc sorry my bad.
I missed the fact that we were talking about the ProxyConnectHeaders.
You will also have to reset connectReq.URL.Opaque = "" to have it appear on the CONNECT line.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Apr 28, 2021

Probably easier to rewrite the url since the host is only used of there is no opaque bit in the url.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Apr 28, 2021

Sorry I confused myself again. Ignore what I said. I need to double check my test case which didn't see the host header on the other side.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Apr 28, 2021

I forgot that ReadRequest deletes the host header. So as long as the other side isn't using Go, this should work.

@rsc
Copy link
Contributor

@rsc rsc commented May 5, 2021

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Accept in Proposals May 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Likely Accept
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants