net/http: add switches to accept/send bogus HTTP?

[crhino]

We are writing a reverse proxy that we want to be completely transparent in terms of the HTTP request URI. If a client sends an invalid Request URI, we want to forward it on as is and not modify it in any way.

We have been able to almost achieve this goal using the url.Opaque field, expect for http requests starting with //. In the URL.RequestURI() method here, we see that an explicit check is being made for a prefix of // and the Scheme is appended. This was added for issue #4860.

The issue #10433 mentions a workaround for the specific usecase of //, but using RawPath makes URL.RequestURI() return a url-encoded RequestURI through the use of EscapedPath(). This is also a violation of our goal of transparency.

If there is another workaround that we are not aware, help finding that out would be appreciated, otherwise we would ask whether or not this is a feature the Go developers would support.

Reference issue: cloudfoundry/gorouter#60

@crhino && @shashwathi

[bradfitz]

It's true that in general we don't let you send illegal stuff.

But why can't you send a request with two leading slashes? It would just look like the absolute-path form of http://tools.ietf.org/html/rfc7230#section-5.3 e.g. http://play.golang.org/p/9fU1tChOrU

What are the bytes of the HTTP request you're trying to send?

[crhino]

Here is a playground example of what we are trying to do: http://play.golang.org/p/r5-UXAoBsu

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

[bradfitz]

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

Go has no problem with them. See the snippet I wrote above (http://play.golang.org/p/9fU1tChOrU) for generating one.

[bradfitz]

I guess //£^sdf would be a better example of a path which can't be generated with Go.

[bradfitz]

What I can't quickly find reading cloudfoundry/gorouter#60 is why this matters.

What client is generating GET //foo paths?
What server is requiring //foo requests?

[youngm]

@bradfitz I believe the root problem is that when a client requests GET //foo, as it passes through the go written transparent proxy it is transformed to GET http://foo which as it passes onto the final destination server becomes a 404 since the destination server was expecting //foo or /foo but not http://foo. You can see it @crhino playground example that what the go request handler thinks it got a request for http://foo instead of //foo that was originally sent.

I think you're simple example only deals with generating the request. I think the problem is on the go request handler side not the go client request side.

Does that help?

Mike

[bradfitz]

@youngm, can you answer my two questions, though?

[youngm]

@bradfitz In the case of cloudfoundry/gorouter#60 the client is a Commercial off the shelf web based product performing an xhr request. Because it is a commercial product it is not easily modified unlike a custom build application. The Server is a Java Tomcat Server.

The javacript in this commercial application is obviously not supposed to send //foo requests. But, the desire here is that regardless of this client invalid request we'd like to write a transparent proxy using go http that doesn't munge a request like this.

[bradfitz]

But, the issue here is that regardless of this client invalid request it should be possible to write a transparent proxy using go http that doesn't munge a request like this.

It should certainly be possible to write a proxy in Go, but the answer might be using the net package and not the net/http package. The net/http package has only been getting stricter over time.

Even if we found a nice way for you to send bogus outgoing HTTP requests, I would want to make it opt-in, so we never speak invalid HTTP by default. But then we'd want to do the same thing for the HTTP server.

For instance, I bet there there already a dozen HTTP requests you won't be able to handle with the net/http server because they're rejected before your Handler gets it. I might even add rejecting bogus Request-Line targets now that I'm aware of it.

So really this bug is about deciding whether we add an opt-in mechanism for both client & server to allow bogus stuff.

[youngm]

@bradfitz I think that sums it up correctly. Do you want to weigh in @crhino ? Since has implications for gorouter?