New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: add switches to accept/send bogus HTTP? #14952

Open
crhino opened this Issue Mar 24, 2016 · 13 comments

Comments

Projects
None yet
5 participants
@crhino

crhino commented Mar 24, 2016

We are writing a reverse proxy that we want to be completely transparent in terms of the HTTP request URI. If a client sends an invalid Request URI, we want to forward it on as is and not modify it in any way.

We have been able to almost achieve this goal using the url.Opaque field, expect for http requests starting with //. In the URL.RequestURI() method here, we see that an explicit check is being made for a prefix of // and the Scheme is appended. This was added for issue #4860.

The issue #10433 mentions a workaround for the specific usecase of //, but using RawPath makes URL.RequestURI() return a url-encoded RequestURI through the use of EscapedPath(). This is also a violation of our goal of transparency.

If there is another workaround that we are not aware, help finding that out would be appreciated, otherwise we would ask whether or not this is a feature the Go developers would support.

Reference issue: cloudfoundry/gorouter#60

@crhino && @shashwathi

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 25, 2016

It's true that in general we don't let you send illegal stuff.

But why can't you send a request with two leading slashes? It would just look like the absolute-path form of http://tools.ietf.org/html/rfc7230#section-5.3 e.g. http://play.golang.org/p/9fU1tChOrU

What are the bytes of the HTTP request you're trying to send?

@crhino

This comment has been minimized.

crhino commented Mar 30, 2016

Here is a playground example of what we are trying to do: http://play.golang.org/p/r5-UXAoBsu

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 30, 2016

My understanding is that a path beginning with "//" is not allowed, according to http://tools.ietf.org/html/rfc3986#section-3.3. However, many clients allow this to be sent, and as such we want to pass it on unmodified.

Go has no problem with them. See the snippet I wrote above (http://play.golang.org/p/9fU1tChOrU) for generating one.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 30, 2016

I guess //£^sdf would be a better example of a path which can't be generated with Go.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 1, 2016

What I can't quickly find reading cloudfoundry/gorouter#60 is why this matters.

What client is generating GET //foo paths?
What server is requiring //foo requests?

@bradfitz bradfitz added this to the Unplanned milestone Apr 1, 2016

@bradfitz bradfitz added the Thinking label Apr 1, 2016

@youngm

This comment has been minimized.

youngm commented Apr 1, 2016

@bradfitz I believe the root problem is that when a client requests GET //foo, as it passes through the go written transparent proxy it is transformed to GET http://foo which as it passes onto the final destination server becomes a 404 since the destination server was expecting //foo or /foo but not http://foo. You can see it @crhino playground example that what the go request handler thinks it got a request for http://foo instead of //foo that was originally sent.

I think you're simple example only deals with generating the request. I think the problem is on the go request handler side not the go client request side.

Does that help?

Mike

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 1, 2016

@youngm, can you answer my two questions, though?

@youngm

This comment has been minimized.

youngm commented Apr 1, 2016

@bradfitz In the case of cloudfoundry/gorouter#60 the client is a Commercial off the shelf web based product performing an xhr request. Because it is a commercial product it is not easily modified unlike a custom build application. The Server is a Java Tomcat Server.

The javacript in this commercial application is obviously not supposed to send //foo requests. But, the desire here is that regardless of this client invalid request we'd like to write a transparent proxy using go http that doesn't munge a request like this.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 1, 2016

But, the issue here is that regardless of this client invalid request it should be possible to write a transparent proxy using go http that doesn't munge a request like this.

It should certainly be possible to write a proxy in Go, but the answer might be using the net package and not the net/http package. The net/http package has only been getting stricter over time.

Even if we found a nice way for you to send bogus outgoing HTTP requests, I would want to make it opt-in, so we never speak invalid HTTP by default. But then we'd want to do the same thing for the HTTP server.

For instance, I bet there there already a dozen HTTP requests you won't be able to handle with the net/http server because they're rejected before your Handler gets it. I might even add rejecting bogus Request-Line targets now that I'm aware of it.

So really this bug is about deciding whether we add an opt-in mechanism for both client & server to allow bogus stuff.

@youngm

This comment has been minimized.

youngm commented Apr 1, 2016

@bradfitz I think that sums it up correctly. Do you want to weigh in @crhino ? Since has implications for gorouter?

@bradfitz bradfitz changed the title from net/url: No way to completely control the Request URI generated for an HTTP request to net/http: add switches to accept/send bogus HTTP? Apr 1, 2016

@oxtoacart

This comment has been minimized.

Contributor

oxtoacart commented May 3, 2016

Where this gets interesting is with custom error pages. We have a project that was already doing some validation of its own and then returning not just a 400 status, but an error page with details (similar to what's suggested in #10123). Now with Go 1.6, the request doesn't even reach the handler infrastructure, so we don't have the ability to return a custom error page. It seems like there are two solutions to this:

  1. Allow user of http.Server to disable certain validation as this ticket suggests
  2. Allow user of http.Server to register custom error handlers for any validations that are performed

1 seems easier to implement initially and more broadly applicable. 2 seems more convenient for the specific case of displaying custom error pages.

@bradfitz What do you think?

cc: @uaalto

@oxtoacart

This comment has been minimized.

Contributor

oxtoacart commented Jul 11, 2016

@bradfitz This is the bug we were just talking about. Basically our main use for this is that we have an HTTP proxy server that mimics an unconfigured Apache server under certain circumstances. In the case of a 400 bad request, Apache has specific HTTP headers that it sends when responding, as well as specific HTML body content. We'd like to be able to continue responding with these.

This may be a slightly different use case than for this original bug, namely we're not interested in being able to handle and process bad requests, we just want to have control over the specifics of the response. However, if it was possible to let these bad requests just fall through to a regular http.Handler, that would allow us to do what we need.

@glasser

This comment has been minimized.

Contributor

glasser commented Dec 16, 2016

What client is generating GET //foo paths?

For what it's worth, plenty of browsers (I just verified this in Chrome, Safari, and curl) will send GET //foo if a human user chooses to type http://host//foo into them. Popular websites seem to vary as to whether or not they ignore extraneous slashes (eg, www.google.com appears to, www.facebook.com appears to not).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment