Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net/http: header fields order #24375

Closed
plambein opened this issue Mar 13, 2018 · 8 comments

Comments

Projects
None yet
6 participants
@plambein
Copy link

commented Mar 13, 2018

With reference to #19292, which was closed because no concrete problem could be defined, and #21853.

We have a transparant http proxy server, used for calculating statistics, not modifying requests.
When proxying http traffic, the http header fields order sent to the destination will always get scrambled (alphabetically sorted). This means any golang proxy server can easily be fingerprinted by looking at the header field order.

Keeping the original order will probably be hard to do? However if it could be defined, people could work around this by defining the default order of the user agent.

@gopherbot gopherbot added this to the Proposal milestone Mar 13, 2018

@gopherbot gopherbot added the Proposal label Mar 13, 2018

@odeke-em odeke-em changed the title proposal: http header fields order proposal: net/http: header fields order Mar 14, 2018

@slrz

This comment has been minimized.

Copy link

commented Mar 14, 2018

What is the actual problem this kind of fingerprinting causes for you? It's unlikely to be prevented by sorting the headers. There is probably enough specific behaviour in Go's HTTP and/or TLS libraries (and every single other implementation, FWIW) to allow for an educated guess regardless of the order of sent HTTP headers.

@plambein

This comment has been minimized.

Copy link
Author

commented Mar 15, 2018

There is indeed many specific behaviour, that comes from default implementations (e.g. keep alive periods, supported cipher suites,...), but these are way more subtle, headers aren't (and most of these can be overwritten, headers can't).

Also, bot-detection services check for header order, to verify if the request came from an actual browser, since major browsers send their headers in a specific order. Thus every browser request passing through the golang proxy gets blocked by such hosted server.

Apart from that, I don't see why a request that has been proxied by golang code should come out with scrambled headers. It completely takes away the transparency of the server.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 2, 2018

The only way to do this, considering a lot of our existing API, would be to overload the net/http.Header type and shove the order in there somewhere.

Since the representation of a Header is just a map[string][]string, that doesn't give us any choice of putting it somewhere nice. We'd need to pick a magic string.

Fortunately we've gone down that ugly-ish path before with net/http.TrailerPrefix:

const TrailerPrefix = "Trailer:"

The godoc doesn't explain why that value is safe, but it's safe because it has a colon in it, and colons aren't valid in header field names.

So we could do something like "Header-Order:".

Assuming we do that, there are two paths to consider: serialization, and parsing.

Serializing is easy. We just modify WriteSubset to respect the order instead of sorting.

For parsing, we might want it to be opt-in to not pay the cost each time & to not confuse code that's not ready for that weirdo key to be present. The question is where to put that opt-in mechanism.

In a Handler, it's too late. A Server option is kinda too broad.

It could be kept always in the internal responseWriter and made available via a special function that takes the ResponseWriter interface and returns it, I suppose.

@plambein

This comment has been minimized.

Copy link
Author

commented Apr 6, 2018

Hi @bradfitz

That solves the problem for sending the response back to the client. However while executing requests, you get stuck on this line, from the Transport's RoundTrip:
!httplex.ValidHeaderFieldName(k)
https://github.com/golang/go/blob/master/src/net/http/transport.go#L367

Should the Transport have an extra option as well?

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 6, 2018

Should the Transport have an extra option as well?

No new options, but yes, the Transports would have to know about this and do the right thing.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 9, 2018

Change https://golang.org/cl/105755 mentions this issue: http: allow headers to be ordered

@RalphCorderoy

This comment has been minimized.

Copy link

commented Apr 9, 2018

The CL refers to RFC 2616. That's made obsolete by RFC 7230 that has a Field Order section: https://tools.ietf.org/html/rfc7230#section-3.2.2 It points out when order can be significant, as well as the separate issue of good practice.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Apr 9, 2018

Getting back to the original question:

Keeping the original order will probably be hard to do?

It seems like it's actually hard. It's invasive to the APIs and we're not sure where to stuff the data. We can put the order into Header["Header-Order:"] or some hack like that, but we can't set that by default, for fear of confusing old code. So everything carrying an http.Header has to also have a new field listing the order, and code that wants to set the order has to copy that into the map. That means http.Request and http.Response both need a new field, modify ReadRequest, ReadResponse, Header.WriteSubset. And any code iterating over the header separately would have to know not to look at this fake header, at least when people have opted in to setting that fake header.

This seems pretty hard for not much benefit. Let's leave this for a future rethink of the HTTP library, which is #5465.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.