proposal: net/http: header fields order

[plambein]

With reference to #19292, which was closed because no concrete problem could be defined, and #21853.

We have a transparant http proxy server, used for calculating statistics, not modifying requests.
When proxying http traffic, the http header fields order sent to the destination will always get scrambled (alphabetically sorted). This means any golang proxy server can easily be fingerprinted by looking at the header field order.

Keeping the original order will probably be hard to do? However if it could be defined, people could work around this by defining the default order of the user agent.

[slrz]

What is the actual problem this kind of fingerprinting causes for you? It's unlikely to be prevented by sorting the headers. There is probably enough specific behaviour in Go's HTTP and/or TLS libraries (and every single other implementation, FWIW) to allow for an educated guess regardless of the order of sent HTTP headers.

[plambein]

There is indeed many specific behaviour, that comes from default implementations (e.g. keep alive periods, supported cipher suites,...), but these are way more subtle, headers aren't (and most of these can be overwritten, headers can't).

Also, bot-detection services check for header order, to verify if the request came from an actual browser, since major browsers send their headers in a specific order. Thus every browser request passing through the golang proxy gets blocked by such hosted server.

Apart from that, I don't see why a request that has been proxied by golang code should come out with scrambled headers. It completely takes away the transparency of the server.

[bradfitz]

The only way to do this, considering a lot of our existing API, would be to overload the net/http.Header type and shove the order in there somewhere.

Since the representation of a Header is just a map[string][]string, that doesn't give us any choice of putting it somewhere nice. We'd need to pick a magic string.

Fortunately we've gone down that ugly-ish path before with net/http.TrailerPrefix:

const TrailerPrefix = "Trailer:"

The godoc doesn't explain why that value is safe, but it's safe because it has a colon in it, and colons aren't valid in header field names.

So we could do something like "Header-Order:".

Assuming we do that, there are two paths to consider: serialization, and parsing.

Serializing is easy. We just modify WriteSubset to respect the order instead of sorting.

For parsing, we might want it to be opt-in to not pay the cost each time & to not confuse code that's not ready for that weirdo key to be present. The question is where to put that opt-in mechanism.

In a Handler, it's too late. A Server option is kinda too broad.

It could be kept always in the internal responseWriter and made available via a special function that takes the ResponseWriter interface and returns it, I suppose.

[plambein]

Hi @bradfitz

That solves the problem for sending the response back to the client. However while executing requests, you get stuck on this line, from the Transport's RoundTrip:
!httplex.ValidHeaderFieldName(k)
https://github.com/golang/go/blob/master/src/net/http/transport.go#L367

Should the Transport have an extra option as well?

[bradfitz]

Should the Transport have an extra option as well?

No new options, but yes, the Transports would have to know about this and do the right thing.

[gopherbot]

Change https://golang.org/cl/105755 mentions this issue: http: allow headers to be ordered

[RalphCorderoy]

The CL refers to RFC 2616. That's made obsolete by RFC 7230 that has a Field Order section: https://tools.ietf.org/html/rfc7230#section-3.2.2 It points out when order can be significant, as well as the separate issue of good practice.

[rsc]

Getting back to the original question:

Keeping the original order will probably be hard to do?

It seems like it's actually hard. It's invasive to the APIs and we're not sure where to stuff the data. We can put the order into Header["Header-Order:"] or some hack like that, but we can't set that by default, for fear of confusing old code. So everything carrying an http.Header has to also have a new field listing the order, and code that wants to set the order has to copy that into the map. That means http.Request and http.Response both need a new field, modify ReadRequest, ReadResponse, Header.WriteSubset. And any code iterating over the header separately would have to know not to look at this fake header, at least when people have opted in to setting that fake header.

This seems pretty hard for not much benefit. Let's leave this for a future rethink of the HTTP library, which is #5465.