net/http: copy headers when following redirects

[rsc]

http changed recently to follow a redirect on POST. However, the followup request does
not include the headers from the original request. This caused a problem in a Selenium
setting because the Accept: app/json header was not propagated. Is there a standard for
which headers should be copied when following a redirect? Should we, uh, follow it?

[bradfitz]

Comment 1:

I see nothing either way in:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3
On one hand, it seems sane to copy all the headers when following a redirect.
On the other hand, what if one of those headers is "Authorization" and it contains
credentials you don't want to pass along to another party?
I don't know the answer.  I'm curious what other implementations do.

[bradfitz]

Comment 2:

Copy/paste of an internal G+ comment I got about this:
-----------
As per these threads, it's not totally specified by HTTP itself:
http://lists.w3.org/Archives/Public/ietf-http-wg/2011AprJun/0489.html
http://lists.w3.org/Archives/Public/ietf-http-wg/2011JulSep/0014.html
See HTML which specifies some part of it:
http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html
"""
First, apply any relevant requirements for redirects (such as showing any appropriate
prompts). Then, redo main step, but using the target of the redirect as the resource to
fetch, rather than the original resource. For HTTP requests, the new request must
include the same headers as the original request, except for headers for which other
requirements are specified (such as the Host header). [HTTP]
"""
As far as Chrome's HTTP stack, we only preserve embedder headers. Here's the algorithm:
* Embedder (Chrome/WebKit) specifies "extra" request headers in the URL request. A URL
request follows HTTP redirects, so since these are tied to the URL request, they are
sent on all HTTP requests.
* All other headers are generated internally during HTTP processing:
  - At the HTTP transaction level (actually, in URLRequestHttpJob), we add some headers, like cookies.
  - At the network transaction level, we generate the Host header and others. This is also the level at which we do authorization. We re-run the authorization algorithm there, which includes querying our auth cache and re-applying a token if applicable.

[gopherbot]

Comment 3 by stephane.travostino:

+1, in a Go crawler I've written recently, I had to create a custom http.Client function
which, upon redirect, kept the "Range" and "User-Agent" headers I set.
Looking at Python's urllib2 implementation, they have a map of "unredirected headers"
such as "Authorization" and other internally generated ones, while headers set by the
user are kept upon redirection.
What if a user sets an Authorization header manually and the URL redirects to a
different domain? 
I think the implementation should keep it, so the idea of having a "custom headers"
array and "unredirected headers" one is the way to go, IMHO.

[rsc]

Comment 4:

Labels changed: added go1.1maybe, removed go1.1.

[gopherbot]

Comment 5 by alberto.garcia.hierro:

I've checked what libcurl does (which I'd consider a pretty solid implementation) and it
forwards any headers but authentication info. From the curl manual:
"When authentication is used, curl only sends its credentials to the initial host. If a
redirect takes curl to a different host, it won't be able to intercept the
user+password. See also --location-trusted on how to change this."
 --location-trusted
              (HTTP/HTTPS) Like -L, --location, but will allow sending the name + password to all hosts that the site may redirect to. This may or may not introduce a security breach if the site redirects you to a site to which you'll send your authentication info (which  is  plain-
              text in the case of HTTP Basic authentication).
I think this is a pretty sensible behavior and I already have a patch which implements
this is the net/http package. By default, all headers but WWW-Authenticate are copied
and the User property in the initial URL is not copied. If the property TrustRedirects
in the request it set, it will copy everything, including the User property of the
initial URL. So, should I just submit this patch as a CL or do you have any objections?

[robpike]

Comment 6:

Labels changed: added go1.2maybe, removed go1.1maybe.

[rsc]

Comment 7:

Issue #5782 has been merged into this issue.

[rsc]

Comment 8:

Labels changed: added feature.

[robpike]

Comment 9:

Not for 1.2.

Labels changed: removed go1.2maybe.

[rsc]

Comment 10:

Labels changed: added go1.3maybe.

[rsc]

Comment 11:

Labels changed: removed feature.

[rsc]

Comment 12:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 13:

Labels changed: added repo-main.

[bradfitz]

Comment 14:

Issue #7801 has been merged into this issue.

[bradfitz]

Comment 15:

Issue #8027 has been merged into this issue.