net/http: No easy way of servicing permanent redirects

[hlandau]

When a permanent redirect occurs, calling code should have a way to get notified of this
so that it can update the URL which is specified in some database. For example, a
crawler which receives a permanent redirect should not access the original URL on future
occasions; it should use the new URL specified in the redirect, but only if that
redirect is permanent.

net/http provides no easy way to obtain this information.

CheckRedirect is not suitable since it does not provide access to the Response
indicating the new Location, so the status code of that response cannot be polled to
determine whether it is a permanent redirect.

One way of solving this would be to add a PermanentURL field to Response. This would
contain the URL of the latest consecutive permanent redirect target. In the following
case:

Original URL -[Permanent Redirect To]-> Second URL -[Permanent Redirect To]->
Third URL -[Temporary Redirect To]-> Fourth URL -[Permanent Redirect To]-> Fifth
URL.

PermanentURL would contain the third URL.

This is an imperfect solution because it does not allow Permanent Redirects to be
recorded in situations where the redirect target is inaccessible and results in an error
and not a Response. Though it could be argued that that might be a feature as it causes
potentially erroneous Permanent Redirects to be ignored.

[bradfitz]

Comment 1:

This is already possible if you use the underlying RoundTripper (Transport) directly,
which doesn't have any special handling of redirects.
I'm not sure this needs to be "easy". If you need this capability, you're already
fancier than most people, who just want to fetch a resource.

Labels changed: added release-none, repo-main.

Status changed to Thinking.

[odeke-em]

Related to #7912?

[odeke-em]

/cc @bradfitz on whether you think there is more work we could do here or should we close this issue now that we can follow 308s.

[bradfitz]

(Cleaning up old bugs)

This can be implemented by using https://golang.org/pkg/net/http/#Client.CheckRedirect and checking the https://golang.org/pkg/net/http/#Request.Response StatusCode field.