HEAD requests should be HEAD requests upon redirect

[haampie]

Bug report

Currently the following is False

from urllib.request import Request, urlopen

len(urlopen(Request("http://google.com", method="HEAD")).read()) == 0  # False

But this is True

len(urlopen(Request("http://www.google.com", method="HEAD")).read()) == 0 # True

This is because http://google.com redirects with 302 to http://www.google.com.

This means that checking for existence of some file by URL will actually download the file when the URL responds with a redirect. This makes no sense. Also the HTTP spec says nothing about changing HEAD requests into GET requests; it just says that everything but GET and HEAD requests should require user interaction on redirect, which Python violates, but there's a comment on that explaining it's an active choice to violate the spec there.

To me it seems like this is an oversight. Note that curl -LI http://google.com also sticks to HEAD requests.

Linked PRs

• gh-99730: HEAD should remain HEAD request on redirect #99731

[bcail]

@haampie is there anything I can do to help with this issue/PR?

[haampie]

I guess it should be compared to other tools. A while after reporting this issue and implementing my own redirect handler, I hit some issue where sending a HEAD request to a server that redirects to an AWS bucket with temporary credentials would only work if the redirect was using a GET request, and error with Unauthorized if it was HEAD. Dunno how common this is.

[bcail]

OK, here's confirmation that the requests package stays with HEAD:

>>> r = requests.head('http://google.com', allow_redirects=True)
>>> r.url
'http://www.google.com/'
>>> len(r.content)
0

And I'm seeing curl stay with the HEAD request as well.

Here's an MDN doc that doesn't come down strongly on one side or the other.

[bcail]

@orsenthil any thoughts on this issue, whether the request handling can be updated to not change a HEAD to a GET on a redirect?

[orsenthil]

@orsenthil any thoughts on this issue, whether the request handling can be updated to not change a HEAD to a GET on a redirect?

The PR #99731 is approved. We will merge this change, and make the behavior change.

[encukou]

Merged; it'll be in 3.13.
Since this is a behaviour change, I don't think backporting to maintenance branches is appropriate.