HEAD requests can behave oddly... if you don't explicitly tell the server it can close the connection after serving the request, it will sometimes leave the connection open, and eventually the client or server will time out. You can test this directly, with curl: curl -v -XHEAD 'http://download.mozilla.org/?lang=sw&product=firefox-16.0b6&os=osx'; -o /dev/null (hangs for a while after receiving the headers) vs curl -v -XHEAD -H 'Connection: close' 'http://download.mozilla.org/?lang=sw&product=firefox-16.0b6&os=osx'; -o /dev/null (exits immediately after receiving the headers) In this case, I think we definitely want to be creating new connections for each request (not using keep-alives), so this is actually the proper way to do it in the first place. The RFC indicates that you should always send this header, unless you want to make additional requests on the same connection. We don't, so we should send it.
requests, don't automatically traverse the first level of possible redirects, but catch it and then follow it. This way, if what we are being redirected to fails, that's what we'll report on. Otherwise, this obscures what URL is failing/timing-out