Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refresh http header is not followed #92

Closed
rbirkby opened this issue Oct 24, 2011 · 10 comments
Closed

Refresh http header is not followed #92

rbirkby opened this issue Oct 24, 2011 · 10 comments
Labels
NeedsCode

Comments

@rbirkby
Copy link

rbirkby commented Oct 24, 2011

Some websites still send the (non-standard) Netscape-era Refresh HTTP header. This header is supported by all browsers (IE, Firefox, Chrome etc)

eg: Webkit's implementation can be found here:
http://trac.webkit.org/browser/trunk/Source/WebCore/platform/network/HTTPParsers.cpp#L106

The websites which send this Refresh HTTP header are not obscure sites which can be ignored. For example, I've received the refresh header from microsoft.com:

*   Trying 207.46.19.254... connected
* Connected to www.microsoft.com (207.46.19.254) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: www.microsoft.com
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: Close
< Pragma: no-cache
< cache-control: no-cache
< Refresh: 0.1
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>
@mikeal
Copy link
Member

mikeal commented Oct 24, 2011

What does the refresh header do?

Why does it not work in request? You should just be able to request({headers:{refresh:0.1}})

@rbirkby
Copy link
Author

rbirkby commented Oct 24, 2011

It's a response header, not a request header. See the transcript.

On 24 Oct 2011, at 18:55, Mikeal Rogers
reply@reply.github.com
wrote:

What does the refresh header do?

Why does it not work in request? You should just be able to request({headers:{refresh:0.1}})

Reply to this email directly or view it on GitHub:
#92 (comment)

@mikeal
Copy link
Member

mikeal commented Oct 24, 2011

so, refresh === Connection: close?

if that is the case then it'll need to be fixed in core.

@rbirkby
Copy link
Author

rbirkby commented Oct 24, 2011

I see it as similar to a 302 redirect. However, this only really makes sense for a Refresh with a time of 0. In the transcript above, should request pause for 0.1s before redirect/refreshing? That would seem a bit crazy.

@katiecrain
Copy link

i'm having this issue as well.

i'm trying to crawl a site that has an interim loading page, the response header has Refresh that is not being followed. i've looked at the code, and i'm considering a work around for this, but would like to know if any attempt to fix this is in the works.

here's the debug infoz.

DEBUG: 200 http://www.mysite.com/blarg (response 44716)
DEBUG: | Date: Tue, 08 Nov 2011 19:32:11 GMT
DEBUG: | Server: Apache
DEBUG: | Vary: *
DEBUG: | Cache-control: max-age=86400
DEBUG: | Expires: Wed, 09 Nov 2011 19:32:11 GMT
DEBUG: | Refresh: 70; url=/finalresultspage
DEBUG: | Set-cookie: wosid=5KA7AdtwpyYFxtMtnDLtr0; version="1"; path=/WebObjects/mysite.woa,woinst=427; version="1"; path=/WebObjects/mysite.woa
DEBUG: | Content-length: 37900
DEBUG: | Content-type: text/html
DEBUG: | Connection: close

@pedrofaustino
Copy link

@mikeal when you said "needs to be fixed in core" do you mean http?

@spollack
Copy link
Contributor

any news on this one? handling of meta-refresh as a redirect within request would be fantastic.

@mikeal
Copy link
Member

mikeal commented Aug 27, 2014

Is this still an issue?

This is so old I'm closing, if it is actually still an issue just let me know and I'll re-open.

@mikeal mikeal closed this as completed Aug 27, 2014
@polomoshnov
Copy link

polomoshnov commented Jun 27, 2016 •
edited
Loading

I've just stumbled upon this issue myself. Some info on this: https://en.wikipedia.org/w/index.php?title=URL_redirection&oldid=723633886#Refresh_Meta_tag_and_HTTP_refresh_header

@ThatAnton ThatAnton mentioned this issue Jul 23, 2017
Closed
@teebu
Copy link

teebu commented Sep 21, 2017

has any progress been made to this?

@bagder bagder mentioned this issue Mar 10, 2019
Closed
@dkarvounaris dkarvounaris mentioned this issue Mar 15, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsCode
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mikeal @rbirkby @spollack @polomoshnov @pedrofaustino @katiecrain @teebu