You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# python3 -m wpull "http://www.techcrunch.com/" -r
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 537.2 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
WARNING Ignoring robots.txt redirect loop.
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 0.0 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO FINISHED.
INFO Time length: 0:00:03.
INFO Downloaded: 0 files, 0.0 B.
INFO Exiting with status 0.
This works fine, though:
# python3 -m wpull "http://www.techcrunch.com/robots.txt"
INFO Fetching ‘http://www.techcrunch.com/robots.txt’.
Requesting http://www.techcrunch.com/robots.txt... 301 Moved Permanently
Length: 178 [text/html]
100.0% [=========================] 178.0 B 0:00:00 637.1 B/s
Bytes received: 178
INFO Fetched ‘http://www.techcrunch.com/robots.txt’: 301 Moved Permanently. Length: 178 [text/html].
INFO Fetching ‘http://techcrunch.com/robots.txt’.
Requesting http://techcrunch.com/robots.txt... 200 OK
Length: None [text/plain; charset=utf-8]
[O ] 5.0 B 0:00:00 15.5 B/s
Bytes received: 867
INFO Fetched ‘http://techcrunch.com/robots.txt’: 200 OK. Length: None [text/plain; charset=utf-8].
INFO FINISHED.
INFO Time length: 0:00:00.
INFO Downloaded: 1 file, 855.0 B.
INFO Exiting with status 0.
The text was updated successfully, but these errors were encountered:
Re: #58
When next_request was called multiple times, the redirect URL was set to
None. This caused it to fetch the same redirecting robots.txt repeatly.
This works fine, though:
The text was updated successfully, but these errors were encountered: