Read timed out

[Kavan72]

full traceback:- https://pastebin.com/1c61vqAS

is there any way to handle this error in iter obj ?

[wumpus]

The code already retries 10 times for 30 seconds each.

Common Crawl's index server has been having problems with someone who's fetching data from thousands of AWS EC2 nodes. That person is making the CC server more unreliable than usual.

If you'd like to restart your computation roughly where it went off, you can use timestamps to do that.

If you'd like to make sure your code doesn't get one of these errors in the middle of the iteration, you can raise the maximum in the line if connect_errors > 10 in myrequests.py

[Kavan72]

okay got it. can you give support to increase this on itor with kwargs ?

[wumpus]

Yeah, I recall thinking about that potentially being a problem when I wrote the code :-) really I should raise the limits until it always works for the typical cdx server -- I just need to convince it to complain visibly after enough failures (i.e. --source hostname that resolves but no one's there) without making the test suite run longer (it's already a little annoying.)

[wumpus]

I just pushed a new version which tries 10x more times to connect, while failing fast if the hostname DNS fails. That ought to be enough to battle through CC's occasional problems.

[Kavan72]

thanks