This repository has been archived by the owner on Nov 14, 2019. It is now read-only.
How to enable retry on crawler? #125
Comments
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Some pages may be failed during the crawling, is there any way to enable retry for the crawler? so that i can make sure all pages are downloaded. Meanwhile, is there any way to figure out which pages are not downloaded? There seems no such information in log or es. Thanks a lot.
The text was updated successfully, but these errors were encountered: