You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently users cannot configure a delay between crawl requests. A simple x seconds delay configurable from the yaml file would allow users to slow down crawls.
This is similar to #46 (abiding by Crawl-delay in robots.txt). However, a general delay would be useful for users who can't or don't want to change the robots.txt.
When both are implemented, the highest value of the two should be used for the crawl delay.
Problem Description
Currently users cannot configure a delay between crawl requests. A simple x seconds delay configurable from the yaml file would allow users to slow down crawls.
This is similar to #46 (abiding by
Crawl-delay
in robots.txt). However, a general delay would be useful for users who can't or don't want to change the robots.txt.When both are implemented, the highest value of the two should be used for the crawl delay.
Proposed Solution
Add a configurable crawl delay in yaml file.
Additional Context
Link to discuss forum: https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/4
The text was updated successfully, but these errors were encountered: