Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reload robots.txt after some appropriate interval #54

Open
kkrugler opened this issue Oct 15, 2017 · 0 comments
Open

Reload robots.txt after some appropriate interval #54

kkrugler opened this issue Oct 15, 2017 · 0 comments
Labels
enhancement

Comments

@kkrugler
Copy link
Member

kkrugler commented Oct 15, 2017 •
edited
Loading

For a robots.txt file that was loaded & parsed successfully, use something like 1 day.

Use different (shorter) retry time for robots.txt that have had problems, e.g. 1 hour.
@kkrugler kkrugler changed the title Use different retry time for robots.txt that have had problems Reload robots.txt after some appropriate interval Oct 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kkrugler