Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom robots.txt #338

Closed
melmothx opened this issue Apr 16, 2020 · 1 comment
Closed

Custom robots.txt #338

melmothx opened this issue Apr 16, 2020 · 1 comment

Comments

@melmothx
Copy link
Owner

Received request via IRC:

how do i change robots.txt? in particular i want to set a crawl delay to try to deal with some high load from some bots.

Currently no mechanism is in place but I think we could look into the site_files directory and if found, use that. Alternatively, set in the DB, unclear which is better.
melmothx added a commit that referenced this issue Apr 18, 2020
@melmothx
Implement robots.txt override from database
e87cc9b 
Address #338
@melmothx
Copy link
Owner Author

Closed in e87cc9b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@melmothx