Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Allow customization of robots.txt #7884

Merged
merged 5 commits into from Jul 15, 2019

Conversation

@OsamaSayegh
Copy link
Member

commented Jul 11, 2019

This allows admins to customize/override the content of the robots.txt
file at /admin/customize/robots. That page is not linked to anywhere in
the UI -- admins have to manually type the URL to access that page.

Meta topic: https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879?u=osama

Screenshots:

image

image

@eviltrout does it make sense to prepend a comment to robots.txt that says something along the lines of "this robots.txt file has been customized at /admin/customize/robots" if the file is customized? It might help with figuring out why certain things are in the file and how to remove/change them?

OsamaSayegh added 2 commits Jul 11, 2019
This allows admins to customize/override the content of the robots.txt
file at /admin/customize/robots. That page is not linked to anywhere in
the UI -- admins have to manually type the URL to access that page.
@discoursebot

This comment has been minimized.

Copy link

commented Jul 11, 2019

You've signed the CLA, OsamaSayegh. Thank you! This pull request is ready for review.

@discoursebot

This comment has been minimized.

Copy link

commented Jul 11, 2019

This pull request has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/needing-to-edit-robots-txt-file-where-is-it/93879/40

@coding-horror

This comment has been minimized.

Copy link
Member

commented Jul 11, 2019

how does this interact with the handful of existing site settings that modify robots.txt, such as whitelisted crawler user agents, slow down crawler user agents, and allow index in robots txt ?

@OsamaSayegh

This comment has been minimized.

Copy link
Member Author

commented Jul 11, 2019

@coding-horror this allows overriding the whole robots.txt file. So if an admin goes to this page and makes some changes, then that's what's going to be served. Changes to the site settings that you mentioned won't apply if there is an overriding copy in the database (admins can remove the overriding copy and restore the default robots.txt). Is this how you think it should work?

@coding-horror

This comment has been minimized.

Copy link
Member

commented Jul 11, 2019

@OsamaSayegh

This comment has been minimized.

Copy link
Member Author

commented Jul 11, 2019

Yes, there is a "revert changes" button. I will add a copy to the page to make that clear.

Copy link
Member

left a comment

Overall it's looking good but I'd like you to make some changes.

@eviltrout

This comment has been minimized.

Copy link
Member

commented Jul 12, 2019

does it make sense to prepend a comment to robots.txt that says something along the lines of "this robots.txt file has been customized at /admin/customize/robots" if the file is customized?

The problem is that information is only really useful to staff. You could put in logic that only includes that message if the request is via a staff account, but some people will use those with API keys to generate the robots.txt for the root of a subfolder site.

OsamaSayegh added 2 commits Jul 13, 2019
Copy link
Member

left a comment

This looks good now thanks, merge when ready!

@OsamaSayegh OsamaSayegh merged commit 6515ff1 into discourse:master Jul 15, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@OsamaSayegh OsamaSayegh deleted the OsamaSayegh:customize-robots-txt branch Jul 15, 2019
@discoursebot

This comment has been minimized.

Copy link

commented Sep 19, 2019

This pull request has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/discourse-sitemap-plugin/40348/134

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can’t perform that action at this time.