Create dynamic robots.txt page #5794
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pr-triage
bot
added
PR: reviewed-approved
pr-triage
bot
added
PR: merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Description
This PR moves
robots.txtfrom the totally static public folder to an action where code and env vars are available.This is needed to point robots to the new place for our sitemap download and enable any further dynamic elements in the sitemap.
I also took the moment to add a few new
Disallow:lines to avoid crawling of a few unnecessary pages which either result in errors or result in an endless list of endpoints which don't need indexing.If you're reviewing this, just double check the format of the new lines to make sure it is appropriate when checked against instructions for
robots.txt. We definitely don't want to accidentally instruct the bots to not crawl important pages.This endpoint will be cached, so for all intents and purposes it will be as if it is static. It will rarely change, so we will have to cachebust it manually if we do change it, but it will also fade from the cache on its own every 24 hours and be re-fetched cold from origin, which I think is fine.