[Feature Request]: Ability to disable crawling #249
Description
Prerequisites
- I have searched existing issues to ensure this feature hasn't already been requested
- I have tested using the latest version of docs-builder
What problem are you trying to solve?
Ability to set <meta name="robots" content="noindex, nofollow" /> or a robots.txt
See https://developers.google.com/search/docs/crawling-indexing/block-indexing for further details
This is needed for staging and qa environments, where we don't want the documenation to be indexed by search engines.
Proposed Solution
Add a CLI flag and/or config file option to enable or disable crawling.
By default we should disable crawling, for production we then have to explicitly enable crawling.
Examples and Research
https://developers.google.com/search/docs/crawling-indexing/block-indexing
Alternative Solutions
No response
Additional Context
No response
How important is this feature to you?
Important