Add robots.txt to restrict crawling of version-specific docs

[josh-wong]

Description

This PR adds a robots.txt file to restrict search engines from crawling the docs site so that only the latest version of docs appear as results. We need to implement this because the results are random, often show older or no longer maintained versions of docs, or aren't versions of docs that visitors are using, which is confusing.

Related issues and/or PRs

N/A

Changes made

• Created a robots.txt file that:
  • Disallows crawling of version number folders (/docs/3.14/, /ja-jp/docs/3.14/, etc.) by using wildcards.
  • Allows crawling of the latest versions of docs at /docs/latest/ and /ja-jp/docs/latest/.

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

• I have updated the side navigation as necessary.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes.
• Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
• My changes generate no new warnings.
• Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A