Add custom robots.txt to docs

[allcaps]

This PR adds a custom robots.txt to the docs.

It allows crawling of /en/stable/ and blocks latest, versions and pdf.

• Do the tests still pass? N/A
• Does the code comply with the style guide? YES
• For Python changes: Have you added tests to cover the new/fixed behaviour? N/A
• For front-end changes: Did you test on all of Wagtail’s supported browsers? Please list the exact versions you tested. N/A
• For new features: Has the documentation been updated accordingly? N/A

[squash-labs]

Manage this branch in Squash

Test this branch here: https://allcapsadd-robots-txt-p22ax.squash.io