[website] Add robots.txt

[jackkoenig]

This will hopefully make the Algolia web crawler work.

With the website facelift, I'm trying to enable Algolia Docsearch. Things seem to be set up properly, but the webcrawler won't index the website. I'm following the instructions here (https://www.algolia.com/doc/tools/crawler/getting-started/verify-domain/) to try to fix this. It's a bit weird that this isn't mentioned in the Docusaurus docs, so we'll see if it fixes things.

Contributor Checklist

• Did you add Scaladoc to every public function/method?
• Did you add at least one test demonstrating the PR?
• Did you delete any extraneous printlns/debugging code?
• Did you specify the type of improvement?
• Did you add appropriate documentation in docs/src?
• Did you request a desired merge strategy?
• Did you add text to be included in the Release Notes for this change?

Type of Improvement

• Documentation or website-related

Desired Merge Strategy

• Squash

Release Notes

This will hopefully make the Algolia web crawler work.

Reviewer Checklist (only modified by reviewer)

• Did you add the appropriate labels? (Select the most appropriate one based on the "Type of Improvement")
• Did you mark the proper milestone (Bug fix: 3.5.x, 3.6.x, or 5.x depending on impact, API modification or big change: 6.0)?
• Did you review?
• Did you check whether all relevant Contributor checkboxes have been checked?
• Did you do one of the following when ready to merge:
  • Squash: You/ the contributor Enable auto-merge (squash), clean up the commit message, and label with Please Merge.
  • Merge: Ensure that contributor has cleaned up their commit history, then merge with Create a merge commit.