docs: update robots.txt to allow crawlers

[thesmiler]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• angular.io application / infrastructure changes
• Other... Please describe:

What is the current behavior?

angular.dev doesn't allow any crawlers

Issue Number: N/A

What is the new behavior?

Allow crawlers on angular.dev

Does this PR introduce a breaking change?

• Yes
• No

Other information

[thesmiler]

Changed from Allow: / to Disallow:
as this is the original robots.txt standard and not a Google extension

[pkozlowski-opensource]

This PR was merged into the repository by commit a67c059.