chore: add proper robots.txt config

[derberg]

Before we release new version of website, we need to merge this PR. It reverts #1846 that was done for the purpose of v3.asyncapi.com

[netlify]

✅ Deploy Preview for shimmering-choux-eb0798 ready!

Name	Link
🔨 Latest commit	e7c597c
🔍 Latest deploy log	https://app.netlify.com/sites/shimmering-choux-eb0798/deploys/649c4a7a29cfc20007460cf7
😎 Deploy Preview	https://deploy-preview-1866--shimmering-choux-eb0798.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this pull request, add a comment with detailed explanation.

There can be many reasons why some specific pull request has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this pull request forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[derberg]

already done