fix: improved robots.txt matching

[harlan-zw]

🔗 Linked issue

#130

❓ Type of change

• 📖 Documentation (updates to the documentation or readme)
• 🐞 Bug fix (a non-breaking change that fixes an issue)
• 👌 Enhancement (improving an existing functionality)
• ✨ New feature (a non-breaking change that adds functionality)
• 🧹 Chore (updates to the build process or auxiliary tools and libraries)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Currently, we have very basic path matching against the robots.txt to add the meta tag and header automatically, this is leading to inconsistent states.

In this PR we I'm to:

• Match robots.txt using the same logic as Google
• Apply rules for specific user agent
• See in devtools which robots.txt rule is blocking current page from indexing