[WIP] Add robots.txt with AI bot Allow rules to docs site

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add robots.txt with AI bot Allow rules to docs site (github.github.com/gh-aw/)</issue_title>
<issue_description>## GEO Improvement: Add robots.txt to docs site

Source audit: GEO Audit Report — 2026-07-16
Audit date: 2026-07-16
Run: §29517087856

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"

The docs site at github.github.com/gh-aw/ currently scores 0/15 on the Robots.txt check. No robots.txt file is present at the docs domain, meaning major AI crawlers (GPTBot, ClaudeBot, PerplexityBot) have no explicit permission signal to index the site. This affects all 10+ sitemap pages.

Why this matters

Without explicit Allow rules, AI search engines may deprioritize or skip crawling the docs site entirely, reducing citation coverage in ChatGPT, Claude, Perplexity, and similar tools. Adding robots.txt is the single highest-impact fix (+15 pts on the robots check alone) and immediately benefits every page on the docs domain.

Suggested fix

1. Create robots.txt at the root of the docs site (github.github.com/gh-aw/robots.txt) with the following content:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

2. If the docs site is generated by a static site generator (e.g., Docusaurus, Jekyll), add the robots.txt to the static/ or root assets directory so it is published at the correct path.

3. After publishing, verify with: curl https://github.github.com/gh-aw/robots.txt

4. Re-run gh aw run geo-optimizer to confirm the score improvement.

Generated by 🌍 GEO Optimizer Daily Audit · 27.8 AIC · ⌖ 9.73 AIC · ⊞ 4.9K · ◷

• expires on Jul 23, 2026, 8:56 AM UTC-08:00

• Closes [geo-optimizer] Add robots.txt with AI bot Allow rules to docs site (github.github.com/gh-aw/) #46054

Comments on the Issue (you are @copilot in this section)