[WIP] Add robots.txt to docs site for AI crawler indexing

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add robots.txt to docs site to allow AI crawler indexing</issue_title>
<issue_description>## GEO Improvement: Add robots.txt to docs site

Source audit: GEO Audit Report — 2026-07-15
Audit date: 2026-07-15
Run: §29433981536

Finding

robots.txt check scored 0/18 for github.github.com/gh-aw/. The file is missing entirely — AI bots (GPTBot, ClaudeBot, PerplexityBot) are receiving 301 redirects with no explicit crawl permissions defined.

Audit recommendation: "Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"

Why this matters

The robots.txt file is the highest-weighted single check in the GEO audit (18 points maximum). Without it, AI search engines have no explicit permission signal to index and cite the docs site. The README already correctly allows 27 AI bots — the docs site needs the same treatment to unlock AI citation coverage across all 10 audited pages (currently all in the Foundation band, 36–44/100).

Estimated score impact: +~15 points on docs site GEO score (44 → ~59/100).

Suggested fix

1. Create static/robots.txt (or equivalent static file path for the Astro docs site) at the root of github.github.com/gh-aw/.
2. Add Allow directives for key AI bots, mirroring the README's robots.txt pattern:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

3. Verify by re-running: gh aw geo audit --url https://github.github.com/gh-aw/

Generated by 🌍 GEO Optimizer Daily Audit · 32.8 AIC · ⌖ 20.2 AIC · ⊞ 4.9K · ◷

• expires on Jul 22, 2026, 8:57 AM UTC-08:00

• Closes [geo-optimizer] Add robots.txt to docs site to allow AI crawler indexing #45791

Comments on the Issue (you are @copilot in this section)