[WIP] Add robots.txt with Allow rules for AI bots

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add robots.txt with explicit AI bot Allow rules to the docs site</issue_title>
<issue_description>## GEO Improvement: Add robots.txt with AI bot Allow rules

Source audit: GEO Audit Report — 2026-07-24
Audit date: 2026-07-24
Run: §30110998991

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"

The docs site at https://github.github.com/gh-aw/ currently has no robots.txt. Without explicit Allow rules, AI crawlers may not index the site, preventing it from being cited by AI engines such as ChatGPT, Claude, and Perplexity.

Why this matters

This is the highest-structural-signal fix available: AI crawlers check robots.txt before crawling. Without explicit Allow entries for GPTBot, ClaudeBot, and PerplexityBot, the entire docs site may be skipped for AI-engine citation — the docs site currently scores 44/100 (Foundation band), and fixing crawler access is a prerequisite for all other GEO improvements to take effect.

Suggested fix

Add a robots.txt file to the root of the docs site (static/robots.txt or equivalent) with at minimum:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

Verify by running:

curl https://github.github.com/gh-aw/robots.txt

Generated by 🌍 GEO Optimizer Daily Audit · sonnet46 · 23.2 AIC · ⌖ 19.6 AIC · ⊞ 5K · ◷

• expires on Jul 31, 2026, 9:02 AM UTC-08:00

• Closes [geo-optimizer] Add robots.txt with explicit AI bot Allow rules to the docs site #47828

Comments on the Issue (you are @copilot in this section)

• Fixes [geo-optimizer] Add robots.txt with explicit AI bot Allow rules to the docs site #47828