[WIP] Add AI-crawler robots.txt allow rules and llms.txt for the docs site

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add AI-crawler robots.txt allow rules and llms.txt for the docs site</issue_title>
<issue_description>### GEO Improvement: Add AI-crawler robots.txt allow rules and llms.txt for the docs site

Source audit: GEO Audit Report — 2026-08-11
Audit date: 2026-08-11
Run: https://github.com/github/gh-aw/actions/runs/31513457906

Finding

The robots and llms sub-scores are 0/0 on every single audited page — the docs homepage, all 20 sampled sitemap pages, and average 0.0/14 across the site. Audit recommendation: "Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)" and "Create /llms.txt for AI indexing: geo llms --base-url https://github.github.com/gh-aw".

Why this matters

robots and llms are two of the highest-weighted, currently-zero categories across the entire docs site (github.github.com/gh-aw/) — fixing both would add an estimated combined ~20+ points to every page's GEO score (robots max 15, llms max 14), likely lifting the docs homepage from 44 to the 60s and pulling multiple "critical"-band blog pages (currently 33–35) up into "foundation"/"good" territory. This is the single largest, most mechanical fix available since it requires no content rewrite, only two static files.

Suggested fix

1. Add a robots.txt to the docs site root (https://github.github.com/gh-aw/robots.txt) with explicit Allow rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar AI crawler user-agents, alongside the existing sitemap directive.
2. Generate /llms.txt using the geo-optimizer-skill helper: geo llms --base-url https://github.github.com/gh-aw and commit the output to the docs site's static assets so it's served at https://github.github.com/gh-aw/llms.txt.
3. Verify both files are served with 200 and correct Content-Type: text/plain after the next docs site deploy.
4. Re-run the GEO audit workflow to confirm the robots and llms sub-scores move from 0 to their maximum values.

Generated by 🌍 GEO Optimizer Daily Audit · auto · 16.5 AIC · ⌖ 2.73 AIC · ⊞ 7K · ◷

• expires on Aug 18, 2026, 8:44 AM UTC-08:00

• Closes [geo-optimizer] Add AI-crawler robots.txt allow rules and llms.txt for the docs site #52113

Comments on the Issue (you are @copilot in this section)

• Fixes [geo-optimizer] Add AI-crawler robots.txt allow rules and llms.txt for the docs site #52113