GEO Improvement: Add AI-crawler robots.txt allow rules and llms.txt for the docs site
Source audit: GEO Audit Report — 2026-08-11
Audit date: 2026-08-11
Run: https://github.com/github/gh-aw/actions/runs/31513457906
Finding
The robots and llms sub-scores are 0/0 on every single audited page — the docs homepage, all 20 sampled sitemap pages, and average 0.0/14 across the site. Audit recommendation: "Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)" and "Create /llms.txt for AI indexing: geo llms --base-url https://github.github.com/gh-aw".
Why this matters
robots and llms are two of the highest-weighted, currently-zero categories across the entire docs site (github.github.com/gh-aw/) — fixing both would add an estimated combined ~20+ points to every page's GEO score (robots max 15, llms max 14), likely lifting the docs homepage from 44 to the 60s and pulling multiple "critical"-band blog pages (currently 33–35) up into "foundation"/"good" territory. This is the single largest, most mechanical fix available since it requires no content rewrite, only two static files.
Suggested fix
- Add a
robots.txt to the docs site root (https://github.github.com/gh-aw/robots.txt) with explicit Allow rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar AI crawler user-agents, alongside the existing sitemap directive.
- Generate
/llms.txt using the geo-optimizer-skill helper: geo llms --base-url https://github.github.com/gh-aw and commit the output to the docs site's static assets so it's served at https://github.github.com/gh-aw/llms.txt.
- Verify both files are served with
200 and correct Content-Type: text/plain after the next docs site deploy.
- Re-run the GEO audit workflow to confirm the
robots and llms sub-scores move from 0 to their maximum values.
Generated by 🌍 GEO Optimizer Daily Audit · auto · 16.5 AIC · ⌖ 2.73 AIC · ⊞ 7K · ◷
GEO Improvement: Add AI-crawler robots.txt allow rules and llms.txt for the docs site
Source audit: GEO Audit Report — 2026-08-11
Audit date: 2026-08-11
Run: https://github.com/github/gh-aw/actions/runs/31513457906
Finding
Why this matters
robotsandllmsare two of the highest-weighted, currently-zero categories across the entire docs site (github.github.com/gh-aw/) — fixing both would add an estimated combined ~20+ points to every page's GEO score (robots max 15, llms max 14), likely lifting the docs homepage from 44 to the 60s and pulling multiple "critical"-band blog pages (currently 33–35) up into "foundation"/"good" territory. This is the single largest, most mechanical fix available since it requires no content rewrite, only two static files.Suggested fix
robots.txtto the docs site root (https://github.github.com/gh-aw/robots.txt) with explicitAllowrules forGPTBot,ClaudeBot,PerplexityBot,Google-Extended, and similar AI crawler user-agents, alongside the existing sitemap directive./llms.txtusing thegeo-optimizer-skillhelper:geo llms --base-url https://github.github.com/gh-awand commit the output to the docs site's static assets so it's served athttps://github.github.com/gh-aw/llms.txt.200and correctContent-Type: text/plainafter the next docs site deploy.robotsandllmssub-scores move from 0 to their maximum values.