feat(docs): improve GEO scores — robots.txt AI crawlers, homepage stats, JSON-LD sameAs/dateModified#31260
Merged
Merged
Conversation
8 tasks
…ts, JSON-LD dateModified/sameAs Agent-Logs-Url: https://github.com/github/gh-aw/sessions/9f877597-0019-4ce7-8e33-49ca7da7b8fb Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Improve GEO scores for docs site and README
feat(docs): improve GEO scores — robots.txt AI crawlers, homepage stats, JSON-LD sameAs/dateModified
May 9, 2026
pelikhan
approved these changes
May 9, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Improves the docs site’s GEO/SEO signals by explicitly allowing common AI crawlers in robots.txt, adding quantitative “By the Numbers” stats to the docs homepage, and enriching homepage JSON-LD metadata (expanded sameAs, added dateModified).
Changes:
- Added an explicit allow-list of AI/search crawler user-agents in
docs/public/robots.txt. - Added a “By the Numbers” metrics table to the docs homepage.
- Expanded JSON-LD
Organization.sameAslinks and addeddateModifiedfields inCustomHead.astro.
Show a summary per file
| File | Description |
|---|---|
| docs/src/content/docs/index.mdx | Adds a homepage metrics table to provide concrete quantitative signals. |
| docs/src/components/CustomHead.astro | Updates homepage JSON-LD with more sameAs links and dateModified metadata. |
| docs/public/robots.txt | Adds explicit AI/search crawler user-agent groups that allow full-site crawling. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 5
| | Security layers | 5 (read-only token, zero secrets, network firewall, safe outputs, threat detection) | | ||
| | Design patterns | 18+ (IssueOps, ChatOps, DailyOps, BatchOps, and more) | | ||
| | Supported GitHub event triggers | 10+ (issues, pull_request, push, schedule, discussion, label, …) | | ||
| | Safe output types | 8+ (create-issue, create-pull-request, add-comment, add-label, …) | |
| | Supported AI engines | 4 (GitHub Copilot, Claude, OpenAI Codex, custom) | | ||
| | Security layers | 5 (read-only token, zero secrets, network firewall, safe outputs, threat detection) | | ||
| | Design patterns | 18+ (IssueOps, ChatOps, DailyOps, BatchOps, and more) | | ||
| | Supported GitHub event triggers | 10+ (issues, pull_request, push, schedule, discussion, label, …) | |
Comment on lines
+52
to
+56
| 'https://github.com/github', | ||
| 'https://githubnext.com/', | ||
| 'https://github.blog/', | ||
| ], | ||
| dateModified: '2026-05-09', |
Comment on lines
62
to
+66
| url: 'https://github.github.com/gh-aw/', | ||
| description: defaultDescription, | ||
| applicationCategory: 'DeveloperApplication', | ||
| operatingSystem: 'Any', | ||
| dateModified: '2026-05-09', |
Comment on lines
+9
to
+85
| Allow: / | ||
|
|
||
| User-agent: ChatGPT-User | ||
| Allow: / | ||
|
|
||
| User-agent: OAI-SearchBot | ||
| Allow: / | ||
|
|
||
| User-agent: Google-Extended | ||
| Allow: / | ||
|
|
||
| User-agent: Googlebot | ||
| Allow: / | ||
|
|
||
| User-agent: ClaudeBot | ||
| Allow: / | ||
|
|
||
| User-agent: Claude-Web | ||
| Allow: / | ||
|
|
||
| User-agent: anthropic-ai | ||
| Allow: / | ||
|
|
||
| User-agent: PerplexityBot | ||
| Allow: / | ||
|
|
||
| User-agent: Perplexity-User | ||
| Allow: / | ||
|
|
||
| User-agent: Applebot | ||
| Allow: / | ||
|
|
||
| User-agent: Applebot-Extended | ||
| Allow: / | ||
|
|
||
| User-agent: Amazonbot | ||
| Allow: / | ||
|
|
||
| User-agent: cohere-ai | ||
| Allow: / | ||
|
|
||
| User-agent: Omgilibot | ||
| Allow: / | ||
|
|
||
| User-agent: FacebookBot | ||
| Allow: / | ||
|
|
||
| User-agent: Bytespider | ||
| Allow: / | ||
|
|
||
| User-agent: bingbot | ||
| Allow: / | ||
|
|
||
| User-agent: BingPreview | ||
| Allow: / | ||
|
|
||
| User-agent: DuckDuckBot | ||
| Allow: / | ||
|
|
||
| User-agent: Slurp | ||
| Allow: / | ||
|
|
||
| User-agent: YandexBot | ||
| Allow: / | ||
|
|
||
| User-agent: Baiduspider | ||
| Allow: / | ||
|
|
||
| User-agent: ia_archiver | ||
| Allow: / | ||
|
|
||
| User-agent: CCBot | ||
| Allow: / | ||
|
|
||
| User-agent: DataForSeoBot | ||
| Allow: / | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the critical GEO gaps identified in the 2026-05-09 audit (docs site scored 40/100). Three targeted changes close the highest-impact gaps.
Changes
docs/public/robots.txt— explicit AI crawler allow-listAdded 27 named AI crawler entries (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, anthropic-ai, Amazonbot, cohere-ai, Applebot, bingbot, …). The wildcard
User-agent: *already permits all bots; explicit entries signal unambiguous welcome to AI indexers that evaluate per-bot directives.docs/src/content/docs/index.mdx— quantitative stats tableAdded a "By the Numbers" section to the homepage. The audit flagged
numbers_count: 0, which reduces AI snippet selection. Table provides 6 concrete figures: AI engines (4), security layers (5), design patterns (18+), event triggers (10+), safe output types (8+), install command.docs/src/components/CustomHead.astro— JSON-LD improvementsOrganization.sameAsexpanded from 1 to 4 URIs (github/gh-aw,github.com/github,githubnext.com,github.blog) to improve Knowledge Graph entity disambiguationdateModified: '2026-05-09'to bothOrganizationandSoftwareApplicationschema nodes to reduce content-decay scoringAlready present (no changes needed)
llms.txt(viastarlightLlmsTxtplugin), sitemap (via@astrojs/sitemap), and AI discovery endpoints (/.well-known/ai.txt,/ai/summary.json,/ai/faq.json,/ai/service.json) were already wired up correctly.