Add explicit AI crawler allowlist to docs robots.txt#43599
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Great work on the This PR looks well-scoped and ready for review. Nothing blocking here — nice and tidy.
|
|
✅ Test Quality Sentinel completed test quality analysis. |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #43599 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100). |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
|
✅ PR Code Quality Reviewer completed the code quality review. |
There was a problem hiding this comment.
Pull request overview
Updates the docs site’s crawler signaling so AI indexers can clearly discover and crawl the github.github.com/gh-aw/ docs, aligning robots.txt with the GEO audit expectations and the deployed sitemap shape.
Changes:
- Tightens
docs/public/robots.txtto an explicit AI crawler allowlist while keeping a global allow rule. - Updates the sitemap reference to
sitemap-index.xmlto match the built/deployed output. - Adds a Playwright spec to validate
robots.txtcontents and sitemap URL under the/gh-aw/base path.
Show a summary per file
| File | Description |
|---|---|
| docs/public/robots.txt | Updates crawler directives and points sitemap discovery to sitemap-index.xml. |
| docs/tests/robots-txt.spec.ts | Adds Playwright coverage to assert the deployed robots policy and sitemap index URL. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Low
| import { test, expect } from '@playwright/test'; | ||
|
|
||
| test.describe('robots.txt', () => { | ||
| test('should expose the docs crawler policy and sitemap index', async ({ request }) => { | ||
| const response = await request.get('/gh-aw/robots.txt'); | ||
| expect(response.ok()).toBeTruthy(); | ||
|
|
||
| const body = await response.text(); | ||
|
|
||
| expect(body).toContain('User-agent: *\nAllow: /'); | ||
| expect(body).toContain('User-agent: GPTBot\nAllow: /'); | ||
| expect(body).toContain('User-agent: OAI-SearchBot\nAllow: /'); | ||
| expect(body).toContain('User-agent: ChatGPT-User\nAllow: /'); | ||
| expect(body).toContain('User-agent: anthropic-ai\nAllow: /'); | ||
| expect(body).toContain('User-agent: ClaudeBot\nAllow: /'); | ||
| expect(body).toContain('User-agent: PerplexityBot\nAllow: /'); | ||
| expect(body).toContain('User-agent: Perplexity-User\nAllow: /'); | ||
| expect(body).toContain('User-agent: Google-Extended\nAllow: /'); | ||
| expect(body).toContain('User-agent: Google-CloudVertexBot\nAllow: /'); | ||
| expect(body).toContain('Sitemap: https://github.github.com/gh-aw/sitemap-index.xml'); | ||
| }); | ||
| }); |
🧪 Test Quality Sentinel Report
📊 Metrics (1 test)
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd — commenting on test robustness. The robots.txt change itself is clean and purposeful.
📋 Key Themes & Highlights
Key Themes
- Test assertion fragility: All directive checks use exact
\nsubstring matching; CRLF line endings or a blank line between stanzas would break every assertion while the policy itself is correct. - Missing Content-Type assertion: The spec checks HTTP 200 but not that the file is served as
text/plain, which some crawlers require. - Sitemap URL assertion could be tighter: A multiline-anchored
toMatchwould prevent a future path-prefix regression from slipping past.
Positive Highlights
- ✅ Trimming to just the 9 audit-targeted AI crawlers is exactly right — narrower allowlists are easier to audit.
- ✅ Fixing
sitemap.xml→sitemap-index.xmlaligns with the actual GitHub Pages build output. - ✅ Adding a Playwright regression spec is the right approach for a static-file policy change.
- ✅ The single test block is concise and readable.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 38.5 AIC · ⌖ 5.24 AIC · ⊞ 6.7K
Comment /matt to run again
| expect(response.ok()).toBeTruthy(); | ||
|
|
||
| const body = await response.text(); | ||
|
|
| expect(response.ok()).toBeTruthy(); | ||
|
|
||
| const body = await response.text(); | ||
|
|
There was a problem hiding this comment.
[/tdd] The toContain('User-agent: GPTBot\nAllow: /') pattern is brittle — it relies on exact Unix \n line endings and zero blank lines between directives. Both are valid in real robots.txt files and common on Windows build agents, yet either would silently pass the HTTP check while failing every directive assertion.
💡 More resilient approach
Use toMatch with \r?\n so CRLF-normalised files don't break CI:
expect(body).toMatch(/User-agent: GPTBot\r?\nAllow: \//)
expect(body).toMatch(/User-agent: OAI-SearchBot\r?\nAllow: \//)
// ...Or assert each directive independently (order and whitespace agnostic):
expect(body).toMatch(/User-agent:\s+GPTBot/)
expect(body).toMatch(/Allow:\s+\//)@copilot please address this.
| expect(body).toContain('User-agent: Google-Extended\nAllow: /'); | ||
| expect(body).toContain('User-agent: Google-CloudVertexBot\nAllow: /'); | ||
| expect(body).toContain('Sitemap: https://github.github.com/gh-aw/sitemap-index.xml'); | ||
| }); |
There was a problem hiding this comment.
[/tdd] The test verifies the sitemap URL only as a substring, but not its format validity. If the URL accidentally gains a double slash or path prefix, the toContain check would still pass.
💡 Tighter assertion
Use toMatch with a pattern anchored to the label:
expect(body).toMatch(/^Sitemap: https:\/\/github\.github\.com\/gh-aw\/sitemap-index\.xml$/m)The ^...$ multiline anchors ensure no extra path segments can sneak in before or after the URL.
@copilot please address this.
| test('should expose the docs crawler policy and sitemap index', async ({ request }) => { | ||
| const response = await request.get('/gh-aw/robots.txt'); | ||
| expect(response.ok()).toBeTruthy(); | ||
|
|
There was a problem hiding this comment.
[/tdd] The test confirms the file is served with HTTP 200, but not its Content-Type. Serving robots.txt with the wrong content type (e.g. text/html) causes some crawlers to ignore it entirely.
💡 Add a content-type assertion
expect(response.headers()['content-type']).toMatch(/text\/plain/)This guards against the static server accidentally serving the file as HTML on a path mismatch.
@copilot please address this.
There was a problem hiding this comment.
robots.txt AI crawler allowlist
The changes are straightforward and correct:
User-agent: *\nAllow: /is preserved, so all previously-allowed bots (Googlebot, bingbot, etc.) remain unblocked — removing their explicit entries has no behavioral effect.- The sitemap URL update to
sitemap-index.xmlis validated:docs/public/sitemap.xmlis itself a sitemapindex already pointing to that exact URL. - The new Playwright spec uses
toContainwith\n-separated assertion strings, which match correctly against the Linux-style robots.txt. - No security, correctness, or reliability issues found.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 42.4 AIC · ⌖ 5.88 AIC · ⊞ 4.9K
There was a problem hiding this comment.
REQUEST_CHANGES — The PR's core correctness claim is invalid: robots.txt at a GitHub Pages subpath is never read by any crawler.
Blocking issues found
RFC 9309 path violation (critical)
Per RFC 9309, all crawlers fetch robots.txt exclusively from the origin root (https://github.github.com/robots.txt). This site deploys under the subpath /gh-aw/, so docs/public/robots.txt lands at https://github.github.com/gh-aw/robots.txt — a URL no compliant crawler ever consults. Every User-agent directive added here is silently ignored at runtime. The PR goal (AI crawler indexing) is not achieved.
Test validates an irrelevant URL (high)
The new Playwright spec GETs /gh-aw/robots.txt. This confirms the static asset is in the build output, but confirms nothing about crawler policy effectiveness. The test name and description imply it validates crawler behavior, which is misleading. It will pass even when the deployment is fully broken for crawlers.
Other non-blocking notes (already flagged by prior review)
- Tab vs 2-space indentation inconsistency in the new spec.
\n-hardcoded containment assertions are brittle on non-LF environments.
🔎 Code quality review by PR Code Quality Reviewer · 88.7 AIC · ⌖ 6.59 AIC · ⊞ 5.4K
Comment /review to run again
|
|
||
| const body = await response.text(); | ||
|
|
||
| expect(body).toContain('User-agent: *\nAllow: /'); |
There was a problem hiding this comment.
Test validates the wrong URL — crawlers never fetch /gh-aw/robots.txt: the test confirms the file is served correctly under the subpath, but this is not the URL any crawler consults; it gives a false green on the PR's core correctness claim.
💡 Detail
All crawler user-agents (GPTBot, ClaudeBot, etc.) perform exactly one GET per origin: https://github.github.com/robots.txt. The test hits GET /gh-aw/robots.txt, which is the Vite dev-server path for the static asset — useful for checking that the file is included in the build, but not for validating that the crawler policy is actually effective.
If the intent is purely to verify the file is present in the build output, rename the test describe/title to reflect that ('robots.txt static asset is served in the build') rather than implying it enforces a crawler policy. This avoids misleading future readers.
github-actions
Bot
added
pr-action:fast_track
pr-agent:copilot
pr-priority:medium
labels
🤖 PR Triage — Run §28750744185
Score breakdown: Impact 28 + Urgency 18 + Quality 16 Rationale: Straightforward docs-only change (2 files, +27/-63). CI passing (3/5 SUCCESS, 2 SKIPPED). Bot reviewer commented but no blocking changes_requested. GEO audit compliance has moderate business impact. Low risk — no logic changes. Recommended next step: Human review + merge.
|
|
@copilot please run the
|
…content assertion Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Addressed in commit
Local |
|
@copilot please run the
|
All review feedback was addressed in commit |
The docs site already ships sitemap assets, but its
robots.txtdid not match the GEO audit’s expected AI crawler signaling and pointed at the wrong sitemap entrypoint. This updates the docs-site crawler policy so AI indexers can unambiguously discover and crawlgithub.github.com/gh-aw/.Robots policy
docs/public/robots.txtto the crawler directives called out by the auditGPTBotOAI-SearchBotChatGPT-Useranthropic-aiClaudeBotPerplexityBotPerplexity-UserGoogle-ExtendedGoogle-CloudVertexBotSitemap discovery
sitemap.xmltositemap-index.xml, which matches the built docs output and GitHub Pages deployment shapeFocused regression coverage
/gh-aw/robots.txtand asserts the expected bot directives and sitemap index URL