feat: spec-conformant llms.txt for AI agents#9618
Merged
Merged
Conversation
/llms.txt previously fell through to SPA routing and returned the index.html shell — Lighthouse's Agentic Browsing audit flagged it as a non-conformant llms.txt (no H1, no links). Adds a proper llmstxt.org file (H1 + summary blockquote + H2 link sections) covering the catalog, docs, MCP endpoint and repo, plus exact-match nginx locations in both server blocks so mapped crawler UAs get the file instead of being proxied to /seo-proxy/llms.txt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a spec-conformant llms.txt file for AI agents and ensures it’s served as a real static asset (not the SPA shell) by nginx, addressing Lighthouse “Agentic Browsing” audit findings.
Changes:
- Add
app/public/llms.txtformatted per llmstxt.org (H1 + summary + link sections). - Update
app/nginx.confto serve/llms.txtvia exact-matchlocation = /llms.txtin both server blocks, bypassing bot proxy / SPA routing. - Document the change in
CHANGELOG.mdunder[Unreleased].
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
CHANGELOG.md |
Adds an [Unreleased] “Added” entry describing the new llms.txt behavior and nginx serving change. |
app/public/llms.txt |
Introduces the new llms.txt content with catalog/docs links and MCP endpoint reference. |
app/nginx.conf |
Adds exact-match nginx routing for /llms.txt to avoid bot proxy / SPA fallback behavior. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Review feedback (Copilot): that server block's fallback has no bot proxy, so the location is defensive consistency, not a necessity — say so instead of implying the main-block rationale applies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntries) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 tasks
MarkusNeusinger
added a commit
that referenced
this pull request
Jul 8, 2026
… 403s runner IPs) (#9619) ## Summary - The first dispatched `bot-serving-check.yml` run failed with **403 on all four checks — including the human-UA control**: Cloudflare's bot management blocks GitHub-runner (datacenter) IPs, and a UA-spoofed "Googlebot" only passes verified-bot checks when coming from real Google IPs. The monitor was structurally unable to run from Actions against the proxied domain. - Switch all checks to the Cloud Run origin (`anyplot-app-…-ez.a.run.app`) — exactly the layer that broke in the 2026-06/07 incident, so this is also the more precise probe. Cloudflare-edge issues are explicitly out of this monitor's reach (documented in the header comment). - Adds an `llms.txt` check (Googlebot UA must get the file, never a seo-proxy 404) — guards the exact-match nginx location from #9618. ## Verification - Dispatched run 28981172030 documented the failure mode (4× 403). - The updated script was dry-run against the live origin: **5/5 OK, exit 0**. - After merge I will dispatch the workflow once — expected green. ## Test plan - [x] YAML valid; script dry-run against live origin passes 5/5 - [ ] Post-merge `workflow_dispatch` run is green Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/llms.txtpreviously fell through to SPA routing and returned the index.html shell — Lighthouse's new Agentic Browsing audit flagged it as a non-conformant llms.txt ("missing required H1", "contains no links"). There was in fact no llms.txt at all.app/public/llms.txtfollowing the llmstxt.org format: H1 title, summary blockquote, context paragraph, and H2 sections (Catalog,Docs,Optional) with described links — catalog pages, sitemap, palette, the MCP endpoint (https://api.anyplot.ai/mcp/) and the GitHub repo.location = /llms.txt { try_files $uri =404; }to both nginx server blocks (same pattern as robots.txt): llms.txt consumers are bots, and any UA in the$is_botmap would otherwise be proxied to/seo-proxy/llms.txtand 404.Notes
Disallow: /for ClaudeBot/GPTBot/CCBot (audit finding — dashboard setting, undermines llms.txt for those crawlers), and the$is_botmap doesn't include AI-agent UAs (GPTBot, ClaudeBot, PerplexityBot, …), so they get the SPA shell instead of prerendered HTML.Plan
N/A (direct request — Lighthouse Agentic Browsing finding)
Test plan
yarn build→dist/llms.txtpresent with correct contentcurl https://anyplot.ai/llms.txtreturns the file (human UA) andcurl -A Googlebot https://anyplot.ai/llms.txtreturns the file too (not a seo-proxy 404)