geo: 加 /llms.txt，AI 爬虫按用途分开放行 - #395
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a statically generated /llms.txt index for AI engines and updates /robots.txt generation to distinguish “training” crawlers (blocked site-wide) from “citation” crawlers (allowed via the * group). It also factors shared doc filtering + slug encoding into a reusable helper to keep sitemap and llms enumeration consistent.
Changes:
- Add
app/llms.txt/route.tsandlib/llms-txt.tsto generate a sectioned, absolute-linkllms.txtfrom the existing doc source. - Update
app/robots.tsto block a curated list of training crawlers while keeping citation crawlers allowed via the wildcard rule set; remove the ineffectivepublic/robots.txt. - Extract shared
isDraftOrHiddenanddocPathnameintolib/doc-entry.tsand update sitemap to use it; add Vitest coverage for robots + llms behaviors.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/robots.test.ts |
Adds policy-level tests for robots rules (training blocked, citation not blocked). |
tests/llms-txt.test.ts |
Adds tests for llms.txt grouping/rendering and shared doc-entry helpers. |
public/robots.txt |
Removes shadowed/ineffective static robots file. |
lib/llms-txt.ts |
Implements pure llms.txt renderer (grouping, sanitization, truncation, absolute URLs). |
lib/doc-entry.ts |
Centralizes shared draft/hidden filtering and slug→pathname encoding. |
generated/doc-contributors.json |
Regenerates metadata timestamp. |
app/sitemap.ts |
Switches sitemap doc entry generation to shared doc-entry helpers. |
app/robots.ts |
Implements updated crawler policy and consolidates disallow paths/constants. |
app/llms.txt/route.ts |
Adds force-static route handler to enumerate docs and return llms.txt content. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+36
| * 这份策略取代了 public/robots.txt —— 那份手写文件把两类一起屏蔽，而且 | ||
| * 早就不生效了：App Router 下 app/robots.ts 才是线上实际服务的 /robots.txt， | ||
| * 那份文件被静默遮蔽，读代码的人却会以为 AI 爬虫已经被挡住了。已删除。 | ||
| * |
Comment on lines
+11
to
+13
| * force-static：内容全部来自构建期的 MDX，没有请求相关的东西， | ||
| * 构建时产出一次即可，别让它变成每次请求现算的 dynamic 路由 | ||
| * （i18n/routing.ts 文件头记着上次全站 dynamic 把 Vercel CPU 打爆的事）。 |
Comment on lines
+8
to
+15
| * 原来这两个函数是 `app/sitemap.ts` 的私有函数。`app/llms.txt/route.ts` | ||
| * 要枚举同一批文档，如果各写一份，早晚会出现「sitemap 过滤了 draft、 | ||
| * llms.txt 没过滤」这种单边漂移 —— 草稿泄漏给 AI 引擎和泄漏给搜索引擎 | ||
| * 一样糟。抽到这里，两边共用一份，理由同 `lib/site-url.ts` 文件头。 | ||
| * | ||
| * 刻意不 import `@/lib/source`：那条链会把整个 fumadocs-mdx 管线拖进来， | ||
| * vitest 没配 MDX 插件会直接 parse 失败。入参用结构化的宽类型， | ||
| * 让本文件保持纯函数、可单测。 |
llms.txt（llmstxt.org 约定）给 AI 引擎一份无导航噪音的全站索引： 标题 + 描述 + 绝对链接，构建期从 source 枚举，和 sitemap 同源。 309 条（中英各半），只做索引不做全文 —— 全站正文拼进去几 MB， 反而挤爆它本来要省的上下文。 robots 按用途分开，不再一刀切： - 引用型（OAI-SearchBot / ChatGPT-User / Claude-User / PerplexityBot） 用户提问时实时取用并附出处链接，走 * 组放行。刻意不给它们开 UA 专属组 —— robots.txt 里专属组整体覆盖 * 而不是叠加，开了就得把 admin/editor 那串 disallow 再抄一遍，抄漏一条等于把后台放给它们。 - 训练型（GPTBot / ClaudeBot / Google-Extended / CCBot / Bytespider 等） 只收语料不给回链，内容是 CC BY-NC-SA，整站 disallow。 顺手删掉 public/robots.txt：那份手写文件把两类一起屏蔽，而且早就不生效 —— App Router 下 app/robots.ts 才是线上实际服务的 /robots.txt，它被静默 遮蔽，读代码的人却会以为 AI 爬虫已经挡住了。线上实测放行所有爬虫， 和文件里写的意图正好相反。 draft 过滤和 slug 编码从 app/sitemap.ts 抽到 lib/doc-entry.ts 共用： 两边各写一份的话，早晚出现 sitemap 过滤了草稿、llms.txt 没过滤， 草稿泄漏给 AI 引擎和泄漏给搜索引擎一样糟。 测试守的是策略不变量而不是实现：训练型必须被挡、引用型必须不在名单里。 两边错任何一边都不报错 —— 要么内容白送进训练集，要么再也拿不到 AI 引用。
tests/robots.test.ts 原来只断言 5 个训练型 UA，而策略里有 12 个： 从 AI_TRAINING_CRAWLERS 删掉没被断言的那几个，测试照样绿，等于白守。 补全成 12 个（新增不用同步，放宽是安全方向）。变异验证过：删掉 Applebot-Extended 会红，还原后绿。 注释违反 CLAUDE.md §3「严禁历史叙事 / 大段 docstring 描述功能」： 删掉 public/robots.txt 的来龙去脉、doc-entry 的抽取过程、force-static 里那段 Vercel CPU 往事，只留维护时会踩的不变量 —— UA 专属组整体覆盖 `*` 不叠加（所以不给引用型开 Allow 组）、不要 import @/lib/source （会拖进 MDX 管线让 vitest 挂）、sitemap 与 llms.txt 的过滤必须同进同退。 Co-authored-by: copilot-pull-request-reviewer[bot] <198982749+copilot-pull-request-reviewer[bot]@users.noreply.github.com>
longsizhuo
force-pushed
the
geo/llms-txt-and-ai-robots
branch
from
August 2, 2026 20:17
07fbb22 to
2ec1ea4
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.
Suppressed comments (1)
app/robots.ts:34
- PRIVATE_PATHS 的注释说“
*组和下面每个 UA 专属组都要带上”，但当前下方的训练爬虫组是Disallow: /，实际上不需要再带 PRIVATE_PATHS（而且现在也没带）。这会让后续维护者误以为这里漏配或需要复制粘贴。建议把注释改成：只有当某个 UA 组不是整站 Disallow 时，才需要显式带上 PRIVATE_PATHS（因为 UA 专属组覆盖*且不继承）。
/**
* 登录态 / 接口路径，任何爬虫都不该进。
* `*` 组和下面每个 UA 专属组都要带上（专属组覆盖 `*`，不继承）。
*/
longsizhuo
force-pushed
the
geo/llms-txt-and-ai-robots
branch
from
August 2, 2026 20:28
2ec1ea4 to
410b8de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
线上实测（2026-08）发现两件事：
/llms.txt不存在（404）。AI 引擎抓站时先看这个文件拿全站索引，没有就只能靠 sitemap 逐个页面抓，才知道每篇讲什么。public/robots.txt早就不生效，而且和线上行为完全相反。 App Router 下app/robots.ts才是实际服务/robots.txt的那份，public/robots.txt被静默遮蔽。前者对所有 AI 爬虫全放行，后者写着 "Block AI scrapers" 并屏蔽 GPTBot / ClaudeBot / Google-Extended / CCBot。读代码的人会以为 AI 爬虫已经挡住了，实际一个都没挡。改动
robots：按用途分开，不再一刀切
*组）Disallow: /引用型实时取用并在回答里附出处链接，训练型只收语料不给回链。内容是 CC BY-NC-SA，这个分法既拿得到 AI 引用，又守住不进训练集的原意。
刻意不给引用型开 UA 专属组：robots.txt 里专属组是整体覆盖
*而不是叠加，开了就得把admin/editor/settings那串 disallow 再抄一遍，抄漏一条等于把后台放给它们。少写一组反而更安全，这点在app/robots.ts的注释里写清楚了。顺手删掉失效的
public/robots.txt。/llms.txtapp/llms.txt/route.ts，force-static构建期产出一次（○ (Static)，和/robots.txt、/sitemap.xml同级）。从source现枚举，和 sitemap 同源，不会过期。309 条（中英各半），只做索引不做全文 —— 全站正文拼进去是几 MB，反而挤爆它本来要省的上下文。头部声明了 CC BY-NC-SA，向引用方讲清署名要求。
顺带：抽出
lib/doc-entry.tsdraft 过滤和 slug 编码原本是
app/sitemap.ts的私有函数，llms.txt 要枚举同一批文档。各写一份的话早晚出现「sitemap 过滤了草稿、llms.txt 没过滤」，草稿泄漏给 AI 引擎和泄漏给搜索引擎一样糟。理由同lib/site-url.ts文件头。测试
tests/robots.test.ts守的是策略不变量而不是实现：训练型必须被挡、引用型必须不在名单里。两边错任何一边都不会报错 —— 要么内容白送进训练集，要么再也拿不到 AI 引用，只能靠测试拦。tests/llms-txt.test.ts覆盖分组、绝对链接、超长描述截断、标题方括号（会破坏 markdown 链接）、以及docPathname的中文 slug 编码。tsc --noEmit干净pnpm build通过，/llms.txt预渲染为静态，绝对 URL 是https://involutionhell.com验证方式
pnpm build && head -20 .next/server/app/llms.txt.body curl -s http://localhost:3000/robots.txt🤖 Generated with Claude Code