feat: serve llms.txt and llms-full.txt from the docs site

[vivek7405]

Closes #261

What

The docs site shipped no machine-readable agent entrypoints, so an agent had to crawl ~40 HTML page.ts routes and parse around the html template wrapper to reconstruct prose. Add the open llms.txt standard (llmstxt.org) surfaces, generated live from the doc pages with no build step:

• /llms.txt: the structured index. Every doc topic as a titled, described, absolutely-linked entry, in sidebar-nav order.
• /llms-full.txt: the concatenated full prose corpus (52x the index).
• /docs/<topic>/llms.txt: the per-page markdown variant for every topic (the llmstxt.org "append the markdown URL" convention).

All three reuse the search route's page-walk and text extraction, cache the page list in memory, and derive absolute links from the request origin (correct in dev and prod).

Design notes

• No collision: the per-page route is a dynamic app/docs/[topic]/llms.txt/route.ts. It only matches /docs/<x>/llms.txt, never /docs/<x>, so the 39 static page.ts pages still serve their HTML. Verified by booting the app: /docs/getting-started returns 200 text/html, /docs/getting-started/llms.txt returns 200 text/plain.
• Deployment-safe sourcing: the corpus is built from the docs app's own app/docs/** pages (always shipped). Repo-root agent-docs/*.md are folded in only via a try/catch read that no-ops when absent, so a standalone deploy never 500s. Paths anchor on import.meta.url, not process.cwd().
• No drift: the generator walks live pages at request time (no hardcoded list). The test ties the index link count to the on-disk topic count, so a new page appears automatically.

Tests

test/docs/llms.test.mjs (9): index H1 plus absolute links, the drift count-match, the corpus carrying distinct full prose, the per-page markdown, a 404 on an unknown topic, and the explicit no-collision check (human page still 200 text/html). Boots the docs app via createRequestHandler. Full suite 1897. Round 2 confirmed the new routes do not disturb the search index, home redirect, or CSS pipeline.

Scope

Touches only docs/ plus test/docs/. No framework change, so the relevant dogfood is the docs-app boot (all surfaces serve 200 text/plain; the human pages and search route are unaffected). No version bump.

[vivek7405]

Reviewed by booting the docs app and hitting every surface, since the real risk in a docs site is a dynamic route shadowing the human pages. It does not: /docs/[topic]/llms.txt only matches /docs//llms.txt, so /docs/getting-started still serves its 200 text/html page, /docs/getting-started/llms.txt serves the markdown, and a bogus topic 404s cleanly rather than 500ing. The index (39 links) and the full corpus (52x larger) are genuinely distinct, links are origin-derived absolutes, and the text extraction reuses the search route's stripping so no template syntax leaks into the prose.

Second round checked the new dynamic dir does not disturb the rest of the app: the search index walk skips the page-less [topic] folder (no junk entry, no crash), the human pages and home redirect still work, and the routes serve text/plain independent of the Tailwind build. Deployment-safe sourcing too, the generator anchors on import.meta.url and the agent-docs enrichment is a no-op when absent, so a standalone deploy never breaks. The no-drift test ties the index link count to the on-disk page count so a new doc page appears automatically. Two rounds, last clean. 9/9 new tests, full suite 1897. Good to merge once CI is green.