Generate llms.txt and llms-full.txt for the docs site

[vivek7405]

Problem

The docs site (docs/) and marketing site (website/) ship no machine-readable agent entrypoints: no /llms.txt index, no /llms-full.txt concatenation, and no per-page raw/markdown export. The docs JSON search route is a UI typeahead, not an ingestible corpus, so an agent must crawl ~40 HTML page.ts routes and parse around the html`` wrapper to reconstruct prose. The clean agent-docs/*.md files would feed llms-full.txt directly.

Design / approach

Serve /llms.txt and /llms-full.txt as plain text built from the existing agent-docs/*.md, wired into the docs app's run scripts. Plain files following the llms.txt standard, no generator framework.

Web-standards fit: Follows the open llms.txt standard and reuses the existing markdown; output is plain static text, no build pipeline.

Prior art: llms.txt standard (llmstxt.org); Next.js, Mintlify, and Anthropic docs emit /llms.txt + /llms-full.txt and per-page .md.

Acceptance criteria

• /llms.txt serves a structured index of doc pages with titles, descriptions, and links
• /llms-full.txt serves the concatenated full prose corpus (agent-docs/ and doc pages)
• Per-page raw/markdown variants are fetchable for at least the core API pages
• Generation is wired into the docs app's run scripts (no separate build) and stays in sync
• A smoke check asserts /llms.txt and /llms-full.txt respond with non-empty machine-readable content
• Tests cover the new behaviour at the applicable layer(s)
• Docs / AGENTS.md / CONVENTIONS.md updated if the public surface changed

---

_{Filed from the production-readiness audit (webjs vs Next.js / Remix / Rails / Turbo / Lit). Theme: ai-dx. Priority: P1. Kept to webjs identity: no-build, progressive enhancement, web-components-first, AI-first, batteries-included, close to web standards.}