feat(mcp): optional built-in MCP server — list/get/search docs over Streamable HTTP

[mhenrixon]

Part of #8 (Phase 4). Depends on the markdown-twin, llms.txt, and search issues (it is a thin protocol skin over them).

Problem / Goal

Developers increasingly read docs through agents. llms.txt covers fetch-style consumption, but MCP is the native protocol: a user adds https://docs.example.com/mcp to Claude Code/Claude.ai/Cursor once and the docs become first-class tools (search, targeted page retrieval) instead of scraped text. Nothing exists today.

Goal: any docs-kit site can turn on a read-only MCP endpoint with one gem + one route — every current and future docs-kit site becomes agent-queryable for free.

Context (read these first)

• The official MCP Ruby SDK: mcp gem (github.com/modelcontextprotocol/ruby-sdk) — MCP::Server with tools/resources, JSON-RPC over Streamable HTTP; its Rails-controller integration pattern (server.handle_json(request.body.read) from a POST action).
• README.md:8-10 — the gem's precedent for optional runtime-detected dependencies (phlex-reactive, pgbus): docs-kit must NOT hard-depend on mcp.
• DocsKit::SearchIndex, DocsKit::MarkdownExport, Registry v2, c.tagline — everything the tools expose; they must NOT be reimplemented here.
• DocsKit::LlmsController pattern — gem controller, host-drawn route.

Decision

A read-only, stateless Streamable HTTP MCP endpoint built on the official mcp gem, mounted by the host app, off unless both the gem is present and the route is drawn.

1. Optional dependency: defined?(MCP::Server) gates everything; docs-kit adds mcp to no gemspec list. The generator offers it as a commented Gemfile line + commented route.
2. DocsKit::McpController (gem-provided): POST /mcp → builds/memoizes an MCP::Server named from c.brand, instructions from c.tagline + a pointer to /llms.txt, and three tools:
   • list_pages → [{slug, title, group, url}] from c.nav_registries (authored only).
   • get_page(slug:) → the page's Markdown twin (MarkdownExport), or a not-found message listing valid slugs.
   • search_docs(query:) → top SearchIndex hits as {page_title, section_title, url, snippet}.
3. Stateless JSON-RPC (each POST independent — no SSE session; the SDK's stateless mode) so it works behind the existing Kamal/Cloudflare deploy unchanged. GET/DELETE on the endpoint → 405.
4. Security posture: read-only tools over already-public content; CSRF skip on the API action; rate limiting documented as host responsibility (app-4 precedent: rate_limit in their base controller). A c.mcp config knob (default true) lets a site with the gem installed still disable the endpoint.
5. /llms.txt gains a final line advertising the MCP endpoint when enabled (agents discover it).

Alternatives rejected: a stdio binary (docs-kit mcp) — per-user install + stale local copies vs one server serving every consumer; hand-rolling the JSON-RPC protocol — the official SDK exists, is maintained by the MCP org, and its stateless HTTP mode is exactly this use case; an authoring-side MCP server — agents WRITING docs work inside the repo with generators + conventions (see the agent-authoring-kit issue), a protocol adds nothing there; SSE/session transport — needless statefulness for read-only tools behind a CDN.

Implementation steps (TDD)

1. Specs first (guard with if defined?(MCP) + add mcp to the gem's development dependencies): spec/docs_kit/mcp_tools_spec.rb — a pure DocsKit::McpTools module (testable without HTTP): list excludes unauthored pages; get returns markdown + graceful unknown-slug; search returns ranked hits. spec/docs_kit/mcp_server_spec.rb — server builds with brand/instructions; tools/list over handle_json returns the three tools; a tools/call round-trips.
2. Implement lib/docs_kit/mcp_tools.rb + lib/docs_kit/mcp_server.rb (both no-op gracefully when the mcp gem is absent) + the thin controller; c.mcp knob with spec.
3. Generator: commented # gem "mcp" Gemfile note + commented route with a one-line explanation; README "Add your docs to an agent" section with the Claude Code claude mcp add --transport http one-liner.
4. Dogfood: enable on the gem's own docs site; verify from Claude Code against localhost.
5. llms.txt advertisement line (in LlmsText, conditional on c.mcp + gem presence).

Verification gates

• bundle exec rspec — green with mcp in the bundle AND (CI matrix leg or local bundle exec rspec after bundle config without mcp-style exclusion) the suite still green without it — proving the optional-dependency gate.
• bundle exec rubocop — no offenses.
• Manual: claude mcp add --transport http docs http://localhost:3000/mcp then ask Claude to search the dogfood docs — tools appear and answer.
• Backwards compat: sites without the gem/route are byte-identical.

Out of scope

• Write tools (creating/editing pages over MCP) — the authoring loop is git.
• Auth (public docs; private-docs auth hook only when a real site needs it).
• MCP resources/prompts primitives (tools cover the consumption story; revisit on demand).