feat(mcp): optional built-in MCP server — list/get/search docs over Streamable HTTP

[mhenrixon]

Closes #20.

What & why

Developers increasingly read docs through agents. llms.txt covers fetch-style consumption, but MCP (Model Context Protocol) is the native protocol: a reader adds https://docs.example.com/mcp to Claude Code / Claude.ai / Cursor once and the docs become first-class tools (search, targeted page retrieval) instead of scraped text. This makes every current and future docs-kit site agent-queryable for free.

A read-only, stateless Streamable HTTP MCP endpoint built on the official mcp Ruby SDK, mounted by the host app, off unless both the gem is present and the route is drawn.

The three tools

All over the same registry the docs render from — an agent queries live docs, never a stale scrape:

Tool	Returns
list_pages	every authored page — slug, title, group, url
get_page(slug:)	the page's Markdown twin (the same .md twin /llms.txt links), or a graceful not-found listing valid slugs
search_docs(query:)	ranked DocsKit::SearchIndex hits — page_title, section_title, url, snippet (with the <mark> highlight stripped to plain text)

Architecture — mirrors the existing llms.txt/search split

• DocsKit::McpTools — the pure, Rails-free, mcp-free core (all three functions over LlmsText.pages / MarkdownExport / SearchIndex). Fully unit-testable without HTTP or the SDK.
• DocsKit::McpServer — wraps those into an MCP::Server via define_tool, named from c.brand, instructions from c.tagline + a /llms.txt pointer. No-ops (returns nil) when the gem is absent.
• DocsKit::McpController — thin: POST /mcp → server.handle_json(request.body.read); owns only the Rails view context (for the twin render) + the read-only method policy. GET/DELETE → 405.

Optional-dependency gate

docs-kit adds mcp to no gemspec list. Configuration#mcp_enabled? gates on the c.mcp knob (default true) AND the gem being loadable. The mcp gem lives in its own bundler group, and a new CI leg installs --without mcp and runs the suite — proving the feature no-ops (the MCP specs self-skip) when the gem is absent. A site that never bundles mcp is byte-identical to before this PR.

Discovery + opt-in wiring

• /llms.txt grows a final ## MCP line advertising the endpoint when enabled, so agents discover it.
• The install generator draws the /mcp route commented out (opt-in — the gem is optional); config/initializers/docs_kit.rb and the docs-kit new template document the gem "mcp" + c.mcp opt-in.
• README: a new "Add your docs to an agent (MCP)" section with the claude mcp add --transport http one-liner.

Dogfooded end-to-end

Enabled on the gem's own docs/ app (real /mcp route + gem "mcp") and verified from curl against a booted server — through the real Rails render path, not just unit tests:

• tools/list → the three tools
• search_docs("theme switcher") → top hit Components → ThemeSwitcher with a correct anchored absolute URL, snippet as plain text
• get_page("installation") → the real Markdown twin; get_page("nope") → graceful not-found
• GET/DELETE /mcp → 405
• /llms.txt → the ## MCP advertisement line

Test plan

• spec/docs_kit/mcp_tools_spec.rb — the pure core (runs without the mcp gem): authored-only listing, twin + unknown-slug, ranked hits with <mark> stripped
• spec/docs_kit/mcp_server_spec.rb — the real SDK via handle_json: tools/list + each tools/call round-trips; .build no-ops gem-absent (self-skips without the gem)
• spec/docs_kit/mcp_controller_spec.rb — source-wiring (the class is Rails-only), mirrors llms_controller_spec
• configuration_spec / llms_text_spec / install_generator_spec — the knob + mcp_enabled?, the advertisement line, the commented route

Verification

• bundle exec rspec — 448 examples, 0 failures, 96% line coverage
• bundle exec rubocop — no offenses
• Suite green without the mcp gem (bundle config set --local without mcp) — the optional-dependency gate
• Dogfooded end-to-end against docs/ (curl → every tool answers)
• Backwards compatible — a site without the gem/route is byte-identical

Out of scope (per the issue)

Write tools (authoring is git), auth (public docs; a host wraps the route), and MCP resources/prompts primitives (tools cover the consumption story).