feat(framework): pnpm local docs, a local docs index for search_docs evals

[barryroodt]

Stack 4/5 for #128. Base: #195.

pnpm local docs up   --docs <path-to-supabase-monorepo>
pnpm local docs seed             # full embed, ~$0.12 OpenAI, asks first
pnpm local docs api  [--port N]  # foreground, keep it running
pnpm local docs down
pnpm local run <eval> --content-api http://127.0.0.1:3001/docs/api/graphql --mcp <checkout>

Lets a docs edit be measured end to end. The docs checkout is yours, passed via --docs, so there is no submodule and no patching: edit a page there, re-seed, re-run.

Details a reviewer should not have to infer

The 443xx port block is off the evals local-stack range (54321+) and below the macOS ephemeral range (49152+), where a transient outbound socket stealing a listen port is a real failure mode we hit. Files are copied rather than symlinked so this works on Windows.

content-api-server.ts serves the docs app's own GraphQL route over plain node:http, run with the docs checkout's tsx so the route's TS and its react-server condition resolve. It binds every interface and advertises loopback, matching platform-lite in tools mode: a sandboxed agent reaches host-side services through host.docker.internal, which arrives on the bridge interface, not loopback. Flagging it explicitly since it is the one thing here that widens exposure on a dev machine.

The Sentry stub is required, not cosmetic. Outside Next's instrumentation the real package resolves without captureException and every request crashes.

Why --content-api is gated

Ungated it is a trap: the server would query production docs while the receipt recorded the local URL, so a paid run would measure the wrong world and report the right one.

The gate consults the MCP_SERVER_VERSION pin, because the --content-api-url flag shipped in v0.10.0 (supabase/mcp#343), and only asks for --mcp when the pin cannot honour it. Verified against the tags rather than the merge date: 6fcaaa3 is an ancestor of mcp-server-supabase-v0.10.0 and is in neither v0.9.0 nor v0.8.3.

It probes the built server for the flag, not the env var, because the flag is what we pass. createConfig forwards the URL in argv, since a CLI agent spawns that command inside the container where our environment is not inherited. A build with only the env fallback would silently ignore the override, and there is a check for exactly that build. The needle is content-api-url without dashes, which is how parseArgs registers the option; the dashed form appears nowhere in the built server.

Version handling is deliberately strict, both directions chosen against the expensive failure:

• Malformed versions throw. Reading 0.10.foo loosely as 0.10.0 would report "capable" and buy the bad run.
• No prerelease counts. Semver defines precedence, not content, so a 0.11.0-beta could come off a branch that forked before the flag landed. Refusing one costs a flag; trusting one wrongly costs a run.

Two deliberate limitations

docs seed is full-embed only. The upstream incremental path has known bugs: guide checksums are never set so every guide re-embeds, and a source skipped for missing credentials has its still-valid rows purged. It waits for those fixes upstream.

CLI output is buffered and replayed only on failure, because supabase start prints ANON_KEY, SERVICE_ROLE_KEY, SECRET_KEY and JWT_SECRET on every run and those do not belong on a screen recording. Dropping stdout is a stream boundary, not a redaction regex, so there is no pattern to keep in step with the CLI's output shapes.

Verified

• smoke 27/27, 8 new
• core 115/115, sandbox 39 passed
• tsc --noEmit and biome check clean

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored		Aug 13, 2026 3:30pm