feat: index freshness trust bundle

.changeset/index-freshness-trust-bundle.md

@@ -0,0 +1,5 @@
+---
+"@stainless-code/codemap": patch
+---
+
+Add `index_freshness` metadata on `context`, MCP tool responses, HTTP headers, and boot stderr warnings so agents can detect commit drift, pending watcher sync, and disk-ahead-of-index states before trusting structural queries.

docs/agents.md

@@ -103,6 +103,8 @@ Recipe ids cited in the playbook are machine-validated in tests against the live
 
 ## MCP tool allowlist
 
+**`context.index_freshness`** — session bootstrap includes index-level freshness metadata: `commit_drift` (HEAD ≠ `last_indexed_commit`), `pending_sync` (watcher debounce queue or in-flight reindex), optional disk-drift counts when watch is off, and a single `warning` string when agents should pause or re-index. **MCP:** array-shaped JSON tools (`query`, …) keep row payloads verbatim and append a second `content` block prefixed `@codemap/index_freshness`; object-shaped tools merge `index_freshness` inline. **HTTP:** `POST /tool/*` adds `X-Codemap-Pending-Sync`, `X-Codemap-Commit-Drift`, and `X-Codemap-Warning` headers without changing JSON bodies; **`GET /health`** includes full cheap `index_freshness` when the DB is readable. Complements per-file `validate` / snippet `stale`. See [`architecture.md` § Context wiring](./architecture.md).
+
 **`CODEMAP_MCP_TOOLS`** — comma-separated snake_case MCP tool names. When set, only listed tools register (stderr lists the active set). Unknown names are ignored with a warning. Unset = all tools (default). **`query_batch`** registers only when listed or when unset (eval ablation).
 
 Example: `CODEMAP_MCP_TOOLS=query,context,show codemap mcp --no-watch`

docs/architecture.md

@@ -129,7 +129,7 @@ A local SQLite database (`.codemap/index.db`) indexes the project tree and store
 
 **PR-comment wiring:** **`src/cli/cmd-pr-comment.ts`** (argv — `<input-file>` (or `-` for stdin) + `--shape audit|sarif` + `--json`) + **`src/application/pr-comment-engine.ts`** (engine — `renderAuditComment` / `renderSarifComment` / `detectCommentInputShape`). Renders an audit-JSON envelope or SARIF doc as a markdown PR-summary comment; designed for surfaces SARIF→Code-Scanning doesn't cover (private repos without GHAS, aggregate audit deltas without `file:line` anchors, bot-context seeding). Output: bare markdown by default; `--json` envelope `{markdown, findings_count, kind}` for action.yml steps. Audit-mode groups by delta with `<details>` sections (added + removed); SARIF-mode groups by `ruleId`. Lists >50 entries collapse to `… and N more`. v1.0 ships the (b) summary-comment shape; (c) inline-review comments deferred per Q4 of [`plans/github-marketplace-action.md`](./plans/github-marketplace-action.md).
 
-**Context wiring:** **`src/cli/cmd-context.ts`** (argv + render) + **`src/application/context-engine.ts`** (engine — **`buildContextEnvelope`**, **`classifyIntent`**, `ContextEnvelope` type). `buildContextEnvelope` composes the JSON envelope from existing recipes (`fan-in` for `hubs`, `markers` SELECT for `sample_markers`, `QUERY_RECIPES` map for the catalog). **`classifyIntent`** maps `--for "<text>"` to one of `refactor | debug | test | feature | explore | other` via regex against the trimmed input; whitespace-only intents are rejected. `--compact` drops `hubs` + `sample_markers` and emits one-line JSON; otherwise pretty-prints with 2-space indent.
+**Context wiring:** **`src/cli/cmd-context.ts`** (argv + render) + **`src/application/context-engine.ts`** (engine — **`buildContextEnvelope`**, **`classifyIntent`**, `ContextEnvelope` type). `buildContextEnvelope` composes the JSON envelope from existing recipes (`fan-in` for `hubs`, `markers` SELECT for `sample_markers`, `QUERY_RECIPES` map for the catalog) and **`index_freshness`** via **`src/application/index-freshness.ts`** (`computeIndexFreshness` with disk-drift). **`classifyIntent`** maps `--for "<text>"` to one of `refactor | debug | test | feature | explore | other` via regex against the trimmed input; whitespace-only intents are rejected. `--compact` drops `hubs` + `sample_markers` and emits one-line JSON; otherwise pretty-prints with 2-space indent. Product-shape constraint: [No split-brain incremental index](./roadmap.md#floors-v1-product-shape).
 
 **Impact wiring:** **`src/cli/cmd-impact.ts`** (argv — `<target>` + `--direction up|down|both` + `--depth N` + `--via dependencies|calls|imports|all` + `--limit N` + `--summary` + `--json`; bootstrap absorbs `--root`/`--config`) + **`src/application/impact-engine.ts`** (engine — `findImpact({db, target, direction?, via?, depth?, limit?})`). Pure transport-agnostic walker over the calls + dependencies + imports graphs; CLI / MCP / HTTP all dispatch the same engine function via `tool-handlers.ts`'s `handleImpact`. Target auto-resolves: contains `/` or matches `files.path` → file target; otherwise symbol (case-sensitive). Walks compatible backends per resolved kind: **symbol** → `calls` (callers / callees by `caller_name` / `callee_name`); **file** → `dependencies` (`from_path` / `to_path`) + `imports` (`file_path` / `resolved_path`, `IS NOT NULL` filter). `--via <b>` overrides; mismatched explicit choices land in `skipped_backends` (no error — agents see why their backend selection yielded fewer rows than expected). One `WITH RECURSIVE` per (direction, backend) combo with cycle detection via path-string `instr` check (SQLite has no native cycle predicate); JS-side merge + dedup by `(direction, kind, name?, file_path)` keeping the shallowest depth. `--depth 0` uses an unbounded sentinel (`UNBOUNDED_DEPTH_SENTINEL = 1_000_000`); cycle detection + `LIMIT` keep cyclic graphs cheap regardless. Termination reason classification: `limit` (truncated) > `depth` (any node sat at the cap) > `exhausted`. Result envelope: `{target, direction, via, depth_limit, matches: [{depth, direction, edge, kind, name?, file_path}], summary: {nodes, max_depth_reached, by_kind, terminated_by}, skipped_backends?}`. `--summary` blanks `matches` (transport bandwidth saver) but preserves `summary.nodes` so CI gates (`jq '.summary.nodes'`) still see the count. SARIF / annotations not supported (graph traversal, not findings — the parser accepts the flag combos but the engine only emits JSON).
 

docs/roadmap.md

@@ -78,11 +78,11 @@ Long-running MCP / HTTP sessions dominate agent workflows; one-shot CLI keeps th
 - [ ] **MCP shared daemon per project** — one watcher + one SQLite writer per indexed root; Unix socket / named pipe so concurrent agent sessions share a live index instead of each spawning watchers and contending on WAL. Complements perf item **6.1** (read pool) but is a separate write-side + lifecycle concern. Effort: L.
 - [ ] **Rich `context` composer** — expand session bootstrap beyond counts + recipe catalog: ranked hub files, sample markers, optional inlined signatures, fan-in leaders, and links to high-value recipes for the detected intent. Goal: one bootstrap call replaces a common show → explore chain on session start. Effort: M.
 - [ ] **Codebase map in bootstrap responses** — hash-stable structural summary (top hubs, CLI entry hints, schema version, index freshness) auto-included in `context` / MCP initialize payload. Opt-out via flag. Effort: S–M.
-- [ ] **Index staleness surfacing** — when the watcher has pending files (debounce window), MCP/HTTP responses and `context` carry an explicit pending-sync indicator; align with `validate` staleness rows. Effort: S.
+- [x] **Index staleness surfacing** — `index_freshness.pending_sync` on `context`, MCP tool metadata, and HTTP headers when the watcher debounce queue or in-flight reindex is active. Shipped [#149](https://github.com/stainless-code/codemap/pull/149).
 - [ ] **Adaptive output budgets** — scale explore/trace/node snippet char caps and row limits from indexed file/symbol counts so large trees do not blow token budgets. Effort: S.
 - [ ] **MCP session lifecycle hygiene** — idle timeout, client disconnect detection, graceful watcher shutdown on last client; avoid orphaned watchers after agent host crashes. Effort: S–M.
 - [ ] **`agents init` uninstall (teardown)** — symmetric inverse of init for failed pilots, template mistakes, or leaving a repo: remove codemap-managed MCP entries, pointer sections, and IDE symlinks only (same scoped paths as init; never delete user-authored `.agents/` siblings). `--target` filter, `--yes` non-interactive. Not the happy-path docs story — adoption stays `init --mcp --git-hooks` + committed `.agents/`. Effort: S.
-- [ ] **HEAD / index freshness warning** — stderr or `context` field when `meta.last_indexed_commit` ≠ current `HEAD`, or when the checkout (including linked git worktrees) may not match the tree the index was built for. Effort: S.
+- [x] **HEAD / index freshness warning** — `index_freshness.commit_drift` + `warning` on `context` / tool metadata; boot stderr on `codemap mcp` / `serve` when concerns remain after prime. Shipped [#149](https://github.com/stainless-code/codemap/pull/149).
 
 ### Recipe & audit enrichment
 

src/application/context-engine.ts

@@ -1,6 +1,8 @@
 import { getMeta, SCHEMA_VERSION } from "../db";
 import type { CodemapDatabase } from "../db";
 import { CODEMAP_VERSION } from "../version";
+import { computeIndexFreshness } from "./index-freshness";
+import type { IndexFreshness } from "./index-freshness";
 import { QUERY_RECIPES } from "./query-recipes";
 
 /**
@@ -32,6 +34,7 @@ export interface ContextEnvelope {
     content: string;
   }[];
   recipes: { id: string; description: string }[];
+  index_freshness: IndexFreshness;
   intent?: {
     input: string;
     classified_as: string;
@@ -136,6 +139,7 @@ export function buildContextEnvelope(
       id,
       description: meta.description,
     })),
+    index_freshness: computeIndexFreshness(db, { include_disk_drift: true }),
   };
 
   if (!opts.compact) {

src/application/http-server.test.ts

@@ -142,6 +142,8 @@ describe("http-server — POST /tool/query", () => {
     });
     expect(r.status).toBe(200);
     expect(r.headers.get("content-type")).toContain("application/json");
+    expect(r.headers.get("X-Codemap-Pending-Sync")).toBe("false");
+    expect(r.headers.get("X-Codemap-Commit-Drift")).toBe("false");
     expect(r.json).toEqual([
       { name: "bar", kind: "const" },
       { name: "foo", kind: "function" },

src/application/http-server.ts

@@ -12,6 +12,12 @@ import {
   getTsconfigPath,
   initCodemap,
 } from "../runtime";
+import {
+  readCheapIndexFreshness,
+  resolveTransportIndexFreshness,
+  warnIndexFreshnessToStderr,
+} from "./index-freshness";
+import type { IndexFreshness } from "./index-freshness";
 import { listResources, readResource } from "./resource-handlers";
 import {
   affectedArgsSchema,
@@ -118,6 +124,35 @@ const TOOL_NAMES = [
 export async function runHttpServer(opts: HttpServerOpts): Promise<void> {
   await bootstrapForServe(opts);
 
+  let stopWatch: (() => Promise<void>) | undefined;
+  let watchReady: Promise<void> = Promise.resolve();
+  if (opts.watch === true) {
+    const prime = createPrimeIndex({ quiet: false, label: "codemap serve" });
+    const handle = runWatchLoop({
+      root: getProjectRoot(),
+      excludeDirNames: getExcludeDirNames(),
+      recipesWatchPrefix: resolveRecipesWatchPrefix(getProjectRoot()),
+      debounceMs: opts.debounceMs ?? DEFAULT_DEBOUNCE_MS,
+      onPrime: async () => {
+        try {
+          await prime();
+        } finally {
+          warnIndexFreshnessToStderr("codemap serve");
+        }
+      },
+      onChange: createReindexOnChange({
+        quiet: false,
+        label: "codemap serve",
+      }),
+    });
+    stopWatch = handle.stop;
+    watchReady = handle.ready;
+  } else {
+    warnIndexFreshnessToStderr("codemap serve");
+  }
+
+  await watchReady;
+
   const server = createServer((req, res) => {
     handleRequest(req, res, opts).catch((err: unknown) => {
       const msg = err instanceof Error ? err.message : String(err);
@@ -138,30 +173,6 @@ export async function runHttpServer(opts: HttpServerOpts): Promise<void> {
     });
   });
 
-  let stopWatch: (() => Promise<void>) | undefined;
-  if (opts.watch === true) {
-    try {
-      const handle = runWatchLoop({
-        root: getProjectRoot(),
-        excludeDirNames: getExcludeDirNames(),
-        recipesWatchPrefix: resolveRecipesWatchPrefix(getProjectRoot()),
-        debounceMs: opts.debounceMs ?? DEFAULT_DEBOUNCE_MS,
-        onPrime: createPrimeIndex({ quiet: false, label: "codemap serve" }),
-        onChange: createReindexOnChange({
-          quiet: false,
-          label: "codemap serve",
-        }),
-      });
-      stopWatch = handle.stop;
-    } catch (err) {
-      // Watcher boot threw AFTER `server.listen()` resolved — close
-      // the listener so we don't leak an orphaned HTTP socket on a
-      // failed boot. Caught by CodeRabbit on PR #47.
-      await new Promise<void>((res) => server.close(() => res()));
-      throw err;
-    }
-  }
-
   await new Promise<void>((resolve) => {
     const shutdown = (signal: string) => {
       // eslint-disable-next-line no-console -- intentional shutdown log on stderr
@@ -248,10 +259,16 @@ export async function handleRequest(
 
   // Liveness probe — auth-exempt so monitoring works without the token.
   if (method === "GET" && path === "/health") {
+    const freshness = readCheapIndexFreshness();
+    applyIndexFreshnessHeaders(res, freshness);
     return writeJson(
       res,
       200,
-      { ok: true, version: opts.version },
+      {
+        ok: true,
+        version: opts.version,
+        ...(freshness !== null ? { index_freshness: freshness } : {}),
+      },
       opts.version,
     );
   }
@@ -385,11 +402,36 @@ async function readJsonBody(
  * 4xx / 5xx with `{"error": "..."}` (same shape `codemap query --json`
  * prints on failure — agents and CLI consumers unwrap identically).
  */
+function applyIndexFreshnessHeaders(
+  res: ServerResponse,
+  freshness: IndexFreshness | null,
+): void {
+  if (freshness === null) return;
+  res.setHeader(
+    "X-Codemap-Pending-Sync",
+    freshness.pending_sync ? "true" : "false",
+  );
+  res.setHeader(
+    "X-Codemap-Commit-Drift",
+    freshness.commit_drift ? "true" : "false",
+  );
+  if (freshness.warning !== null) {
+    res.setHeader("X-Codemap-Warning", freshness.warning);
+  }
+}
+
 function writeToolResult(
   res: ServerResponse,
   result: ToolResult,
   version: string,
 ): void {
+  const freshness = result.ok
+    ? result.format === "json"
+      ? resolveTransportIndexFreshness(result.payload)
+      : readCheapIndexFreshness()
+    : null;
+  applyIndexFreshnessHeaders(res, freshness);
+
   if (!result.ok) {
     return writeJson(
       res,