Enhancement: add typed provenance to composed codedb_context output

[justrach]

What changed

Add an optional structured output mode for codedb_context that identifies the origin and validation state of every composed section while preserving the current compact Markdown response as the default.

A candidate schema should distinguish at least:

• exact source snippets (path, line range, content hash/revision when available)
• parsed symbol declarations or bodies
• graph-derived neighbors/connectors, including edge type and reason
• generated .codedb/reader.md overview content, including validated/stale/malformed state and declared source set
• omissions caused by max_tokens or other caps

This is a focused follow-up to #531: JSON provenance shipped for codedb_search and codedb_symbol, but composed codedb_context output still lacks an equivalent machine-readable contract.

Why

Problem / failure mode

codedb_context intentionally mixes exact source, parsed structure, graph-derived context, ranked snippets, and an optional agent-authored reader.md. Human-readable headings help, and stale reader.md hashes are already detected, but MCP clients cannot reliably distinguish or threshold those evidence classes without parsing prose/comments.

That becomes important when generated orientation conflicts with current source, when a graph neighbor is only indirectly relevant, or when the token budget omits higher-cost evidence.

TencentDB Agent Memory's useful transferable principle is to preserve the layer and lifecycle of memory instead of flattening every result into one undifferentiated prompt. We should apply that principle to codedb evidence without adopting conversation storage or LLM extraction.

Reason for this approach

An opt-in structured schema gives clients trustworthy provenance while retaining the low-token Markdown path that existing agents use. It also extends the provenance model already shipped for search/symbol rather than creating a parallel API.

Constraints and trade-offs

• Default output and token economics must remain backward compatible.
• Hashes/revisions should validate content, not act as stable identifiers across unrelated projects.
• Structured output must not expose absolute host paths or files outside the project root.
• Provenance metadata can consume substantial tokens; compact encoding and optional fields are required.
• reader.md remains untrusted, agent-authored context even when its source hash is current.

Rejected alternatives

• Do not mark generated overview text as equivalent to exact source.
• Do not require every client to parse Markdown headings or HTML comments.
• Do not replace the existing compact output unconditionally.

Experiments required

E1: schema prototype

Prototype format=json or an MCP structured-content equivalent using the same conventions as search/symbol provenance. Test at least these states:

• no reader.md
• valid reader.md
• stale source hash
• malformed metadata
• graph neighbors present/absent
• max_tokens omissions
• deferred/incomplete indexes

Compare a section-oriented schema against a flat result list. Prefer the smallest schema that preserves evidence type, location, validation, score/reason, and omission metadata.

E2: token and latency cost

Across existing codedb_context benchmark tasks, measure:

• response bytes and estimated tokens
• serialization latency
• peak allocation/RSS
• cache behavior

Ship gate: default Markdown is unchanged; structured mode adds less than 5% execution latency and keeps metadata overhead below 10% of the returned evidence tokens on the median task.

E3: trust-conflict evaluation

Create tasks where:

• reader.md is stale
• a current but incorrect overview claim conflicts with exact source
• a graph neighbor looks relevant but does not contain the answer
• exact source is omitted under a tight token budget

Compare agent behavior with prose-only versus typed provenance. Measure whether the agent cites/reads exact source before acting and whether false conclusions decrease.

Ship gate: structured provenance must not lower task pass rate and should reduce unsupported conclusions on the conflict set.

E4: compatibility and schema stability

Run representative MCP clients against both formats. Add golden/schema tests for field names and omission semantics. Unknown future evidence types must be ignorable without breaking clients.

Acceptance criteria

• Structured context output distinguishes exact, parsed, graph-derived, and generated evidence.
• reader.md validation state and source set are explicit.
• Token-budget omissions are machine-readable.
• No absolute paths or out-of-root data are exposed.
• Existing default output remains byte-compatible unless separately justified.
• Token, latency, and trust-conflict experiment results are recorded.
• Unit and MCP E2E tests pass.

[justrach]

Implemented and acceptance-tested.

What landed

• Added opt-in codedb_context format=json output with a schema version, typed evidence sections, stable omission identifiers, requested/estimated token-budget metadata, and explicit reader-validation provenance. Markdown remains the default compatibility format.
• Structured evidence distinguishes request, generated, parsed, lexical-scope, graph, ranked, and exact-source evidence, with project-relative paths, source paths, line ranges, symbol metadata, relations, and explanatory reasons where applicable.
• Reader provenance now retains the validated source-file set plus declared/computed revisions in ready and stale states. Missing hashes are malformed rather than panic-prone.
• Hardened source validation so every declared source is canonicalized under the canonical project root; symlink escapes cannot hash files outside the repository.
• Preserved reader evidence in no-candidate responses, corrected budget accounting and lean-symbol estimates, and clamped import traversal depth to 1..64 before casting.

Evidence

• Focused issue-688 MCP tests pass, covering schema parsing, typed reader/symbol/graph provenance, stable omissions, no-candidate output, linked-document source paths, and byte-identical default versus explicit Markdown output.
• Cross-corpus provenance probe: 12 ok, 0 unsupported, 12 runs; reader states included missing, ready, and stale, with parse coverage median 1.0 and minimum approximately 0.8088.
• Median raw output was 376 bytes versus 1506.5 modeled structured JSON bytes (median metadata overhead ratio 3.0939), so structured output remains opt-in. Median modeled serialization latency was 49.25 µs.
• zig build test passes with the project Zig (0.17.0-dev.813+2153f8143).
• scripts/e2e_mcp_test.py --binary zig-out/bin/codedb --project <repo>: 20/20 passed.

Closing as completed.