mcp: guidance ranking is coupled to corpus-wide term statistics, so regeneration flips a locked expectation

[rickylabs]

Summary

packages/mcp/tests/guidance-evaluation_test.ts:25 fails whenever the shipped agent-docs corpus is
regenerated, because the locked guidance ranking is coupled to corpus-wide term statistics rather
than to the content of the pages being ranked.

Surfaced by #1531 / PR #1608 (deterministic corpus freshness). That PR is correctly scoped to the
stale-snapshot defect and does not touch the fixture — git diff --quiet 0551ff592 9e9a9b6f6 -- packages/mcp/tests/
is clean. This issue owns the ranking decision so the corpus fix does not land a required-test red.

The exact case

Fixture packages/mcp/tests/fixtures/guidance-evaluation.json:

{
  "intent": "pick direct application ownership versus a reusable integration",
  "mode": "exact",
  "rankingAxis": "score",
  "expected": [
    "pages/data-persistence/how-to/use-a-second-database#4-decision-rule-direct-use-vs-reusable-databaseadapter-wrapper",
    "pages/data-persistence/how-to/use-a-second-database#unsupported-by-netscript-supported-by-prisma-libsql-turso-example",
    "pages/data-persistence/how-to/use-a-second-database#connect-an-external-database-by-hand"
  ]
}

Rank 2 (third entry) changes when the corpus is regenerated:

	anchor
expected (locked)	pages/data-persistence/how-to/use-a-second-database#connect-an-external-database-by-hand
actual (fresh corpus)	pages/explanation/plugin-system#a-plugin-is-a-thin-layer-over-a-core-package

Result: deno task test → exit 1, 3300 passed (624 steps) | 2 failed | 17 ignored.

Why this is not fixture rot, and not a content bug

Three facts, each executed:

1. The expected anchor still exists. Connect an external database by hand is present in the base
   corpus, in the fresh corpus, and at docs/site/data-persistence/how-to/use-a-second-database.md:195.
   The expectation did not lock onto a deleted heading.
2. Neither competing source page changed. Both
   docs/site/data-persistence/how-to/use-a-second-database.md and docs/site/explanation/plugin-system.md
   are byte-unchanged between base 0551ff592 and head 9e9a9b6f6.
3. Therefore the flip comes from elsewhere in the corpus. The corpus grew 174 → 178 files and 74
   docs/site files had moved on since the stale snapshot. Changing other documents changed the
   relative score of these two.

That is the finding: a locked exact-ranking expectation is sensitive to edits in unrelated pages.
Any future docs PR that regenerates the corpus can flip it, and the failure will name two pages that
the PR never touched.

Why the golden must not simply be updated

The previously-green expectation was produced by a corpus that was six hours stale and missing 74
files' worth of changes — it is not a trustworthy oracle. But the fresh result is not self-evidently
correct either: on the query's own wording, "connect an external database by hand" is about an
application owning a datasource directly, which is what "direct application ownership" asks for,
while "a plugin is a thin layer over a core package" is about plugin architecture. Re-locking to
whatever the fresh corpus emits would bury a possible retrieval-quality regression behind a green test.

Scope

Determine a narrow, deterministic fix so current exact guidance stays green without re-locking blind.
Candidate directions, to be decided by evidence:

• a deterministic tie-break when scores are close (measure the actual score gap first — if the two
  candidates are separated by a hair, ordering is currently arbitrary and that is the real defect);
• concept/metadata weighting so an on-topic how-to anchor is not displaced by an architectural
  explainer for an ownership-decision query;
• narrowing the fixture's mode: "exact" to the ranks that are genuinely stable, with a recorded
  reason — acceptable only if the score evidence shows the third rank is legitimately unstable.

Explicitly not in scope: #1260 (corpus presence of SDK guidance — correctly closed) and #1410
(future hybrid retrieval RFC). Current exact deterministic guidance must remain green independent of both.

Acceptance

• The measured score gap between the two candidate anchors for this intent is recorded in the issue or PR.
• deno task test exits 0 with a regenerated corpus.
• A negative control demonstrates the guidance-evaluation test can still fail — raw non-zero exit shown.
• If the golden is changed, the PR states which rank changed and why the new order is correct, not merely current.
• Regenerating the corpus a second time on the same tree does not change the ranking result.

Blocking status

The 0.0.6 stable cut cannot absorb a new required-test red. check-test on main is red today only
for the #1589 JSDoc finding (#1612, fixed by #1614). Once #1614 lands, check-test returns green —
unless the corpus fix merges first without this resolved.