fix(mcp): stabilize close-score guidance ranking

[rickylabs]

Summary

Make close-score guidance ordering deterministic across unrelated corpus-wide statistic changes, based on the measured near-tie in #1615. This keeps the exact golden meaningful without treating either the stale or freshly generated corpus as an oracle.

Scope

• Archetype / area: packages/mcp Archetype 2 guidance ranking
• Closes mcp: guidance ranking is coupled to corpus-wide term statistics, so regeneration flips a locked expectation #1615

Slices

• S0 measure both candidates, lock the design, and activate the harness — 0b73d7333
• S1 implement transitive close-score grouping and prove base/fresh corpus behavior — b943392d7
• S2 prove the negative control and complete the original gate evidence — 5d7ca0f46, c86a4080f, 33e0ef9d0, 543a7b1e9
• S3 restore route-ranked confidence and document the tie-break limits — 7933e044c, 5a0e7303e

Score measurement

Corpus	Direct score	Plugin score	Direct − plugin	Absolute gap
base 6aee2b414	11.80343776647673	11.721196841503339	+0.08224092497339086	0.08224092497339086
fresh PR #1608 9e9a9b6f6	11.502244339113766	11.804224537299888	-0.3019801981861221	0.3019801981861221

The fresh gap is about 2.6% of either score and reverses solely through unrelated corpus changes. Direction 1 is selected: deterministic tie handling for close cross-document scores. Concept weighting is rejected because the gap is not wide; fixture narrowing is rejected because rank three can remain deterministic.

Ordering and confidence semantics

The implementation forms leader-anchored, same-route groups within a 0.5 score gap, avoiding a non-transitive epsilon comparator. Corpus-wide document frequency and average length are the unstable inputs, so different documents inside a statistically tied group are ordered by stable slug. Sections within one document retain raw-score order because those candidate-local relevance differences remain meaningful.

The resulting cross-document order is deterministic but arbitrary: the expected data-persistence anchor wins this tie because its slug sorts before explanation, not because this PR proves that document deserves higher semantic rank. The exact golden is unchanged because a stable total order is justified by the measured tie.

This is deterministic for a fixed candidate set, not insertion-stable under corpus growth. If a third document enters the same leader-anchored 0.5 group and its slug sorts before both current candidates, the locked order can change again. Such a new close candidate is a real membership change and requires semantic review of the fixture/scorer.

There is a second, sibling instability that needs no membership change at all: an existing member can exit the band. Verified — 10.0 / 9.5 groups by slug, 10.0 / 9.4999999 reverts to score order. So this fix converts an instability at gap ~= 0 into an instability at gap ~= 0.5; it is a large improvement, not an elimination. Measured headroom on the real pair: the band span moved from 0.2271214530129786 (base corpus) to 0.3019801981861221 (fresh), i.e. +0.0748587451731435 from a single regeneration, leaving 0.1980198018138779 of headroom — about 2.6x the observed per-regeneration movement. A future regeneration of similar magnitude can re-red this gate with the fix in place.

Confidence follows the post-order route-priority winner, preserving pre-PR route-promotion behavior. Close-score ordering can replace the numeric leader only with an entry at most 0.5 lower, so confidence can differ only when the winning score is within 0.5 of thresholds 8 or 24. A regression test proves that an unhinted global scorer yields high, while activating a route promotes a lower scorer and yields medium.

Validation

Original evidence:

• score instrumentation — exit 0; instrumented and public GuidanceIndex.find() top-three orders agree on base and fresh corpora
• fresh sibling corpus — all 8 fixture cases pass across 2 independently constructed corpora × 2 query reruns; base/fresh exact top three agree
• negative control at 5d7ca0f46 — raw exit 1; 0 passed, 1 failed; deliberate expected llms#deliberate-negative-control versus actual llms#getting-started
• negative-control revert c86a4080f — exit 0; 1 passed, 0 failed; fixture restored exactly
• full repository tests — exit 1; 3321 passed (624 steps), 1 failed, 17 ignored; the sole failure is the dispatched-base Fresh defer canary permits split SDK cache-provider singleton #1589 published-JSDoc finding; guidance passes
• quality:gate — exit 0
• package doc lint — exit 1 at this head, private-type-ref on SchemaViewName in get-operation-schema-flow.ts, a file this PR does not touch. Confirmed exit 1 on origin/main without this PR, so it is pre-existing and not introduced here.
• Package-wide scoped fmt/lint wrappers cannot produce a verdict for packages/mcp at all (tooling: deno fmt cannot verify packages/mcp — a deliberately malformed test fixture aborts config parsing #1618) — they abort on Failed to parse "workspace" configuration from the deliberately malformed doctor fixture. Single-file deno fmt --check and deno lint on the two touched files were used instead: both exit 0.

S3 review cycle:

• focused retrieval + guidance evaluation — exit 0; 8 passed, 0 failed
• packages/mcp package tests — exit 0; 136 passed, 0 failed
• guidance evaluation after fixture restoration — exit 0; 1 passed, 0 failed
• scoped check wrapper — exit 0; 115 files, 0 failed batches/findings
• scoped lint wrapper — exit 0; 115 files, 0 findings
• exact touched-file deno fmt --check — exit 0
• repeated uncommitted negative control — raw exit 1; 0 passed, 1 failed; fixture restored before package tests and commit

No package-wide format verdict is claimed. The deliberately malformed packages/mcp/tests/fixtures/doctor/broken/deno.json prevents the formatter from producing a trustworthy package-scoped verdict; this is pre-existing on origin/main and tracked by #1618. This PR uses exact touched-file format checks only.

Harness

• Run dir: .llm/runs/fix-1615-ranking--leaf/
• Phase: impl; PLAN-EVAL: N/A is recorded because the issue supplied the complete contract, boundaries, and gates and measurement resolved the only open decision.
• Do not merge or mark ready until the orchestrator's separate native Opus 5 read-only IMPL-EVAL completes. Fable is prohibited.

Drift / Debt

• The live fresh PR fix(mcp): keep shipped agent docs corpus current #1608 head moves the plugin explainer to second rather than only the issue's abbreviated third-rank replacement; the measured near-tie and chosen remedy are unchanged.
• The initial confidence implementation used the global maximum; S3 restores route-winner confidence and adds direct regression coverage.
• Package-wide formatting is a tooling: deno fmt cannot verify packages/mcp — a deliberately malformed test fixture aborts config parsing #1618 non-verdict; only exact touched-file format evidence is reported.
• The only full-repository test failure is the known Fresh defer canary permits split SDK cache-provider singleton #1589 finding owned by fix(cli): main fails the published-JSDoc codename guard — bare issue reference #1589 in netscript-web-runtime-closure.ts #1612 / PR fix(cli): replace published JSDoc issue codename #1614; this leaf does not absorb it.
• Pre-existing MCP-A6-V2-SHAPE debt is neither deepened nor resolved.
• No generated assets, docs content, corpus tooling, fixture golden, or lockfiles changed. No CLI/scaffold E2E ran.

Definition of Done

• Both candidate scores and their gap are recorded with reproducible evidence.
• All eight guidance cases pass on the shipped and freshly generated corpus, twice deterministically.
• A throwaway expectation perturbation makes the guidance gate return a raw non-zero exit and is reverted.
• Repository tests have no mcp: guidance ranking is coupled to corpus-wide term statistics, so regeneration flips a locked expectation #1615 guidance failure; the sole pre-existing failure is identified precisely.
• Scoped check/lint and exact touched-file format checks pass without lock or generated-asset churn.
• Mandatory separate-session IMPL-EVAL is complete. Native Claude Opus 5, read-only, separate session, opposite-family to this Codex-authored change; PASS at immutable head 5a0e7303e93413a80999dbf59ba9f8191798e1ee, no blocking findings. Verdict posted at fix(mcp): stabilize close-score guidance ranking #1617 (comment). The feature diff is byte-identical at the current merged head 32d3e1842 (git diff origin/main...HEAD -- packages/ — 6649 bytes before and after the main merge), so the verdict remains bound to unchanged code.

issue: 1615
entries:
  - box-index: 1
    evidence: "Base direct/plugin scores 11.80343776647673 / 11.721196841503339 (gap 0.08224092497339086); fresh scores 11.502244339113766 / 11.804224537299888 (gap 0.3019801981861221)."
  - box-index: 2
    evidence: "All 8 cases pass against the fresh PR #1608 corpus across 2 constructions x 2 reruns; full repo tests have no guidance failure and retain only the documented unrelated #1589 failure."
  - box-index: 3
    evidence: "Commit 5d7ca0f46 deliberately perturbed an expectation and the raw focused gate exited 1; c86a4080f reverted it. At S3 head the uncommitted negative control again exited 1 and was restored before package tests."
  - box-index: 4
    evidence: "The golden is unchanged: the measured near-tie selected deterministic close-score handling, not a relock to the current corpus."
  - box-index: 5
    evidence: "Two independently constructed indexes over the same fresh generated tree each passed all 8 cases twice, with identical exact top-three output."

[rickylabs]

[PHASE: PLAN]

S0 locked the #1615 implementation decision from measured evidence in 0b73d7333.

Evidence

• Base: direct 11.80343776647673, plugin 11.721196841503339, gap +0.08224092497339086.
• Fresh PR fix(mcp): keep shipped agent docs corpus current #1608: direct 11.502244339113766, plugin 11.804224537299888, gap -0.3019801981861221.
• The fresh absolute gap is 0.3019801981861221; direction 1 (deterministic close-score handling) is locked.
• PLAN-EVAL: N/A is recorded because the owner supplied the complete contract/boundaries/gates and measurement resolved the only open decision. Separate IMPL-EVAL remains mandatory.

Next

Implement leader-anchored, transitive close-score groups; preserve numeric order within the same source document; prove all eight cases on base and fresh corpora.

[rickylabs]

[PHASE: IMPL]

S1 landed as b943392d7: close-score guidance candidates are grouped transitively by a leader-anchored 0.5 band, documents use stable slug identity, and sections within one document retain numeric score order.

Evidence

• Focused MCP guidance: exit 0 — 7 passed, 0 failed.
• Requested deno task test --filter guidance: exit 0 — 13 passed, 0 failed, 3326 filtered.
• Fresh PR fix(mcp): keep shipped agent docs corpus current #1608 corpus loaded directly from 9e9a9b6f6: 8/8 fixture cases passed, across 2 independently constructed corpus instances × 2 query reruns.
• Fresh exact top three now match the unchanged golden: decision rule → unsupported example → direct ownership.
• Raw scores remain honest and recorded: direct 11.502244339113766, plugin 11.804224537299888; the stable tie policy, not a hidden score rewrite, determines their close-band order.
• No fixture, docs, generated asset, corpus tool, public export, or lockfile changed.

Slice review

Reviewed transitivity, route precedence, same-document relevance, confidence preservation, and scope boundaries before the sign-off commit. S2 owns the required throwaway-commit negative control and full gates.

[rickylabs]

S2 — implementation gate evidence

Immutable implementation head: 33e0ef9d016d2cd1a874618372db03eb9163acf6

• Measured fresh direct/plugin gap: 0.3019801981861221 (base gap 0.08224092497339086).
• Direction: deterministic close-score tie handling; the golden remains unchanged.
• Guidance filter: exit 0 — 14 passed, 0 failed, 3325 filtered.
• Fresh corpus: all 8 cases passed for 2 independent index constructions × 2 reruns.
• Negative control: commit 5d7ca0f46, raw exit 1 (0 passed, 1 failed); restored by c86a4080f, then exit 0.
• Scoped check/lint/fmt, package doc lint, and quality:gate: exit 0.
• Repository tests: exit 1 — 3321 passed, 1 failed, 17 ignored. The sole failure is the documented dispatched-base Fresh defer canary permits split SDK cache-provider singleton #1589 JSDoc finding; no guidance failure.
• Boundary audit: no golden, lock, generated corpus/publish asset, corpus tooling, or docs/site change.

The PR remains draft with exactly status:impl. Mandatory IMPL-EVAL is intentionally left to the orchestrator's separate native Opus 5 read-only fallback; this implementation session did not dispatch or retrigger it.

[rickylabs]

Final immutable implementation head: 543a7b1e9be555f8de2db2cbe5fe3bf5139fd120. This supersedes the S2 comment's head only by commit 543a7b1e9, which removes two trailing blank lines from harness Markdown; implementation and gate evidence are unchanged.

[rickylabs]

[PHASE: IMPL] [VERDICT: REVIEW_CORRECTION_LANDED]

S3 restores pre-PR route-ranked confidence semantics at immutable head 7933e044c1fec3b3fd965639bbe43b6ee3a1f52d.

Correction

• topScore is again ranked[0]?.score, computed after route and close-score ordering.
• Added guidance confidence follows a route-promoted lower scorer: the unhinted global scorer is high; route activation promotes the lower scorer and returns medium.
• A close-score reorder can lower the score used for confidence by at most 0.5, so only scores within 0.5 of thresholds 8 or 24 can change category.

Explicit limits

• Cross-document slug order is deterministic but arbitrary; data-persistence wins the measured tie lexically, not via a semantic superiority claim.
• A new third document inside the same close-score group can change the fixture order if its slug sorts earlier. This fix protects a fixed candidate set from unrelated corpus-statistic drift; genuine candidate-set growth requires review.

Gates

• focused retrieval + evaluation: exit 0 — 8 passed
• MCP package tests: exit 0 — 136 passed
• scoped check/lint: exit 0 — 115 files, no findings
• exact touched-file format check: exit 0
• repeated negative control: raw exit 1; fixture restored; guidance evaluation then exit 0

Package-wide format is not claimed; #1618 owns the malformed-fixture formatter obstruction. The PR remains draft with exactly status:impl; no evaluator was dispatched or retriggered.

[rickylabs]

Final immutable S3 handoff head: 5a0e7303e93413a80999dbf59ba9f8191798e1ee. The final commit updates only the harness reconcile/context handoff after the implementation commit 7933e044c; behavior and gate evidence are unchanged.

[rickylabs]

Exact-head IMPL-EVAL — PASS

Provenance: native Claude Opus 5, read-only, fresh session, opposite-family to this Codex-authored
change. Not a cloud dispatch. Not Fable at any depth. No GitHub writes by the evaluator; worktree
restored and verified clean after every constructed failure.

Head evaluated: 5a0e7303e93413a80999dbf59ba9f8191798e1ee, base origin/main = 1ed78f508
(unmoved for the duration of the evaluation).

Does merging leave main worse? No — it leaves it green.

The decisive evidence is the merge result itself, built in an isolated clone and run in full:

git merge 5a0e7303e  → exit 0, no conflicts
deno task test       → ok | 3353 passed (624 steps) | 0 failed | 17 ignored (3m33s)
MERGED_ROOT_TEST_RAW_EXIT=0

On the branch alone the suite is exit 1 with a single failure — published JSDoc excludes internal workstream codenames. That is the #1589 finding, base-inherited: PR #1614 merged into main
after this branch's merge-base. The branch is not carrying a failure into main; it is missing a fix
main already has.

The two new tests genuinely pin their subjects

Counterfactual variants were built and each test run against each:

[shipped]   grouping: PASS   confidence: PASS (medium)
[globalmax] grouping: PASS   confidence: FAIL (actual = high)
[nogroup]   grouping: FAIL   confidence: PASS

Each test fails only under its own subject's breakage. The topScore restoration
(ranked[0]?.score ?? 0 computed after ordering) is therefore correct and pinned — the
global-max-before-ordering variant now fails with high instead of medium. Independently
reproduced by the orchestrator: applying that variant gives raw exit 1 on
guidance-retrieval_test.ts.

Sequencing — the 2×2 that decides merge order

BASE(shipped)            main(pre-fix)  -> MATCH
BASE(shipped)            PR1617(fixed)  -> MATCH
FRESH(PR1608 632fa0096)  main(pre-fix)  -> MISMATCH   <- the #1615 defect, reproduced
FRESH(PR1608 632fa0096)  PR1617(fixed)  -> MATCH

This PR passes with the stale corpus and with #1608's live fresh corpus, so merging it first
fixes rather than relocates.

Measurement re-derived independently

Instrumented scoring on the shipped corpus reproduced the PR's figures exactly — gap
0.08224092497339086 (base) and -0.3019801981861221 (fresh), sign reversal real.

Algorithm audit — no defect found

Empty/single-element inputs are no-ops. All-equal scores give deterministic slug order. The band
boundary is inclusive and exact. No chaining: 10.0, 9.6, 9.2 correctly leaves 9.2 outside the
leader's band. Both comparators are total orders (slug#section is unique), so nothing relies on
Array.prototype.sort stability — 200 random shuffles produced 1 distinct outcome. splice with
a spread over a 50,000-element band did not throw. On the real corpus: 33 matched entries, 13 bands,
max band size 7.

Confidence bound is exactly 0.5 and cannot compound — bands are leader-anchored and band-0's
leader is the pre-grouping ranked[0]. Threshold crossing is possible in principle, but measured
across all 8 fixture intents × 2 corpora, confidence is identical between main and the fix in
all 16 cases.

Negative control — the evaluator's own, different in kind

The author perturbed a fixture expectation. The evaluator perturbed the implementation — reversing
the intra-band slug comparator — and ran the real gate: raw exit 1, with the anchors swapping.
Restored; porcelain empty; file byte-identical to backup.

Scope

Exactly two source files plus six run artifacts. Fixtures changed: 0. Lock files: 0. No
docs/site, no generated assets. orderGuidanceSections is reachable only by deep path —
mod.ts does not reference guidance-index and the exports map is only ./mod.ts, ./cli,
./openapi-projection — so no new public API.

Is the tie-break principled or fixture-tuned?

The mechanism is principled; the outcome for this fixture is alphabetical; and the PR says so.
It clusters by document identity, preserves raw score within a document, applies uniformly, and is
input-order independent. It is not special-cased. The PR body states plainly that data-persistence
wins because its slug sorts before explanation, not because this PR proves that document deserves
higher semantic rank. That is the honest claim and it is the true one.

Non-blocking findings — recorded, follow-ups filed

• The band-width constant 0.5 is not pinned by any test. Setting it to 5 (10×) leaves all 8
  fixture cases and the new grouping test passing, because that test's out-of-band element sorts
  last by slug either way. A case whose slug sorts early but whose score is outside the band would
  pin it. Filed as a follow-up — this is the milestone's signature defect class (a guard that cannot
  fire), so it is worth closing even though it blocks nothing.
• One PR-body sentence overclaims. "unrelated corpus-statistic movement alone no longer flips the
  existing pair" omits the existing-member-exits-the-band mode, which needs no membership change —
  only score drift past 0.5. Quantified: band span moved +0.0748587451731435 from a single
  regeneration, leaving headroom 0.1980198018138779 ≈ 2.6× the observed per-regeneration
  movement. The fix converts instability at gap≈0 into instability at gap≈0.5 — a large
  improvement, not an elimination. Body corrected.
• deno doc --lint mod.ts is exit 1 at this head (private-type-ref on SchemaViewName), confirmed
  pre-existing on origin/main without this PR. The body's "exit 0" claim was stale; corrected.
• tooling: deno fmt cannot verify packages/mcp — a deliberately malformed test fixture aborts config parsing #1618 is broader than filed: the scoped lint wrapper aborts identically to fmt, from the
  same malformed doctor fixture. tooling: deno fmt cannot verify packages/mcp — a deliberately malformed test fixture aborts config parsing #1618 updated.

[rickylabs]

Pre-merge gate — merging with the suite's cloud-eval check bypassed, and why

agentic:gh-pr merge gates on a cloud evaluator comment. This PR does not have one by design, so the merge uses --no-eval-gate. Recording the substitution rather than performing it silently.

What stands in its place: a native Claude Opus 5, read-only, separate-session, opposite-family IMPL-EVAL — PASS at immutable head 5a0e7303e, no blocking findings (verdict). Cloud evaluators stalled earlier in this milestone and Fable is prohibited by owner policy, so the native fallback is the documented route.

impl-eval:skip was applied before the ready flip, and verified to have suppressed dispatch rather than assumed:

run 31634858148 -> dispatch: success
  Record attributed IMPL-EVAL skip           :: success
  Resolve and dispatch exactly one evaluator :: skipped

No paid evaluator ran on this branch.

The verdict is still bound to this code. It was issued at 5a0e7303e; the current head 32d3e1842 is that commit plus a merge of origin/main. The feature diff is byte-identical across the merge — git diff origin/main...HEAD -- packages/ is 6649 bytes before and after — so no evaluated line changed.

Merge-time verification at 32d3e1842, not at the evaluated head:

check	result
merge of origin/main	exit 0, no conflicts
feature diff unchanged	byte-identical (6649 B)
root deno task test (local, merged tree)	exit 0 — 3353 passed, 0 failed
check-test	pass
quality	pass
surface-diff	pass
close-gate	pass (mirrored after the label correction)
scaffold-runtime + sqlite variant	pass
mergeState	CLEAN

The evaluator independently reproduced the merged-tree result in an isolated clone (3353 passed, 0 failed), so two separate constructions agree.

Follow-ups filed, neither blocking: #1622 (closeScoreGap is pinned by no test — a 10x change passes green) and #1618 (broadened: packages/mcp has neither a working scoped fmt gate nor a working scoped lint gate).