Skip to content

v2.1.0 — Trust-layer scoping + honest-measurement finish-out

Choose a tag to compare

View all tags
@franciszver franciszver released this 25 Jul 20:11
· 132 commits to main since this release
v2.1.0
923fb7d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's in this release (since v2.0.0)

Trust layer

  • Per-tool-call citation scoping (#158, PR #162): the claim extractor's citable inputs can be scoped to the tool calls the answer actually engaged with — prevention (narrowed catalog) plus fail-closed enforcement (TOOL_CALL_NOT_ENGAGED). Ships default-OFF (see below). Its collateral fix is live unconditionally: the system's machine-generated recency notice can no longer leak into answer-derived trust checks (PlannerResult.answer_pre_notice).
  • Claim-in-answer grounding gate scaffolding (#153, PR #157), default-OFF, documented unfit to enable.

Measurement infrastructure (permanent)

  • Paired same-draw ON/OFF strip-rate harness (#163, PR #165): one live planner draw per eval case, verification run both ways — isolates a verification-layer change's effect from draw variance. Exposure counters, eligible-claim denominators, prevention-loss visibility, mutation-tested metrics, provenance-stamped artifacts.
  • Live-POST /chat N-draw stability harness (#160, PR #166): real planner tool calls against the real chart via the live HTTP path, with per-draw three-way mechanism attribution (tool-selection vs temp=0 generation nondeterminism vs downstream) and per-session sidecar provenance.

Decisions made on evidence (the record this project keeps)

  • Scoping default stays OFF (#164, declined): the corrected 65-case measurement showed 0 fabrications caught and 100% of the exposure surface (2/2 cases) blocking legitimate absence/ambiguity answers — lexical engagement is structurally blind to answers about absence, the same failure class that made #153 unfit, one level up. Fourth measured "don't ship it" (after the 14B model comparison, the relevance gate, and prompt hardening).
  • #149/#150 (live nondeterminism) closed as superseded: 48/48 live draws on the current build are bit-identical, in an outcome regime neither original report observed — the pipeline changed underneath the reports. Caveats (warm stack, API-path-only) are machine-readable in the committed artifact.

Process note: the first #163 measurement read "maximally clean" and was exposed as a statistical false zero by the adversarial review gate; the corrected run reversed the ship decision. Both artifacts, both conclusions, and the reversal are preserved in the issue trail — that's the point.

Full trails: issues #149, #150, #153, #158, #160, #161, #163, #164; PRs #157, #162, #165, #166.

Assets 2
Loading