Releases: franciszver/agentforge-2-evidence-agent
Release list
v2.3.0 — deferred-item close-out: ingestion failure surfacing, reproducible container
Closes out the two items deferred from the v2.2.0 release, plus a version-alignment ruling. 3 PRs, 15 commits since v2.2.0.
Behaviour change — read this if you call ingestion
- Issue 206 — a fully-failed document ingestion previously returned normally with zero facts and no exception, byte-identical on disk to a genuinely empty document. That silence is what let issue 204's text-only vision wiring survive undetected.
attach_and_extractnow raisesIngestionErrorwhen every page fails; no facts sidecar is written, so the document reads as not-ingested and is safely re-runnable. Any caller relying on the old silent-empty return will now see an exception. Partial failure is unchanged — surviving pages' facts are kept, and the facts sidecar now recordspages_total/failed_pagesso partial degradation is durable on disk instead of invisible. The pre-wiredextraction_failure_ratedashboard alert (threshold, explanation text, and banner all existed but were permanently fedNone) is now populated. Measured live in-container before and after, against both a total-failure case and an 8-fact success control.
Reproducible container builds
- Issue 213 — the container build was not reproducible: bounds narrow the resolution window but do not pin it, and transitives (notably
starlette, the layer actually under FastAPI) were unbounded. The container — the actual shipped artifact — now installs from a hash-lockedrequirements.txtgenerated by pip-compile inside the same digest-pinned base image the Dockerfile uses, with--require-hashesenforced at install. CI gains a job that builds the image (previously no workflow built it at all) and audits the lock. Verified empirically at review: double--no-cachebuilds byte-identical, lock regeneration byte-identical to the committed file, and a corrupted hash genuinely fails the build.
Version alignment
- Issue 214 — the package previously declared
version = "0.1.0"while releases were tagged v2.x. By owner ruling, the package version tracks the release tag from this release onward; pyproject, the app, and the OpenAPI document all read2.3.0, and this tag closes the loop. A red-first test forces every future release PR to bump deliberately.
Known limits, stated plainly
- The hash lock covers the container only. Dev venvs and CI's
pip install -e ".[dev]"remain range-based by design — the lock is the machine-facing contract for the shipped artifact; the pyproject ranges stay the human-facing one. - The issue-204 vision-capability pre-flight is a name heuristic; a vision-named model that is not actually pulled passes it. Ruled acceptable because the issue-206 change makes that case raise loudly at first ingestion instead of failing silently.
--allow-unsafeproduced no setuptools/pip pins because nothing in the runtime graph declares them; the build-backend closure is locked via the digest-pinned base image plus--no-build-isolationinstead. A base-digest bump mandates lock regeneration.
Full test state at tag, verified independently on main immediately before tagging (not inherited from the last PR's CI): 1393 passed, 1 skipped, 18 deselected, 12 xfailed (agent), 383 passed, 12 deselected, 15 xfailed (evals), mypy clean (64 source files), CI green on 5f213718.
v2.2.0 — Phase 3 red-team security batch
Closes out the Phase 3 red-team security batch and the follow-up work it generated. 21 PRs, 115 commits since v2.1.0.
The short version: four red-team findings were filed against v2.1.0. Working them surfaced sixteen more defects — every one found by the review gates, not in production — and produced two measured decisions not to ship a fix.
Security fixes
- #168 — the default bearer-token validator accepted any non-empty token. Now fail-closed, with permissive behaviour only behind an explicit dev flag. Took four review rounds: the first fix returned 500 instead of 401 (a fingerprinting oracle plus pre-auth outbound amplification), and the second swallowed the error so the agent answered with zero patient evidence rather than failing loudly.
- #176 —
GET /reviewwas unauthenticated and rendered clinician free text verbatim; a probe returned MRN and patient name with no auth. Found by the #168 review, not by the finding that prompted it. Fixed by redaction rather than gating, because/reviewhas no proxy and gating would 401 the only way to view the page. - #180 / #185 — feedback had no ownership check on
target_correlation_id, so a caller with any valid token could attach forged text to another clinician's trace. Now bound to OpenEMR's signature-verifiedsubclaim under per-user tokens, with an explicit regime column so pre-existing rows can never become accidentally claimable. - #177 — unauthenticated
/chattriggered an outbound OAuth fetch before validation. Now structurally unreachable pre-auth; the timing oracle measured closed (2.14s → 0.0021s). - #167 / #173 — unbounded
/chatmessage length, no conversation eviction, and no pre-parse body-size limit. All bounded, withmem_limitand a restart policy on the agent container. - #174 / #182 — every conversation retained a full copy of every other patient's name (the dominant memory term, ~3GB worst case). Replaced with a TTL'd cache keyed by authenticated principal, bounded by retained rows rather than entry count.
- #204 — document ingestion's vision path resolved to a text-only model by default; the VLM was a per-call override documented only in a demo script. Now a dedicated setting with a fail-closed capability check.
Measured decisions not to ship
- #169 — a status-contradiction gate was built and declined. Four adversarial passes produced four new defect classes with the rate not decaying, and the settling finding was that identifier anchoring false-blocks a correct claim about an active formulation of a discontinued base drug — the most common real medication-list transition. The strict-xfail tripwire on
mainis the standing record that the gap is known and undefended. - #192 — nonce-fencing was built for both LLM judges, measured, and declined: neutral on the judge that ships enabled, 2.4× worse on the other. What ships instead is the measurement — a 152-payload injection battery run live against the shipped model. Across 190 draws per configuration, no payload could push a judge toward
SUPPORTED; the exploitable direction is fail-closed (it strips correct claims rather than certifying false ones). Recorded with its limits: the zero is confounded with scenario distance, and 5 draws per payload gives a ~45% per-payload detection floor.
Infrastructure
- #186 — the Jest suite had been red on
mainand was not run by CI at all. Green, and now a required status check. - #184 / #196 — the OpenAPI contract test failed on hosts for an environmental reason, making real drift locally indistinguishable from noise. Normalised, and dependencies now carry both floors and ecosystem-appropriate ceilings so a stale venv cannot silently resolve the version that caused it.
- #199 / #201 / #203 — issue references inherited from Phase 1's tracker that resolved to different issues here, and docstrings asserting premises the code had already disproved.
Known limits, stated plainly
semantic_supportships enabled with a soft-instruction injection defence. That is now measured rather than assumed, and the battery is committed so a judge-model change can be re-measured rather than re-argued.- Dependency bounds narrow the resolution window; they do not pin it, and transitive dependencies remain unbounded.
- #206 is open: a fully-failed ingestion returns normally with zero facts and no exception, so it is indistinguishable from an empty document. Deferred to Phase 3 with its own measurement.
Full test state at tag: 1369 passed, 1 skipped, 12 xfailed (agent), 383 passed, 15 xfailed (evals), mypy clean.
v2.1.0 — Trust-layer scoping + honest-measurement finish-out
What's in this release (since v2.0.0)
Trust layer
- Per-tool-call citation scoping (#158, PR #162): the claim extractor's citable inputs can be scoped to the tool calls the answer actually engaged with — prevention (narrowed catalog) plus fail-closed enforcement (
TOOL_CALL_NOT_ENGAGED). Ships default-OFF (see below). Its collateral fix is live unconditionally: the system's machine-generated recency notice can no longer leak into answer-derived trust checks (PlannerResult.answer_pre_notice). - Claim-in-answer grounding gate scaffolding (#153, PR #157), default-OFF, documented unfit to enable.
Measurement infrastructure (permanent)
- Paired same-draw ON/OFF strip-rate harness (#163, PR #165): one live planner draw per eval case, verification run both ways — isolates a verification-layer change's effect from draw variance. Exposure counters, eligible-claim denominators, prevention-loss visibility, mutation-tested metrics, provenance-stamped artifacts.
- Live-POST /chat N-draw stability harness (#160, PR #166): real planner tool calls against the real chart via the live HTTP path, with per-draw three-way mechanism attribution (tool-selection vs temp=0 generation nondeterminism vs downstream) and per-session sidecar provenance.
Decisions made on evidence (the record this project keeps)
- Scoping default stays OFF (#164, declined): the corrected 65-case measurement showed 0 fabrications caught and 100% of the exposure surface (2/2 cases) blocking legitimate absence/ambiguity answers — lexical engagement is structurally blind to answers about absence, the same failure class that made #153 unfit, one level up. Fourth measured "don't ship it" (after the 14B model comparison, the relevance gate, and prompt hardening).
- #149/#150 (live nondeterminism) closed as superseded: 48/48 live draws on the current build are bit-identical, in an outcome regime neither original report observed — the pipeline changed underneath the reports. Caveats (warm stack, API-path-only) are machine-readable in the committed artifact.
Process note: the first #163 measurement read "maximally clean" and was exposed as a statistical false zero by the adversarial review gate; the corrected run reversed the ship decision. Both artifacts, both conclusions, and the reversal are preserved in the issue trail — that's the point.
Full trails: issues #149, #150, #153, #158, #160, #161, #163, #164; PRs #157, #162, #165, #166.