Skip to content

Releases: franciszver/agentforge-2-evidence-agent

v2.3.0 — deferred-item close-out: ingestion failure surfacing, reproducible container

Choose a tag to compare

@franciszver franciszver released this 27 Jul 20:53
5f21371

Closes out the two items deferred from the v2.2.0 release, plus a version-alignment ruling. 3 PRs, 15 commits since v2.2.0.

Behaviour change — read this if you call ingestion

  • Issue 206 — a fully-failed document ingestion previously returned normally with zero facts and no exception, byte-identical on disk to a genuinely empty document. That silence is what let issue 204's text-only vision wiring survive undetected. attach_and_extract now raises IngestionError when every page fails; no facts sidecar is written, so the document reads as not-ingested and is safely re-runnable. Any caller relying on the old silent-empty return will now see an exception. Partial failure is unchanged — surviving pages' facts are kept, and the facts sidecar now records pages_total/failed_pages so partial degradation is durable on disk instead of invisible. The pre-wired extraction_failure_rate dashboard alert (threshold, explanation text, and banner all existed but were permanently fed None) is now populated. Measured live in-container before and after, against both a total-failure case and an 8-fact success control.

Reproducible container builds

  • Issue 213 — the container build was not reproducible: bounds narrow the resolution window but do not pin it, and transitives (notably starlette, the layer actually under FastAPI) were unbounded. The container — the actual shipped artifact — now installs from a hash-locked requirements.txt generated by pip-compile inside the same digest-pinned base image the Dockerfile uses, with --require-hashes enforced at install. CI gains a job that builds the image (previously no workflow built it at all) and audits the lock. Verified empirically at review: double --no-cache builds byte-identical, lock regeneration byte-identical to the committed file, and a corrupted hash genuinely fails the build.

Version alignment

  • Issue 214 — the package previously declared version = "0.1.0" while releases were tagged v2.x. By owner ruling, the package version tracks the release tag from this release onward; pyproject, the app, and the OpenAPI document all read 2.3.0, and this tag closes the loop. A red-first test forces every future release PR to bump deliberately.

Known limits, stated plainly

  • The hash lock covers the container only. Dev venvs and CI's pip install -e ".[dev]" remain range-based by design — the lock is the machine-facing contract for the shipped artifact; the pyproject ranges stay the human-facing one.
  • The issue-204 vision-capability pre-flight is a name heuristic; a vision-named model that is not actually pulled passes it. Ruled acceptable because the issue-206 change makes that case raise loudly at first ingestion instead of failing silently.
  • --allow-unsafe produced no setuptools/pip pins because nothing in the runtime graph declares them; the build-backend closure is locked via the digest-pinned base image plus --no-build-isolation instead. A base-digest bump mandates lock regeneration.

Full test state at tag, verified independently on main immediately before tagging (not inherited from the last PR's CI): 1393 passed, 1 skipped, 18 deselected, 12 xfailed (agent), 383 passed, 12 deselected, 15 xfailed (evals), mypy clean (64 source files), CI green on 5f213718.

v2.2.0 — Phase 3 red-team security batch

Choose a tag to compare

@franciszver franciszver released this 27 Jul 18:34
7b3c9cf

Closes out the Phase 3 red-team security batch and the follow-up work it generated. 21 PRs, 115 commits since v2.1.0.

The short version: four red-team findings were filed against v2.1.0. Working them surfaced sixteen more defects — every one found by the review gates, not in production — and produced two measured decisions not to ship a fix.

Security fixes

  • #168 — the default bearer-token validator accepted any non-empty token. Now fail-closed, with permissive behaviour only behind an explicit dev flag. Took four review rounds: the first fix returned 500 instead of 401 (a fingerprinting oracle plus pre-auth outbound amplification), and the second swallowed the error so the agent answered with zero patient evidence rather than failing loudly.
  • #176GET /review was unauthenticated and rendered clinician free text verbatim; a probe returned MRN and patient name with no auth. Found by the #168 review, not by the finding that prompted it. Fixed by redaction rather than gating, because /review has no proxy and gating would 401 the only way to view the page.
  • #180 / #185 — feedback had no ownership check on target_correlation_id, so a caller with any valid token could attach forged text to another clinician's trace. Now bound to OpenEMR's signature-verified sub claim under per-user tokens, with an explicit regime column so pre-existing rows can never become accidentally claimable.
  • #177 — unauthenticated /chat triggered an outbound OAuth fetch before validation. Now structurally unreachable pre-auth; the timing oracle measured closed (2.14s → 0.0021s).
  • #167 / #173 — unbounded /chat message length, no conversation eviction, and no pre-parse body-size limit. All bounded, with mem_limit and a restart policy on the agent container.
  • #174 / #182 — every conversation retained a full copy of every other patient's name (the dominant memory term, ~3GB worst case). Replaced with a TTL'd cache keyed by authenticated principal, bounded by retained rows rather than entry count.
  • #204 — document ingestion's vision path resolved to a text-only model by default; the VLM was a per-call override documented only in a demo script. Now a dedicated setting with a fail-closed capability check.

Measured decisions not to ship

  • #169 — a status-contradiction gate was built and declined. Four adversarial passes produced four new defect classes with the rate not decaying, and the settling finding was that identifier anchoring false-blocks a correct claim about an active formulation of a discontinued base drug — the most common real medication-list transition. The strict-xfail tripwire on main is the standing record that the gap is known and undefended.
  • #192 — nonce-fencing was built for both LLM judges, measured, and declined: neutral on the judge that ships enabled, 2.4× worse on the other. What ships instead is the measurement — a 152-payload injection battery run live against the shipped model. Across 190 draws per configuration, no payload could push a judge toward SUPPORTED; the exploitable direction is fail-closed (it strips correct claims rather than certifying false ones). Recorded with its limits: the zero is confounded with scenario distance, and 5 draws per payload gives a ~45% per-payload detection floor.

Infrastructure

  • #186 — the Jest suite had been red on main and was not run by CI at all. Green, and now a required status check.
  • #184 / #196 — the OpenAPI contract test failed on hosts for an environmental reason, making real drift locally indistinguishable from noise. Normalised, and dependencies now carry both floors and ecosystem-appropriate ceilings so a stale venv cannot silently resolve the version that caused it.
  • #199 / #201 / #203 — issue references inherited from Phase 1's tracker that resolved to different issues here, and docstrings asserting premises the code had already disproved.

Known limits, stated plainly

  • semantic_support ships enabled with a soft-instruction injection defence. That is now measured rather than assumed, and the battery is committed so a judge-model change can be re-measured rather than re-argued.
  • Dependency bounds narrow the resolution window; they do not pin it, and transitive dependencies remain unbounded.
  • #206 is open: a fully-failed ingestion returns normally with zero facts and no exception, so it is indistinguishable from an empty document. Deferred to Phase 3 with its own measurement.

Full test state at tag: 1369 passed, 1 skipped, 12 xfailed (agent), 383 passed, 15 xfailed (evals), mypy clean.

v2.1.0 — Trust-layer scoping + honest-measurement finish-out

Choose a tag to compare

@franciszver franciszver released this 25 Jul 20:11
923fb7d

What's in this release (since v2.0.0)

Trust layer

  • Per-tool-call citation scoping (#158, PR #162): the claim extractor's citable inputs can be scoped to the tool calls the answer actually engaged with — prevention (narrowed catalog) plus fail-closed enforcement (TOOL_CALL_NOT_ENGAGED). Ships default-OFF (see below). Its collateral fix is live unconditionally: the system's machine-generated recency notice can no longer leak into answer-derived trust checks (PlannerResult.answer_pre_notice).
  • Claim-in-answer grounding gate scaffolding (#153, PR #157), default-OFF, documented unfit to enable.

Measurement infrastructure (permanent)

  • Paired same-draw ON/OFF strip-rate harness (#163, PR #165): one live planner draw per eval case, verification run both ways — isolates a verification-layer change's effect from draw variance. Exposure counters, eligible-claim denominators, prevention-loss visibility, mutation-tested metrics, provenance-stamped artifacts.
  • Live-POST /chat N-draw stability harness (#160, PR #166): real planner tool calls against the real chart via the live HTTP path, with per-draw three-way mechanism attribution (tool-selection vs temp=0 generation nondeterminism vs downstream) and per-session sidecar provenance.

Decisions made on evidence (the record this project keeps)

  • Scoping default stays OFF (#164, declined): the corrected 65-case measurement showed 0 fabrications caught and 100% of the exposure surface (2/2 cases) blocking legitimate absence/ambiguity answers — lexical engagement is structurally blind to answers about absence, the same failure class that made #153 unfit, one level up. Fourth measured "don't ship it" (after the 14B model comparison, the relevance gate, and prompt hardening).
  • #149/#150 (live nondeterminism) closed as superseded: 48/48 live draws on the current build are bit-identical, in an outcome regime neither original report observed — the pipeline changed underneath the reports. Caveats (warm stack, API-path-only) are machine-readable in the committed artifact.

Process note: the first #163 measurement read "maximally clean" and was exposed as a statistical false zero by the adversarial review gate; the corrected run reversed the ship decision. Both artifacts, both conclusions, and the reversal are preserved in the issue trail — that's the point.

Full trails: issues #149, #150, #153, #158, #160, #161, #163, #164; PRs #157, #162, #165, #166.