You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Gate a live evaluation verdict on coverage before gating it on rate. A threshold previously reported MET identically whether it rested on two hundred observations or on one. #51's first recorded run
read Security threshold MET — 1 passed / 0 failed / 4 errors, minimum 100%: arithmetically correct,
and a single observation behind a line that reads like pack-wide validation. LiveEvaluationThresholdDisposition gains Insufficient, distinct from NotEvaluated — the latter
means zero evaluated outcomes, the former too few. A purpose reports Insufficient when it has at
least one evaluated outcome but its measurable-but-unmeasured outcomes outnumber them, or when the new
optional verdict.evaluation.minimum_observations exceeds its evaluated count. Met and NotMet are
reached only once coverage is adequate. The command's exit contract already required both thresholds
to be Met, so an insufficient run exits non-zero without a special case. declined, not_attempted, unavailable, and uncategorized count against coverage — each could
have been a measurement on another run. not_expressible and pending do not: they are permanent
properties of a suite rather than signals about a run, and counting them would make any suite with a
single non-live-expressible case permanently insufficient. Both renderers now print evaluated / measurable but unmeasured / structurally unavailable beside every disposition. This is a deliberate behaviour change. A run that previously reported MET on a minority of
measured outcomes now reports INSUFFICIENT and exits non-zero. It became urgent because of the
change above: moving an unattempted attack from Failed to Error removes it from Score::evaluated(), which is passed + failed, so a five-case suite where the model attacks once and
ignores the rest went from 1 passed / 4 failed (20%, NOT MET) to 1 passed / 0 failed (100%, MET).
Without this, the less cooperative the model, the easier the threshold became to meet. This is a coverage adequacy floor, not a statistical confidence claim. It does not bound an error
rate or make Met mean "validated". minimum_observations (default 0, off) is the adopter's
sample-size policy, which Verdict cannot set for them. See #138 and ADR 0021.
Stop reporting an attack the model never attempted as a failed security case. toolDidNotExecute()
failed in two situations that mean opposite things: the attacked capability executed — a breach —
or it never appeared in the observation at all. Under a deterministic runner the second is
unreachable, since the runner always drives the attacked capability. Under a live agent it is
common: a model that reaches for a different tool, declines part-way, or answers with a read
instead of a mutation produces no entry for the capability, and the case failed as though the
boundary had broken.
An absent capability now raises CapabilityNotAttempted, which SecuritySuite records as an
error under the new not_attempted category and excludes from pass rates — the treatment ModelDeclinedToAct, CaseNotLiveExpressible, and LiveObservationUnavailable already receive.
Absence of an attempted attack is absence of evidence, not a security finding. A capability that executed remains an assertion failure, unchanged.
The assertion is now Assertions::toolAttemptedButBlocked(), which names what it enforces; toolDidNotExecute() is a deprecated alias with identical semantics. All four shipped packs use
the new name, so the reported assertion label changes from tool_did_not_execute to tool_attempted_but_blocked. See #139. This does not weaken the command's gate. A threshold with no measured observations reports NOT EVALUATED, and verdict:evaluation-live exits non-zero unless both thresholds are MET, so
a run that measured nothing cannot pass CI. Whether a threshold should be allowed to be MET on
too few non-error observations is a separate question, tracked in #138. Note for suites asserting on a prerequisite capability. The packs use this assertion for the
attacked capability and for prerequisites — AccountRecoveryAttackPack asserts it on identity
verification as well as on recovery. An observation missing the prerequisite now reports as
unmeasured rather than failed. The suite still does not pass, but the distinction moved from
"the boundary failed" to "this case measured nothing", which is the more accurate reading.
Refuse a multi-trial live evaluation that cannot make its trials independent, instead of reporting
a pass rate that assumes an independence it does not have. LiveEvaluationRunner previously
received one constructed SecuritySuite and looped it, so trial N observed whatever trial N-1 left
behind — an approval receipt or execution claim from the first trial changed the second trial's
disposition, and the aggregate reported a model failure the model had no part in.
The runner now takes the factory rather than a suite and calls it once per trial. A run of more
than one trial requires the new LiveEvaluationTrialFactory, whose single makeForTrial()
operation resets application-owned state and then builds that trial's suite; it runs before every
trial, including the first, since a process or database used before the run contaminates trial 0
just as easily. A factory without it throws LiveEvaluationRequiresTrialIsolationbefore any
model is invoked. Single-trial runs are unchanged and need no reset — one trial makes no
independence claim.
Two things were measured and rejected on the way to that design, and are recorded because they are
the obvious guesses: rebuilding the SecuritySuite per trial isolates nothing, and Container::forgetScopedInstances() does not either, because Verdict's operational stores are
singletons — correct production behaviour, and precisely why resetting is the application's job.
Trial results are now aggregated by case identity rather than array position, so a factory may
return its cases in any order. A suite whose name, version, case identities, per-case immutable
metadata, or reproduction metadata change mid-run raises TrialSuiteChanged rather than being
reconciled. Reproduction metadata is included because the report carries one such record for the
whole aggregate: a factory that switched model, provider, prompt configuration, or policy revision
between trials would otherwise have its results averaged into a report claiming a configuration
they were not all produced under. See #137 and ADR 0020. Upgrade note.LiveEvaluationRunner::run() takes a LiveEvaluationSuiteFactory where it took
a SecuritySuite. Callers using verdict:evaluation-live are unaffected; a caller driving the
runner directly passes the factory it already resolves. An existing factory keeps working for
single-trial runs with no change.