Skip to content

v0.6.0

Choose a tag to compare

@github-actions github-actions released this 14 Aug 15:23
· 260 commits to main since this release
  • Gate a live evaluation verdict on coverage before gating it on rate. A threshold previously reported
    MET identically whether it rested on two hundred observations or on one. #51's first recorded run
    read Security threshold MET — 1 passed / 0 failed / 4 errors, minimum 100%: arithmetically correct,
    and a single observation behind a line that reads like pack-wide validation.
    LiveEvaluationThresholdDisposition gains Insufficient, distinct from NotEvaluated — the latter
    means zero evaluated outcomes, the former too few. A purpose reports Insufficient when it has at
    least one evaluated outcome but its measurable-but-unmeasured outcomes outnumber them, or when the new
    optional verdict.evaluation.minimum_observations exceeds its evaluated count. Met and NotMet are
    reached only once coverage is adequate. The command's exit contract already required both thresholds
    to be Met, so an insufficient run exits non-zero without a special case.
    declined, not_attempted, unavailable, and uncategorized count against coverage — each could
    have been a measurement on another run. not_expressible and pending do not: they are permanent
    properties of a suite rather than signals about a run, and counting them would make any suite with a
    single non-live-expressible case permanently insufficient. Both renderers now print
    evaluated / measurable but unmeasured / structurally unavailable beside every disposition.
    This is a deliberate behaviour change. A run that previously reported MET on a minority of
    measured outcomes now reports INSUFFICIENT and exits non-zero. It became urgent because of the
    change above: moving an unattempted attack from Failed to Error removes it from
    Score::evaluated(), which is passed + failed, so a five-case suite where the model attacks once and
    ignores the rest went from 1 passed / 4 failed (20%, NOT MET) to 1 passed / 0 failed (100%, MET).
    Without this, the less cooperative the model, the easier the threshold became to meet.
    This is a coverage adequacy floor, not a statistical confidence claim. It does not bound an error
    rate or make Met mean "validated". minimum_observations (default 0, off) is the adopter's
    sample-size policy, which Verdict cannot set for them. See
    #138 and
    ADR 0021.
  • Stop reporting an attack the model never attempted as a failed security case. toolDidNotExecute()
    failed in two situations that mean opposite things: the attacked capability executed — a breach —
    or it never appeared in the observation at all. Under a deterministic runner the second is
    unreachable, since the runner always drives the attacked capability. Under a live agent it is
    common: a model that reaches for a different tool, declines part-way, or answers with a read
    instead of a mutation produces no entry for the capability, and the case failed as though the
    boundary had broken.
    An absent capability now raises CapabilityNotAttempted, which SecuritySuite records as an
    error under the new not_attempted category and excludes from pass rates — the treatment
    ModelDeclinedToAct, CaseNotLiveExpressible, and LiveObservationUnavailable already receive.
    Absence of an attempted attack is absence of evidence, not a security finding. A capability that
    executed remains an assertion failure, unchanged.
    The assertion is now Assertions::toolAttemptedButBlocked(), which names what it enforces;
    toolDidNotExecute() is a deprecated alias with identical semantics. All four shipped packs use
    the new name, so the reported assertion label changes from tool_did_not_execute to
    tool_attempted_but_blocked. See #139.
    This does not weaken the command's gate. A threshold with no measured observations reports
    NOT EVALUATED, and verdict:evaluation-live exits non-zero unless both thresholds are MET, so
    a run that measured nothing cannot pass CI. Whether a threshold should be allowed to be MET on
    too few non-error observations is a separate question, tracked in
    #138.
    Note for suites asserting on a prerequisite capability. The packs use this assertion for the
    attacked capability and for prerequisites — AccountRecoveryAttackPack asserts it on identity
    verification as well as on recovery. An observation missing the prerequisite now reports as
    unmeasured rather than failed. The suite still does not pass, but the distinction moved from
    "the boundary failed" to "this case measured nothing", which is the more accurate reading.
  • Refuse a multi-trial live evaluation that cannot make its trials independent, instead of reporting
    a pass rate that assumes an independence it does not have. LiveEvaluationRunner previously
    received one constructed SecuritySuite and looped it, so trial N observed whatever trial N-1 left
    behind — an approval receipt or execution claim from the first trial changed the second trial's
    disposition, and the aggregate reported a model failure the model had no part in.
    The runner now takes the factory rather than a suite and calls it once per trial. A run of more
    than one trial requires the new LiveEvaluationTrialFactory, whose single makeForTrial()
    operation resets application-owned state and then builds that trial's suite; it runs before every
    trial, including the first, since a process or database used before the run contaminates trial 0
    just as easily. A factory without it throws LiveEvaluationRequiresTrialIsolation before any
    model is invoked
    . Single-trial runs are unchanged and need no reset — one trial makes no
    independence claim.
    Two things were measured and rejected on the way to that design, and are recorded because they are
    the obvious guesses: rebuilding the SecuritySuite per trial isolates nothing, and
    Container::forgetScopedInstances() does not either, because Verdict's operational stores are
    singletons — correct production behaviour, and precisely why resetting is the application's job.
    Trial results are now aggregated by case identity rather than array position, so a factory may
    return its cases in any order. A suite whose name, version, case identities, per-case immutable
    metadata, or reproduction metadata change mid-run raises TrialSuiteChanged rather than being
    reconciled. Reproduction metadata is included because the report carries one such record for the
    whole aggregate: a factory that switched model, provider, prompt configuration, or policy revision
    between trials would otherwise have its results averaged into a report claiming a configuration
    they were not all produced under. See
    #137 and
    ADR 0020.
    Upgrade note. LiveEvaluationRunner::run() takes a LiveEvaluationSuiteFactory where it took
    a SecuritySuite. Callers using verdict:evaluation-live are unaffected; a caller driving the
    runner directly passes the factory it already resolves. An existing factory keeps working for
    single-trial runs with no change.