Skip to content

v0.4.0 — Trust & audit

Choose a tag to compare

View all tags
@SelfishCoconut SelfishCoconut released this 15 Jul 09:24
· 123 commits to main since this release
v0.4.0
7b31241
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.4.0 — Trust & audit

Milestone M4 (Trust & audit). The retest engine now defends its own results:
every probe runs through an independent sanity check, every verdict is provably
re-derivable from its stored evidence alone, and a complete run exports as one
versioned, schema-validated JSON document — the input the evaluation harness (M5)
will consume.

Highlights

  • FR-08 — Execution sanity checker (#13): an independent verifier
    (guarded_run) wrapped around every approved probe. It re-checks the probe is a
    member of the approved plan before any socket opens — a deviation is
    fail-closed (PlanDeviationError → HTTP 409, no verdict stored) — and
    downgrades an over-confident verdict after (a fixed on a 404/410 or 3xx
    becomes inconclusive, since a moved endpoint is not a proven fix). It only ever
    removes confidence, never manufactures it (ADR-0014).
  • FR-10 — Audit trail + verdict re-derivation (#15): a verdict is a pure
    function of its stored evidence, so rederive_run recomputes every verdict from
    the same deterministic assessment the live path uses and diffs it against
    storage — reproduced from the trail alone, no re-execution (the NFR-02 test).
    VerdictRecord gained created_at/actor; exposed at GET /api/audit
    (ADR-0015).
  • FR-12 — Versioned run export (#17): GET /api/export returns a complete run
    (reports → findings → plans → verdicts + evidence + descriptive metrics) as one
    SCHEMA_VERSION-versioned JSON document. Its JSON Schema is generated from the
    model (GET /api/export/schema, published to
    docs/reference/schemas/run-export.schema.json) and drift-tested, so the
    published schema can never diverge from the document (ADR-0016).

Quality

  • Backend: 180 unit tests @ 97% coverage + 22 integration; the three new M4
    modules (sanity.py, audit.py, export.py) each at 100%.
  • Pre-release codebase-sanity audit: GO — all mechanical gates clean (xenon
    --max-absolute C, radon avg A, vulture, pylint 9.98, mypy --strict, ruff), no
    dead code / complexity regression / import-layer drift.
  • Architecture docs updated for M4 (#62): C4 container + new sequence diagrams
    for the FR-08 guarded execution and the FR-10/FR-12 read-only derivations.
  • ADRs accepted this cycle: 0014, 0015, 0016.

Next

M5 — Evaluation (→ v1.0.0): pin the ground-truth vulnerable lab + an expected
verdict per finding, then build the FR-15 evaluation harness that consumes this
release's JSON export into a reproducible metrics table (NFR-01: ≥70% correct
verdicts, zero confidently-wrong on ambiguity).

Assets 2
Loading