v1.0.0 — agentic retest console, evaluation, and a deployable stack

revalid 1.0 — an AI-driven system for the revalidation of pentest findings.

Feed it a pentest report; it extracts the findings, drafts a retest goal for each, and works one
finding at a time inside an egress-locked sandbox — proposing a command, waiting for you to approve
it, reading the real output, and deciding what to try next. It never declares a finding fixed
without evidence, and it hands back rather than guessing.

This release closes M5 (Evaluation) and M6 (Agentic interactive retest), and makes the tool
deployable.

---

Deploy it

make deploy      # → http://127.0.0.1:8000

Docker is the only prerequisite — no Python or Node toolchain. The stack is the app (backend + built
SPA) plus the pinned Juice Shop lab, both loopback-bound. Your LLM stays on your host, so no model
weights are pulled in. See ADR-0044, which also states
plainly what mounting the Docker socket grants and why it is accepted only under the single-operator
threat model.

The retest model changed completely

v0.4.0 retested by generating a batch of typed HTTP probes, approving the batch, and running it.
One weak payload spoiled the whole batch and could produce a false "fixed" — the one verdict a
revalidation tool must never get wrong.

That path is deleted (ADR-0033). In its place is
an interactive agentic console (FR-17) where the agent reasons, runs one command, observes the
real output, and decides the next — with a human approving every command before it touches the
target.

• Egress-locked sandbox. Each session gets a Docker --internal network with only the scoped
  target attached — no gateway, no route out. Command content is never inspected because it cannot
  matter: there is nowhere else to reach. Online targets go through a deny-all-by-default
  allowlisting proxy instead (ADR-0041).
• Guided by default. The agent does one action per turn and hands back, so you drive. Flip
  Auto-run and it drives itself (ADR-0040).
• One agent, one voice. Type into the console at any point — including at an approval prompt,
  which withdraws the pending command and steers instead
  (ADR-0042).
• You own the goal, you conclude, you can reopen a verdict you think was premature
  (ADR-0043), and the append-only transcript is
  what every verdict is derived from.

Also new since v0.4.0

• Reports chat (FR-18) — a read-only assistant over the whole corpus with typed query tools,
  answering token-by-token over SSE.
• CVSS + MITRE ATT&CK enrichment (FR-19) — copied verbatim when the report states them,
  derived and explicitly flagged inferred when it does not.
• Kali toolbox sandbox image, operator lifecycle controls (start/stop/resume/restart/reopen),
  streamed model reasoning, and scope-driven sandbox targeting.

Evaluation (FR-15) — the honest numbers

Twelve findings from a real external Juice Shop report, against the pinned unpatched v17.1.1 lab,
on a local qwen3.6 27B backend:

Correct	8
Inconclusive (safe hedge)	4
Confidently wrong	0

NFR-01's safety constraint held — the system never issued a false clearance, and hedged on the
four it could not confirm rather than guessing. Its ≥70% accuracy bar was missed at 67%, entirely
on those safe hedges.

The timing study is less flattering and is reported as such: a lean manual pass over the same twelve
findings took ≈60 min and resolved all twelve, against the guided run's 52 min and 8/12. Removing
the approval gate bought no meaningful speed (29.7 min vs 32.5 on the nine non-browser findings),
because the bottleneck is model latency, not the human. On this backend the tool is neither faster
nor more accurate than a careful operator; what it adds is that it constructs the checks, keeps an
append-only audit trail, and never issues a false clearance.

One report, one target, one operator, one backend. A first data point, not a study — the thesis says
so in its threats to validity.

Project state

• 44 ADRs, all accepted or superseded — none left proposed.
• 287 unit + 75 integration backend tests; 160 frontend tests.
• mypy --strict, ruff, xenon --max-absolute C, radon average A, vulture clean, pylint
  duplicate-code 9.99.
• Thesis: 104 pages, zero undefined references, Evaluation chapter included.

Full changelog: v0.4.0...v1.0.0

v0.4.0 — Trust & audit

v0.4.0 — Trust & audit

Milestone M4 (Trust & audit). The retest engine now defends its own results:
every probe runs through an independent sanity check, every verdict is provably
re-derivable from its stored evidence alone, and a complete run exports as one
versioned, schema-validated JSON document — the input the evaluation harness (M5)
will consume.

Highlights

• FR-08 — Execution sanity checker (#13): an independent verifier
  (guarded_run) wrapped around every approved probe. It re-checks the probe is a
  member of the approved plan before any socket opens — a deviation is
  fail-closed (PlanDeviationError → HTTP 409, no verdict stored) — and
  downgrades an over-confident verdict after (a fixed on a 404/410 or 3xx
  becomes inconclusive, since a moved endpoint is not a proven fix). It only ever
  removes confidence, never manufactures it (ADR-0014).
• FR-10 — Audit trail + verdict re-derivation (#15): a verdict is a pure
  function of its stored evidence, so rederive_run recomputes every verdict from
  the same deterministic assessment the live path uses and diffs it against
  storage — reproduced from the trail alone, no re-execution (the NFR-02 test).
  VerdictRecord gained created_at/actor; exposed at GET /api/audit
  (ADR-0015).
• FR-12 — Versioned run export (#17): GET /api/export returns a complete run
  (reports → findings → plans → verdicts + evidence + descriptive metrics) as one
  SCHEMA_VERSION-versioned JSON document. Its JSON Schema is generated from the
  model (GET /api/export/schema, published to
  docs/reference/schemas/run-export.schema.json) and drift-tested, so the
  published schema can never diverge from the document (ADR-0016).

Quality

• Backend: 180 unit tests @ 97% coverage + 22 integration; the three new M4
  modules (sanity.py, audit.py, export.py) each at 100%.
• Pre-release codebase-sanity audit: GO — all mechanical gates clean (xenon
  --max-absolute C, radon avg A, vulture, pylint 9.98, mypy --strict, ruff), no
  dead code / complexity regression / import-layer drift.
• Architecture docs updated for M4 (#62): C4 container + new sequence diagrams
  for the FR-08 guarded execution and the FR-10/FR-12 read-only derivations.
• ADRs accepted this cycle: 0014, 0015, 0016.

Next

M5 — Evaluation (→ v1.0.0): pin the ground-truth vulnerable lab + an expected
verdict per finding, then build the FR-15 evaluation harness that consumes this
release's JSON export into a reproducible metrics table (NFR-01: ≥70% correct
verdicts, zero confidently-wrong on ambiguity).

v0.3.0 — Plan, approve, retest, and an operable SPA

v0.3.0 — Plan, approve, retest, and an operable SPA

Milestone M3 (Plan & approve). The whole tool now runs end-to-end from a
browser on localhost: upload a pentest PDF, review/edit an AI-proposed
safety-gated retest plan, approve it, run the retest, and read
evidence-backed verdicts — still open / fixed / inconclusive.

Highlights

• FR-04 — Retest-plan generation (#9): an LLM proposes typed probe actions;
  a deterministic gate binds each to the allowlisted base URL and drops
  off-allowlist or destructive actions before anything runs (ADR-0011).
• FR-05 — Server-side approval gate (#10): versioned plans, a single
  execution chokepoint that refuses anything unapproved, edits/regenerates
  re-gated through the FR-06 allowlist, executed version stamped on each verdict
  (ADR-0012).
• FR-11 — React SPA, operable from the UI alone (#16): a Vite/React/TS/Tailwind
  app served by FastAPI at /, API under /api; PDF upload runs FR-01→FR-03 as a
  polled background job; the full plan → approve → retest → verdict flow with
  evidence + plan history is driven entirely from the browser (ADR-0013, accepted).
• UI redesign (#55): a dark/light-themeable "forensic verdict instrument"
  console — two-voice colour language (iris = the system's voice; red/amber/green =
  reality's verdict), a left-nav sidebar with a live recent-reports jump list,
  self-hosted IBM Plex fonts (offline, NFR-03), a determination-ledger meter and a
  revalidation pipeline track. Presentational only — no API or architecture change.

Quality

• Backend: 126 tests @ 96% coverage; frontend eslint/tsc/vitest in CI.
• Pre-release codebase-sanity audit: GO — mechanical gates clean, no dead
  code / complexity regression / architectural drift.
• ADRs accepted this cycle: 0011, 0012, 0013.

Known follow-ups (early M4)

• Frontend test health: add @vitest/coverage-v8 + a CI threshold and unit-test
  the redesign's pure logic (PipelineTrack, DeterminationMeter,
  activePlan/latestVerdict, errorMessage).
• De-duplicate the redesign's status→colour mapping and button/pill class strings
  into shared primitives.

Next

M4 — Trust & audit (→ v0.4.0): FR-08 execution sanity checker, FR-10 audit
trail + verdict re-derivation, FR-12 versioned JSON export.

v0.2.0 — M2 Report understanding

M2 — Report understanding

Second milestone: turn a real penetration-test PDF into schema-validated findings, with a model-agnostic LLM backend. Tagged at the M2 close-out commit (72ed61a); the already-merged M3 work (FR-04, FR-05) is intentionally excluded and ships in v0.3.0.

Features

• FR-01 — PDF report ingestion (#6, #42): src/revalid/pdf.py (pdfplumber, ADR-0007) extracts per-page/full text plus best-effort finding candidates and fails closed on non-PDF/corrupt/no-text input. Validated on the real OWASP Juice Shop evaluation report (Nozipho Mthimunye's public write-up, credited and kept local): 11 pages → 8 clean finding candidates with no manual preprocessing.
• FR-03 — LLM finding extraction (#8, #45): src/revalid/extract.py (Pydantic AI, ADR-0009) — per-candidate agent with a list[ExtractedFinding] schema gate; invalid output is flagged, not persisted; extraction lineage kept for the audit trail (NFR-02). make demo-extract runs the FR-01→FR-03 pipeline.
• FR-13 — Model-agnostic LLM config (#18, #47): src/revalid/llm.py (ADR-0010) — the backend is selected by REVALID_LLM_MODEL (default Claude; ollama:<model> + OLLAMA_BASE_URL for a local fallback). Switching is configuration-only; a live-Ollama system test skips when no server is reachable.

Milestone acceptance (Done-when)

The real Juice Shop PDF yields ≥90% well-formed findings — met: a full FR-01→FR-03 run on the local ollama:qwen3.5:9b backend produced 8/8 (100%) well-formed findings; the 4th of an earlier synthetic set was correctly held back by the schema gate on a weaker model.

Decisions ratified

ADRs 0007 (pdfplumber), 0009 (LLM extraction), 0010 (model-agnostic config) ratified proposed → accepted as part of this close-out (docs PR #52 on main; the ratification commit post-dates this tag, so the tree at v0.2.0 still shows them proposed).

Full range: v0.1.0..v0.2.0 — see the commit list below.

v0.1.0 — M1 walking skeleton

First milestone: a thin, deterministic end-to-end slice proving the architecture — ingest → probe → verdict. No LLM and no frontend yet (those begin in M2).

Included

• FR-02 — ingest DefectDojo-style JSON findings into SQLite via the API.
• FR-06 — target-authorization allowlist: an unbypassable httpx transport that refuses any target not explicitly allowlisted (SSRF guard), built only from trusted config.
• FR-07 — one verification-only HTTP probe (the OWASP Juice Shop SQLi login bypass), executed through the FR-06 guard, capturing full request/response/timing evidence.
• FR-09 — evidence-backed verdicts (still_open / fixed / inconclusive) with machine-readable reason codes, persisted and exposed at POST /findings/{id}/retest and GET /verdicts.

Run it

make lab-up                  # OWASP Juice Shop (pinned v17.1.1), localhost only
make demo-walking-skeleton   # prints INGEST / PROBE / EVIDENCE / VERDICT
make test-system             # asserts still_open against the live lab
make lab-down

Quality

Lint, mypy --strict, xenon complexity gate, and 96 unit/integration tests pass; coverage ~98%. Security review of the retest executor and a codebase-sanity audit both came back clean (HEALTHY).

Closes milestone M1 (issues #7, #11, #12, #14).