v0.1.0
Run your cyber-AI agent against a real mission. Watch everything it does. Grade the evidence.
pip install xorcise
xorcise up # boots the stack, prints the console URLNeeds Python 3.12+ and Docker Engine. Tested on Ubuntu. xorcise doctor checks the host first;
xorcise up --stub runs the full loop without Docker.
Added
- CLI-first single distribution —
pip install xorcise, thenxorcise upboots the
whole stack on one host and prints the console URL;xorcise doctorchecks the host,
xorcise downtears everything down. A stub mode (xorcise up --stub) runs the full
loop without Docker. - Agent registry — register any cyber-AI agent (OpenHands, Claude Code CLI, Codex CLI,
or anything custom) as the unit of evaluation; runs and results accrue to the agent's
track record, with versioned re-declarations for comparability. - Missions — real target environments packaged as pullable images (a free mission
library) or authored locally as bundles (mission.json+ compose stack) and ingested
into a locally built, fused mission image. - Isolated runs — each run gets its own private WireGuard tailnet (Tailscale +
Headscale) with a per-run ACL as the hard network boundary: the agent can reach exactly
its one mission and nothing else. Environments are created per run and destroyed at
teardown. - Evidence collection — a built-in OpenTelemetry collector captures every command,
tool call and message the agent emits (traces and logs), correlated per run; harness
adapters normalise the raw telemetry into a replayable event stream. - 50/50 evidence-anchored grading — deterministic, mission-defined checks (the
reproducible half) combined with a rubric-bound, bring-your-own-model LLM judge (the
qualitative half), graded over the sealed run evidence, never the agent's claims. - Run control for agents — a bearer-authenticated REST surface for the agent under
test: fetch the mission brief and intel, submit artifacts, and mark the run done. - Results and reporting — per-run scorecards with full breakdowns, exportable run
reports (Markdown/HTML), per-run event exports (JSONL), agent history, and a
leaderboard across recorded results. - Web console — a live trace feed, terrain map, run replay, results, and settings UI
served by the same process at/ui. - Deployment flexibility — all-local by default; a distributed topology supports
relocating the mission plane to a remote host while the control plane and console stay
local. - Live lifecycle feedback —
xorcise up's frontend install/rebuild andxorcise down's stop/reap/teardown steps show a live spinner with real build phases and elapsed
time on a terminal (piped output stays line-based for scripts). - Update notice —
xorcise upmentions a newer released version when one exists on
PyPI (checked in the background, cached for a day, silent on any failure; opt out with
XORCISE_NO_UPDATE_CHECK=1; source checkouts are never nagged).
Links — Website · Documentation · PyPI · Changelog · Contributing · Security
Missions are deliberately vulnerable by design. Run XORCISE on infrastructure you are
willing to lose — a dedicated VM or an isolated cloud environment, never a workstation
holding credentials you care about.