-
Notifications
You must be signed in to change notification settings - Fork 0
Evaluation and Benchmarks
This page is the honest ledger: what has actually been measured, what hasn't yet, and where the canonical records live. Claims are limited to recorded evidence — published limitations included.
- Deterministic test suites — 700+ tests across three groups (core, lifecycle, trajectory) on Windows and Linux CI, covering the checker, the Init/Doctor lifecycle, path confinement, memory integrity, and adapter parity.
-
Canonical/generated parity — every adapter and the plugin package are regenerated from
skills/docsand compared byte-for-byte in CI. - Dogfooding — the product is exercised against real repositories (including this one) as part of release verification, with zero-write proofs on diagnostic routes.
Provenance-bound model evaluation fixtures and collectors exist in-repo (evals/), with sanitized, reproducible attempt records. The canonical, versioned records are:
- EVALUATION.md — methodology, fixtures, and recorded results.
- BENCHMARK.md — benchmark design and current status.
The broader 108-trajectory model matrix and complete cross-harness live pilots have not run yet. That limitation is published rather than hidden, and the roadmap gates them on established cost expectations and owner approval.
Per-host compatibility is likewise tiered by evidence — live-tested, structurally tested, or generated-only — in the dated compatibility matrix. The project does not claim universal compatibility.
The structural percentage is scope-qualified and versioned (rubric v2). It is reported separately from Trust coverage (are declared current-truth routes verified?) and freshness (do recorded content hashes still match?). Byte measurements of the map and current-state pages are telemetry with provenance — they never feed the score, fail health, or justify deletions.