__     _______ ____  ___ _____ ___ _____ ____  _        _    ____  
 \ \   / / ____|  _ \|_ _|  ___|_ _| ____|  _ \| |      / \  | __ ) 
  \ \ / /|  _| | |_) || || |_   | ||  _| | |_) | |     / _ \ |  _ \ 
   \ V / | |___|  _ < | ||  _|  | || |___|  _ <| |___ / ___ \| |_) |
    \_/  |_____|_| \_\___|_|   |___|_____|_| \_\_____/_/   \_\____/ 

Verifier Assurance Lab

Stress-test graders, reward functions, and policy checkers under optimization — not another model leaderboard.

verifierlab · CLI: valab

---

Pitch

VerifierLab is a local-first lab for running reproducible campaigns that ask a hard question: when something is optimized against your verifier, does the verifier still do its job?

You wrap a grader or reward function, run ordinary baselines alongside optimized-tagged attack cohorts under declared budgets and access models, then rebuild reports from content-addressed artifacts after freeze → adjudicate → release-labels. Finding no exploit is evidence under those conditions — not a proof of correctness. This release candidate (0.2.0rc2) meets executable RC gates 1–6 in tests/test_acceptance_gates.py; do not claim soundness or SOTA verifier assurance. Honest gaps (default process-local trust boundary, research-grade attack depth) are listed in docs/limitations.md.

Why it exists

Modern agents and training loops lean on verifiers: graders, reward models, rubrics, policy checkers. Those components can be gamed — reward hacking, rubric gaming, eval cheating — and the failure modes only show up under pressure.

VerifierLab exists so you can:

• Separate ordinary behavior from optimized-tagged attack cohorts
• Keep runs reproducible with content-addressed artifacts and digests
• Stay local-first: the base install has no PyTorch, Ray, Kubernetes, or model SDKs
• Record budgets (queries, steps, wall time) instead of pretending attacks are free

Status

Release candidate (0.2.0rc2). Gates 1–6 are encoded and passing in tests/test_acceptance_gates.py (isolation, optimization, measurement, repair, reproduction, developer lifecycle). Core loop — workspace, process workers, typed decisions, broker metering with worker budget vouchers, vault v2, freeze→adjudicate→release, StatsPlan reports, packs A–F, and the offline tutorial — is implemented and tested. See docs/limitations.md and SECURITY.md. Optional adapters and launchers ship behind extras or explicit dry-run modes. APIs and CLI may still shift before 0.2.0.

Install

Requires Python >=3.11,<3.14. uv is recommended.

git clone https://github.com/fraware/verifierlab.git
cd verifierlab
uv sync --extra dev
# or: pip install -e ".[dev]"

uv run valab doctor
uv run valab --version

From a published package (when available on your index):

pip install verifierlab
valab doctor

Quick start

Offline refund tutorial (full lifecycle):

uv run valab init
uv run valab inspect examples.refunds.verifier:grade
uv run valab run campaigns/refund-blackbox.yaml
uv run valab campaign freeze .valab/runs/<run-id>
uv run valab campaign adjudicate .valab/runs/<run-id> --campaign campaigns/refund-blackbox.yaml
uv run valab campaign release-labels .valab/runs/<run-id>
uv run valab report builds .valab/runs/<run-id>

Faster smoke:

uv run valab campaign validate campaigns/fake-smoke.yaml
uv run valab campaign run campaigns/fake-smoke.yaml --processes

What you can do

Goal	How
Wrap a verifier	Decorate a grader with @verifier, then valab inspect MODULE:ATTR
Run a campaign	valab run PATH or valab campaign run PATH (attack plane only)
Freeze, adjudicate, release	freeze → adjudicate → release-labels before reports
Rebuild a report	valab report builds RUN_DIR → HTML / JSON / CSV (post-release)
Compare a repair	Call compare_repair in Python: regression corpus plus a mandatory fresh attacker
Plan sample size	valab stats power for binomial power / n planning

Full CLI table: docs/cli.md.

Optional extras

Keep the base install light. Pull only what you need:

Extra	Use
gym	Gymnasium adapter
inspect	Inspect AI adapter
harbor	Harbor adapter (Python ≥3.12)
openenv	OpenEnv adapter
nemo	NeMo Gym HTTP client surface
objectstore	S3-compatible object store
kubernetes	Kubernetes launcher client
adapters	Bundle of the adapter extras above

uv sync --extra gym
# or: pip install -e ".[gym]"

Details: docs/adapters.md.

Documentation

Doc	Topic
Getting started	Install and tutorial
Concepts	Verifiers, cohorts, store, access models
Architecture	Engine, store, run bundles
Methodology	How to interpret campaign results
Attacks	Built-in strategies
Labels and statistics	Vault, freeze, error rates
Threat model	Assets, boundaries, non-goals
Limitations	What not to claim
Security policy	Vulnerability reporting

Contributing

Contributions are welcome — new attack strategies, adapters, campaigns, docs, and bug fixes all help.

• Read CONTRIBUTING.md for setup, PR expectations, and coding standards
• Browse open issues, especially ones labeled for newcomers
• Keep the base dependency graph light; heavy stacks belong behind extras
• Follow the Code of Conduct

uv sync --extra dev
uv run ruff check src tests
uv run mypy
uv run pytest
uv run python scripts/check_base_imports.py

Security

Do not open public issues for security-sensitive reports. Prefer a private GitHub Security Advisory, or see SECURITY.md.

License

Apache-2.0.

Links

• Repository: https://github.com/fraware/verifierlab
• Issues: https://github.com/fraware/verifierlab/issues
• Changelog: CHANGELOG.md