SkillBench

SkillBench compares frozen versions of a Codex skill in fresh, isolated Codex sessions. It stores the workspace, skill, tool trace, patch, artifacts, verifier results, token use, timing, and estimated cost as local files. No service or database is required.

Use it to answer a narrow question such as: “Does this SKILL.md revision preserve the required result while reducing unnecessary reads and cost?” It is an experiment runner, not an automatic quality judge.

This repository is the complete standalone distribution. It does not depend on codex-playground; that workspace is only where SkillBench was incubated and where some product experiments are retained.

The repository includes:

• the skillbench Rust CLI;
• the canonical repo-local $skillbench skill;
• a generic count-lines smoke example;
• fake-Codex tests and the evidence analysis guide.

Requirements

• macOS, Linux, or another environment with a POSIX sh;
• a current Rust toolchain with Cargo and Rust 2024 edition support;
• Codex CLI, authenticated with codex login or CODEX_API_KEY;
• git for isolated workspace baselines;
• curl only for downloading the optional LiteLLM price snapshot;
• any commands used by your case verifiers.

Check the required commands before installing:

rustc --version
cargo --version
codex --version
git --version
curl --version

Install

Install the binary directly from GitHub:

cargo install --locked --git https://github.com/SunChJ/skillbench.git
skillbench --help

Clone the repository when you also want the example and repo-local Codex skill:

git clone https://github.com/SunChJ/skillbench.git
cd skillbench
cargo install --locked --path .
skillbench --help

Build a checkout-local binary without changing global installation:

cargo build --locked --bin skillbench
target/debug/skillbench --version

SkillBench has no separate login. A real benchmark reuses the active Codex login or the CODEX_API_KEY environment variable. Do not put credentials in a case workspace.

Install the Codex skill

Codex discovers the repo-local skill at .agents/skills/skillbench when working in this checkout. Invoke it with a request such as:

Use $skillbench to design a one-run, medium-reasoning benchmark for this SKILL.md change. Dry-run first.

To make the skill available outside this checkout, copy it into your personal Codex skills:

destination="${CODEX_HOME:-$HOME/.codex}/skills/skillbench"
mkdir -p "$destination"
cp -R .agents/skills/skillbench/. "$destination/"

The skill guides installation, case design, dry runs, paid runs, comparison, analysis, and safety. Its detailed case reference is at .agents/skills/skillbench/references/case-format.md.

Five-minute quick start

The included example compares two versions of a count-lines skill. First initialize local evidence storage:

skillbench init

In another Git repository, add /.skillbench/ to that repository's .gitignore; both dry and real runs persist frozen evidence there. This SkillBench source checkout already ignores it.

Then freeze the case and both skills without calling a model:

skillbench bench examples/count-lines/case.json \
  --variant baseline=examples/count-lines/variants/baseline \
  --variant candidate=examples/count-lines/variants/candidate \
  --runs 1 \
  --model gpt-5.6-sol \
  --reasoning medium \
  --sandbox workspace-write \
  --dry-run \
  --root .

The JSON output should contain one casepack_id, two skillpack_id values, and repetitions: 1. A dry run creates frozen packs but launches zero Codex sessions.

When you intentionally want the real comparison, promote those exact frozen IDs instead of reading the source paths again:

skillbench bench cpk_FROM_DRY_RUN \
  --variant baseline=spk_BASELINE_FROM_DRY_RUN \
  --variant candidate=spk_CANDIDATE_FROM_DRY_RUN \
  --runs 1 \
  --model gpt-5.6-sol \
  --reasoning medium \
  --sandbox workspace-write \
  --root .

This example starts two real Codex sessions: 2 variants × 1 run. The command prints a comparison table followed by a stable cmp_* ID.

Inspect the evidence and start a cost-free human analysis:

skillbench show cmp_...
skillbench analyze cmp_...
skillbench analyze note ana_... --file review.md --author your-name

skillbench analyze cmp_... defaults to --by human; it creates an evidence bundle but does not call a model. skillbench analyze cmp_... --by codex and every Codex resume add a model session.

Build a real benchmark

Keep the benchmark next to its fixture and variants:

benchmarks/my-skill/
├── case.json
├── workspace/
├── variants/
│   ├── baseline/SKILL.md
│   └── candidate/SKILL.md
└── README.md

Both variants should use the same frontmatter name. Explicitly invoke that name in the prompt so the experiment measures the skill body instead of trigger variance.

A case file looks like this:

{
  "name": "review a known fixture",
  "workspace": "workspace",
  "prompt": "Use $review-skill to inspect target.md and write report.md.",
  "exclude": [".env", "node_modules", "target"],
  "artifacts": ["report.md"],
  "verify": [
    "test -f report.md",
    "python3 verify.py"
  ],
  "metadata": {
    "decision": "preserve recall while reducing cost"
  }
}

Paths resolve relative to case.json. The workspace and each skill directory are copied into a SHA-256 content-addressed store before execution. Put verifier scripts inside the workspace when they must run from the isolated final tree.

Use deterministic verifiers for observable requirements and safety boundaries. Prefer gold labels, schemas, exact files, and exit codes over another model judge. Evaluate outcome and safety before efficiency.

Before a real run:

1. Start with --runs 1, --reasoning medium, and --dry-run.
2. Confirm the CasePack, SkillPacks, model, reasoning, sandbox, and expected session count.
3. Use read-only for audits and workspace-write only when the result requires edits.
4. Use the dry run's cpk_* and spk_* IDs for the real run; keep every model setting fixed.
5. Increase repetitions only when the sentinel run is valid and variance matters to the decision.

Fresh skillbench init projects default to one run and medium reasoning. Existing .skillbench/config.json files keep their current settings, so specify both flags when cost matters.

The first variant is the comparison baseline. Across repeated runs, SkillBench rotates variant execution order to reduce simple order bias.

Commands

skillbench init [DIR]
skillbench pack skill PATH [--label NAME]
skillbench pack case CASE.json
skillbench bench CASE.json|CPK_ID --variant NAME=PATH|SPK_ID [...]
skillbench compare RUN_ID...
skillbench show ID [--json]
skillbench list [runs|comparisons|analyses|skillpacks|casepacks]
skillbench materialize ID DESTINATION
skillbench analyze COMPARISON_ID [--by human|codex]
skillbench analyze note ANALYSIS_ID --file NOTE.md [--author NAME]
skillbench analyze resume ANALYSIS_ID "follow-up"
skillbench analyze fork ANALYSIS_ID [--by human|codex]
skillbench prices update

Useful benchmark options:

• --root DIR: project containing .skillbench; defaults to the current directory.
• --runs N: repetitions per variant; total benchmark sessions are variants × runs.
• --model MODEL: Codex model.
• --reasoning LEVEL: reasoning effort.
• --sandbox read-only|workspace-write|danger-full-access: Codex sandbox.
• --dry-run: freeze inputs and print the plan; do not launch Codex.
• --offline: do not download a missing price snapshot; Codex still runs.

Run skillbench --help for the current CLI surface.

Source paths create new packs. Frozen cpk_* and spk_* IDs load immutable local packs from the selected .skillbench root. Use IDs to guarantee that an approved dry run and the real run use the same case, workspace, and skills. Model, reasoning, sandbox, and repetition settings remain CLI parameters and must still be repeated exactly. Pack IDs are local to that evidence root; preserve and reuse the same --root through execution and analysis.

Evidence layout

All state lives under the selected benchmark root:

.skillbench/
├── config.json
├── store/sha256/
├── casepacks/cpk_.../
├── skillpacks/spk_.../
├── runs/run_.../
├── comparisons/cmp_.../
├── analyses/ana_.../
└── prices/

A run_* directory includes the run manifest, Codex events, local session JSONL when found, last message, stderr, metrics, verifier logs, patch, Git status, selected artifacts, and a final-tree snapshot. A cmp_* directory includes JSON and Markdown reports. An ana_* directory contains an append-only analysis history and a regenerated summary.

Restore a frozen run without touching the source fixture:

skillbench materialize run_... ./restored-run

To compare one new candidate against a compatible archived baseline without rerunning the baseline:

skillbench bench cases/example.json --variant candidate=variants/candidate --runs 1
skillbench compare run_ARCHIVED_BASELINE run_NEW_CANDIDATE

Manually confirm the same case definition, model, reasoning, sandbox, Codex version, verifier, external dependencies, and price snapshot before interpreting an incremental comparison.

Cost and safety

• --dry-run is the only benchmark flag that guarantees zero model calls.
• --offline only disables a price download. It does not disable Codex.
• Estimated dollars come from a pinned LiteLLM price snapshot and are not an OpenAI bill.
• Case verify strings execute through host sh -c, outside the Codex sandbox. Run only trusted cases.
• .git and .skillbench are excluded automatically; explicitly exclude .env, credentials, personal data, dependency trees, and build output before freezing a workspace.
• Frozen session logs and tool output may contain sensitive text. Review .skillbench before sharing it.
• danger-full-access and fixture commands can push, deploy, or call external services. Use a disposable fixture and the narrowest sandbox that satisfies the case.

On Unix, SIGINT and SIGTERM stop the active child, mark the run aborted, and remove its temporary workspace and CODEX_HOME before SkillBench exits.

Development

cargo fmt -- --check
cargo clippy --all-targets -- -D warnings
cargo test --all-targets

The tests use a fake codex executable and do not make model calls. For analysis principles, see analysis-guide.md.