WARNING: this is meant to be run in a container because we skip all permissions

Skill Benchmark Runner

Runs a batch of prompts through claude -p with your skill loaded as a system prompt, and collects timing, token usage, cost estimates, and human-readable outputs.

Requirements

• claude CLI installed and authenticated
• jq (JSON parsing) — brew install jq or apt install jq
• bc (arithmetic) — usually pre-installed; brew install bc or apt install bc

Quick Start

# 1. Put your prompt files in a folder (one .txt file per prompt)
ls prompts-example/
#   01_hello_counter.txt
#   02_login_form.txt
#   03_settings_panel.txt

# 2. Run the benchmark
./run_benchmark.sh \
  --skill  /path/to/your-skill/SKILL.md \
  --prompts prompts-example/

# 3. Check results
ls results/20240101_120000/
#   summary.md          ← human-readable overview of all runs
#   stats.json          ← machine-readable stats for all runs
#   01_hello_counter.result.md
#   02_login_form.result.md
#   03_settings_panel.result.md
#   *.raw.json          ← raw claude CLI JSON output per prompt

Options

Flag	Description	Default
--skill	Path to your SKILL.md file	(required)
--prompts	Directory containing *.txt prompt files	(required)
--output	Where to write results	./results/<timestamp>
--model	Claude model to use	claude-sonnet-4-20250514

Output files

summary.md

One section per prompt showing the response (truncated at 60 lines) plus a totals table. This is the main file for human review.

stats.json

Structured JSON with per-run and aggregate stats:

{
  "skill": "path/to/SKILL.md",
  "model": "claude-sonnet-4-20250514",
  "timestamp": "2024-01-01T12:00:00Z",
  "prompts_total": 3,
  "prompts_ok": 3,
  "prompts_failed": 0,
  "totals": {
    "input_tokens": 4200,
    "output_tokens": 1800,
    "cache_read_tokens": 0,
    "cache_write_tokens": 0,
    "wall_secs": 18.4,
    "avg_wall_secs": 6.1,
    "estimated_cost_usd": 0.039
  },
  "runs": [...]
}

<prompt-name>.result.md

Full response for a single prompt plus its stats table. Good for deep-diving into one run.

<prompt-name>.raw.json

The raw JSON output from claude --output-format json. Useful for debugging.

Writing good prompt files

• One file = one test case
• Name files with a numeric prefix so they run in order: 01_basic.txt, 02_edge_case.txt
• Make prompts realistic — they should be things a real user would type
• Include at least one "happy path" prompt, one edge case, and one tricky/ambiguous prompt

Cost estimation

Costs are estimated based on Sonnet 4 pricing at time of writing. Update the *_COST_PER_MTK variables at the top of run_benchmark.sh if pricing changes. Actual billed cost may differ slightly due to rounding and prompt caching behaviour.

Comparing two skill versions

Run the benchmark twice with different skill paths and different output dirs:

./run_benchmark.sh --skill skill-v1/SKILL.md --prompts prompts/ --output results/v1
./run_benchmark.sh --skill skill-v2/SKILL.md --prompts prompts/ --output results/v2

# Then diff the summaries
diff results/v1/summary.md results/v2/summary.md