Summary
Add evalcore init to scaffold a runnable eval suite in the current directory, so a newcomer's first successful run is one command away instead of hand-writing YAML.
Proposed behavior
evalcore init writes:
evals.yaml — a minimal suite with a shell target (works with zero API keys) and a contains scorer.
cases.jsonl — two or three example cases.
- A tiny
bot.sh stub the shell target calls.
Then prints the next command (evalcore run evals.yaml). Optionally evalcore init --openai scaffolds an openai-compatible target variant instead.
Constraints
- Refuse to overwrite existing files unless
--force.
- Output must be deterministic (no timestamps in generated files).
- The generated suite must pass
evalcore run immediately, offline.
Good first issue
Mostly wiring in crates/evalcore (the bin) plus a CLI test with assert_cmd asserting the files land and the scaffolded suite runs green.
Summary
Add
evalcore initto scaffold a runnable eval suite in the current directory, so a newcomer's first successful run is one command away instead of hand-writing YAML.Proposed behavior
evalcore initwrites:evals.yaml— a minimal suite with ashelltarget (works with zero API keys) and acontainsscorer.cases.jsonl— two or three example cases.bot.shstub the shell target calls.Then prints the next command (
evalcore run evals.yaml). Optionallyevalcore init --openaiscaffolds anopenai-compatibletarget variant instead.Constraints
--force.evalcore runimmediately, offline.Good first issue
Mostly wiring in
crates/evalcore(the bin) plus a CLI test withassert_cmdasserting the files land and the scaffolded suite runs green.