Skip to content

evalcore init: scaffold a new eval suite in one command #4

Description

@abhishekmanyam

Summary

Add evalcore init to scaffold a runnable eval suite in the current directory, so a newcomer's first successful run is one command away instead of hand-writing YAML.

Proposed behavior

evalcore init writes:

  • evals.yaml — a minimal suite with a shell target (works with zero API keys) and a contains scorer.
  • cases.jsonl — two or three example cases.
  • A tiny bot.sh stub the shell target calls.

Then prints the next command (evalcore run evals.yaml). Optionally evalcore init --openai scaffolds an openai-compatible target variant instead.

Constraints

  • Refuse to overwrite existing files unless --force.
  • Output must be deterministic (no timestamps in generated files).
  • The generated suite must pass evalcore run immediately, offline.

Good first issue

Mostly wiring in crates/evalcore (the bin) plus a CLI test with assert_cmd asserting the files land and the scaffolded suite runs green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions