Summary
Add classic text-overlap scorers so users grading summarization / translation / generation quality don't have to reach for a judge every time. These are cheap, deterministic, and offline — a good fit for the cassette model.
Ask
Three new built-in scorers, one per file in crates/evalcore-scorers/, each following the new-scorer skill (schema in evalcore-config → impl → factory wiring → tests → docs):
bleu — sentence-level BLEU vs. an expected reference.rouge — ROUGE-L (and optionally ROUGE-1/2) vs. reference.levenshtein — normalized edit distance, with a threshold.
Acceptance
- YAML surface designed first (
type: bleu, reference: ..., threshold: ...).
- Deterministic: identical inputs → identical scores, no clock, no network.
- Unit tests inline covering a match, a partial, and an empty/edge case.
- A short docs section + a line in the scorers reference.
Good first issue
Self-contained — the contains/regex scorers are a clean template. Ping me and I'll point you at the skill checklist.
Summary
Add classic text-overlap scorers so users grading summarization / translation / generation quality don't have to reach for a judge every time. These are cheap, deterministic, and offline — a good fit for the cassette model.
Ask
Three new built-in scorers, one per file in
crates/evalcore-scorers/, each following thenew-scorerskill (schema inevalcore-config→ impl → factory wiring → tests → docs):bleu— sentence-level BLEU vs. an expected reference.rouge— ROUGE-L (and optionally ROUGE-1/2) vs. reference.levenshtein— normalized edit distance, with athreshold.Acceptance
type: bleu,reference: ...,threshold: ...).Good first issue
Self-contained — the
contains/regexscorers are a clean template. Ping me and I'll point you at the skill checklist.