Comparing evaluation runs (baselines, regressions) - plans for RubyLLM::Evaluation?

[justi]

Hi Carmine,

Congrats on 2.1 - RubyLLM::Evaluation looks great, and having datasets, model grading and the rake task in core makes a lot of sense.

The "Comparing Runs" section of the running guide says an evaluation is most useful when you compare runs over time - before and after a prompt change, or across two models. Today that comparison is left to the reader. Do you plan to add it to RubyLLM::Evaluation (e.g. diffing two saved reports, a baseline file, a rake run that fails on regressions), and would you rather have it in core or in an extension?

I'm asking because this is the part of RubyLLM::Contract that overlaps most, and I'd rather contribute than build a parallel version. What it does today, on its own eval DSL:

# CI: fail when a case that passed in the saved baseline now fails
expect(SummarizeArticle).to pass_eval("regression").without_regressions

RubyLLM::Contract::RakeTask.new do |t|
  t.minimum_score      = 0.8
  t.maximum_cost       = 0.05   # suite budget; refuses when a case cannot be priced
  t.fail_on_regression = true
end

# Prompt A/B on the same cases
SummarizeArticleV2.compare_with(SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini").safe_to_switch?

# Same cases across models, cheapest one above a score
SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }])
                .best_for(min_score: 0.95)

The baseline diff is per case: regressed, improved, newly skipped, plus the score delta.

If it fits your plans, I'd be happy to open a PR for something small in core, for example a comparison of two Evaluation::Reports (per-case regressed / fixed, pass-rate and cost delta) and an option for the rake task to compare against a saved report. If you'd rather keep core minimal, I'd look into building these on top of Evaluation::Report in Contract, so the two don't drift apart.

Either way, a pointer to the direction you have in mind would help me avoid working against it.

Thanks!