Evals?

[renatodeleao]

Hello thanks for rubby_llm first and foremost 🙏

I'm building my first "agentic" features so it's all new to me (have more of a frontend brackground). I'm now in the quality improvements phase, and it seems there is no standard among the ruby/rails community.

• ruby_llm-tribunal is the first contender from search, but not yet mature (special for structured outputs testing) and not sure if the authors are going forward with it). It resonates a lot because it feels right at home, like we're writing unit tests ( describe > context (when) > expect).
• All other popular frameworks (promptfoo, DeepEval) seem to follow a more dataset based approach instead (start from a set of golden prompts) and then write tests from there (for those specific prompts) not for generic "per feature/case" ones, which is a bit of different mental model when writing. (At least i fell that way, and there's probably good a valid reasons for it, cartesian explosion from cases [prompt1, prompt2, ...] >when context > expect approach might be one and evals actually cost money to run (each prompt runs just x times, not x times x n contexts, guessing here though).

So what's your take? Any strong opinions? How do you approach it? Doesn't matter because ai writes it anyways?

Could evals be part of the ruby-llm core framework or it's out of scope?

Cheers 👋

[vishalhabib99]

Short version: both styles are right, they answer different questions, and you'll want both. The mental-model split you noticed is real.

Per-feature spec tests (tribunal style) answer "does this feature still behave?" They're regression tests. Keep them few, deterministic where possible, and run them in CI on every prompt or model change.

Dataset evals (promptfoo / DeepEval style) answer "how good is it, and on what share of real inputs does it fail?" You only get a rate from a set of cases. One expect per context tells you pass or fail on that one case, not that the feature works 92% of the time.

What has worked for me on agent features:

1. Write your checks as code first, use an LLM judge last. Most of what goes wrong is checkable without a model: did the JSON parse, are the required keys present, did it call the right tool with the right arguments, did it refuse when it should have. Use plain RSpec assertions for these. They're free, fast and don't flake. Only send the fuzzy part ("is this summary faithful?") to an LLM judge, and spot-check the judge against your own labels before you trust it.
2. Build the dataset from real failures, not from imagination. Start with 20–30 cases. Every bug report or bad output you see becomes a new case. That keeps the set small and relevant, and it avoids the combinatorial blowup you mentioned. You aren't crossing every prompt with every context, you're collecting the ones that actually broke.
3. Keep a held-out set. Once you've tuned the prompt against your cases, the score on those same cases looks better than reality. Set aside a handful you never look at while iterating, and check against them before calling it done. This catches more "it passed my evals but failed in prod" than anything else I've tried.
4. Run each case more than once. At temperature > 0 a single run is a coin flip on borderline cases. 3 runs per case and a pass threshold (for example 2 of 3) is enough for most features, and it shows which cases are flaky as opposed to broken.
5. Cost control: run the deterministic checks on every commit, and the full dataset with judges only on prompt or model changes or nightly. Record responses (VCR/webmock) so the spec layer replays them for free.

In Ruby terms, a dataset eval can still look like RSpec. Load cases from YAML, loop over them, and have each one generate an it. You keep the describe/expect ergonomics you like, and the data stays in one file that grows as you find failures.

On whether this belongs in ruby_llm core: I'd guess not the framework, since judges, datasets and reporting are opinionated and move fast. But a small hook that makes it easy to capture request/response pairs into a dataset file would make every external eval tool easier to use. That's the maintainers' call, though.

[crmne]

Evals shipped with 2.1