Repository navigation
RFC: First-class LLM evaluation as a stack component (evaluator type)
#4787
Replies: 4 comments
|
Implementation in flight as draft PR: #4788 Two architectural findings worth flagging here as well (also documented in the PR):
Code is at: https://github.com/demian-overflow/zenml/tree/feat/llm-evaluator-stack-component Open questions in the PR description; happy to adjust scope before more code lands. |
|
Hey @demian-overflow, thank you for contributing this RFC. It’s a strong starting point for a relatively big feature. Unfortunately, our backlog is tight for the next few weeks, but I’ll surface this internally and we’ll follow up on the discussion when we’ve had a chance to dig in 👍 One thing that would help us when we review this: what motivated you to propose this RFC (and the PR)? is this driven by something concrete you’re building or evaluating in production (or planning to), or more of a “gap in ZenML” you’d like to see filled for the ecosystem? Any brief note on your use case or constraints would make it easier to prioritize and give useful feedback. |
|
The stack-component direction makes sense to me, especially because it keeps evaluation close to the pipeline artifact graph instead of treating it as an external dashboard afterthought. A few design points I would consider locking in early:
I would avoid allowing evaluators to mutate inputs. Redaction or normalization should happen in an explicit upstream step, but the evaluator should record whether it received raw, redacted, or synthetic inputs so the result remains audit-ready. |
Uh oh!
There was an error while loading. Please reload this page.
TL;DR
Add a new
evaluatorstack component type that makes LLM evaluation a first-class concept in ZenML — peer toexperiment_trackerandmodel_registry. Pipelines call typedevaluate_pointwise / evaluate_reference / evaluate_ragsteps, produce a typedEvaluationResultartifact, compare against a baseline (default: pulled from the model registry), and gate the run via the existing event-trigger system (default: emit anEvalRegressionevent; opt in to hard-fail).This is Phase 1. A classical-ML mirror (
ModelEvaluator) follows in Phase 2 once the abstraction is proven.Why this, why now
ZenML has
data_validators(Evidently, Deepchecks, Great Expectations) for data checks, andmodel_registriesfor tracking deployed models — but no first-class abstraction for evaluating model/pipeline outputs over time. With the recent investment in agentic / LLM workflows (thememory_callstep type, event triggers, dynamic pipelines, langchain/openai/llama_index integrations), this gap is now load-bearing.This proposal closes it without inventing parallel machinery — it reuses the existing
StackComponentframework, the typed-artifact system, the model registry, and the new event-trigger pipeline.Locked-in decisions (after brainstorming)
evaluator)OpenAIEvaluator,RagasEvaluator(two flavors prove the abstraction isn't shaped to one library)trigger-only(emits event; doesn't fail the run)What v1 scope deliberately excludes
Pipeline-facing API sketch
Two distinct knobs, kept separate by design:
pass_thresholds— per-case pass/fail. Drivespass_rateaggregate.regression_tolerance— how much an aggregate can drop versus baseline before it counts as a regression. Default0.0; set higher to absorb judge variance.Architecture summary
BaseEvaluatorinsrc/zenml/evaluators/base_evaluator.pydeclaressupported_modesand threeevaluate_*methods. Default impls raiseEvaluationModeNotSupportedErrorso flavors opt in to modes incrementally.PointwiseCase,ReferenceCase,RAGCase) are Pydantic models — explicit per-mode inputs, not adictblob.EvaluationResultis a typed artifact (per-case scores, aggregate metrics, metadata linking to model version + dataset). Custom materializer; the dashboard can render it specially.@stepfunctions (evaluate_pointwise / evaluate_reference / evaluate_rag) resolve the active stack's evaluator, compute pass/fail and aggregates, fetch the baseline, and apply the gate.explicit(by artifact_id),model_registry(default — pulls EvaluationResult attached to current production version, matched bysuite_name), andnone.failraises and aborts the run;warnflags the artifact;trigger(default) emits anEvalRegressionEventinto the existing event-trigger system. New event source registration; no new trigger machinery.Spec + implementation plan
Full design with rationale, every interface signature, and concrete flavor sketches:
📄 Spec:
docs/superpowers/specs/2026-04-30-llm-evaluator-stack-component-design.md📋 Implementation plan (15 tasks, TDD):
docs/superpowers/plans/2026-04-30-llm-evaluator-stack-component.md🌿 Branch:
feat/llm-evaluator-stack-component(implementation in progress; a draft PR will be linked here once Tasks 1–9 land)Specific feedback I'd like
Open questions deliberately left for this discussion:
RagasEvaluatoras a separate flavor vs. consolidated intoOpenAIEvaluator(Ragas can use OpenAI as its judge). Current pick: separate, to prove the abstraction. Reasonable to argue redundancy.EvalRegressionas a single event vs. split by severity. Current pick: single event with full delta in payload; severity inferred by handlers.regression_tolerance. Reviewer may push for per-metric ({"faithfulness": 0.05, ...}). Current pick: scalar in v1, per-metric in Phase 2.Beyond those, any structural concerns about:
evaluatorname (alternatives:eval_runner,judge,scorer)?on_regression="trigger"— too permissive? Worth defaulting to"warn"instead?OpenAIEvaluatorinto the existingopenaiintegration vs. a dedicatedopenai_evaluatorpackage?Status
Implementation kicking off now in subagent-driven execution against the linked plan. Phase 1 lands in this branch; Phase 2 (
ModelEvaluator) will get its own RFC after this one is reviewed.Happy to adjust scope or design based on feedback before too much code lands — that's the point of opening this early.
All reactions