feat: Rule-Based Evaluators (Contribution Proposal)

[cpoepke]

Problem

All evaluations currently require an LLM call. Deterministic checks like "output contains disclaimer" or "response is valid JSON" don't need an LLM, they cost money, add latency, hit rate limits, and require API key configuration.

Proposed Solution

A new RULE_BASED evaluator type that evaluates JSON-serializable conditions in the worker. No LLM call, no code execution, no new dependencies. Reuses the existing JobConfiguration → JobExecution → Score pipeline unchanged — only the execution step branches.

Rule Schema

type RuleDefinition = {
  combinator: "and" | "or";
  conditions: RuleCondition[];
};

type RuleCondition = {
  variable: string;       // template variable (same as LLM evals)
  operator:
    | "equals" | "not_equals"
    | "contains" | "not_contains"
    | "starts_with" | "ends_with"
    | "greater_than" | "less_than" | "greater_equal" | "less_equal"
    | "is_empty" | "is_not_empty"
    | "is_valid_json"
    | "matches_regex";  // validated at save time, executed via re2 (linear-time, no ReDoS)
  value?: string | number; // not needed for is_empty, is_not_empty, is_valid_json
  castToNumber?: boolean;  // parse variable as number before numeric comparison
};

Operators mirror InMemoryFilterService / filterOperators — already battle-tested in the codebase.

Example

"Output contains disclaimer AND is valid JSON":

{
  "combinator": "and",
  "conditions": [
    { "variable": "output", "operator": "contains", "value": "not financial advice" },
    { "variable": "output", "operator": "is_valid_json" }
  ]
}

Scoring

• value: 1.0 (all/any conditions pass) or 0.0
• comment: Auto-generated reasoning, e.g. "Rule failed: condition 2 — output is not valid JSON"
• source: EVAL, dataType: NUMERIC (same as LLM evals)

Execution Flow

No changes to job creation or queuing. The only branch point is in evaluate():

evaluate() fetches template
  ├─ template.type == LLM        → executeLLMAsJudgeEvaluation() (existing)
  └─ template.type == RULE_BASED → executeRuleBasedEvaluation()  (new)
                                    ↓
                                    Evaluate conditions in-memory
                                    ↓
                                    buildScoreEvent() → uploadScore() → enqueueScoreIngestion()
                                    (same score persistence as LLM evals)

Same dispatch applies in observationEvalProcessor.ts for observation-level evals.

Database Changes

CREATE TYPE eval_template_type AS ENUM ('LLM', 'RULE_BASED');

ALTER TABLE eval_templates
  ADD COLUMN type eval_template_type NOT NULL DEFAULT 'LLM',
  ADD COLUMN rule_definition JSONB;

Implementation Plan

#	Where	What
1	packages/shared/prisma/	Migration + schema: EvalTemplateType enum, type and rule_definition columns
2	packages/shared/src/features/evals/types.ts	RuleDefinition / RuleCondition Zod schemas
3	worker/src/features/evaluation/executeRuleBasedEvaluation.ts	Core rule engine + unit tests
4	worker/src/features/evaluation/evalService.ts	Dispatch in evaluate() based on template.type
5	worker/src/features/evaluation/observationEval/observationEvalProcessor.ts	Same dispatch for observation evals
6	web/src/features/evals/server/router.ts	Accept type + ruleDefinition in template CRUD
7	web/src/features/evals/components/template-form.tsx	Type selector: LLM vs Rule-Based
8	web/src/features/evals/components/rule-builder.tsx	New rule condition builder UI (modeled after filter-builder.tsx)
9	web/src/features/evals/components/inner-evaluator-form.tsx	Hide LLM fields (model/provider) for RULE_BASED
10	Tests	Integration tests in web/__tests__/async/ and worker/src/__tests__/

Questions for Maintainers

1. Regex engine: Proposal uses re2 (linear-time, no ReDoS). Alternative: built-in RegExp with a timeout. Preference?
2. Nested rules: Flat and/or only, or support nested groups like (A AND B) OR (C AND D)?
3. Scoring model: Always binary 1.0/0.0, or support proportional scoring (3/5 pass = 0.6)?
4. RBAC: Reuse existing evalTemplate:CUD / job:CUD scopes?

[hassiebp]

Thanks for raising this idea! We have plans for this on our near term road map (however with a different implementation approach) 👍🏾

[clemra]

Here you go: https://langfuse.com/changelog/2026-05-28-code-evaluators