Evaluator Execution Details

Every assertion result includes normalized evaluator execution details. This distinguishes an output that failed a completed evaluation from an evaluator that errored or could not run.

Lifecycle Status

Status	Meaning
completed	The evaluator ran and returned a valid result, whether the assertion passed or failed
error	The evaluator started but raised, exited, returned an invalid result, or encountered an infrastructure failure
unavailable	The configured evaluator or judge provider could not be resolved or used

An assertion can have "passed": false with "status": "completed". That is a normal quality result, not an evaluator failure.

Deterministic Metric Example

{
  "type": "contains",
  "passed": true,
  "score": 100.0,
  "reason": "Output contains expected value",
  "evaluator": {
    "status": "completed",
    "duration_ms": 0.042
  }
}

Aludel measures evaluator duration when the metric does not supply its own timing.

Model-Based Evaluator Example

{
  "type": "rubric_judge",
  "passed": true,
  "score": 92.0,
  "reason": "The answer is correct and concise.",
  "evaluator": {
    "status": "completed",
    "duration_ms": 431,
    "provider": "openai",
    "model": "judge-model",
    "input_tokens": 120,
    "output_tokens": 24,
    "cost_usd": 0.00042
  }
}

Judge usage is attributed separately from the provider response being evaluated. This makes it possible to report evaluation overhead without mixing it into system-under-test cost and token totals.

Failure Example

{
  "type": "rubric_judge",
  "passed": false,
  "score": 0.0,
  "reason": "Judge provider is unavailable",
  "evaluator": {
    "status": "unavailable",
    "error": {
      "type": "provider_not_found",
      "message": "Configured judge provider was not found"
    }
  }
}

Error maps use stable categories and safe messages. Aludel does not retain evaluator exception messages, provider response bodies, credentials, or other sensitive failure details.

Unsupported metric types are reported as unavailable with error.type set to unsupported_metric. Exceptions, exits, and invalid return values become isolated error results instead of stopping the suite.

Read the Contract in Elixir

result =
  Aludel.Evals.AssertionEvaluator.evaluate(
    "Paris",
    %{"type" => "contains", "value" => "Paris"}
  )

case result["evaluator"]["status"] do
  "completed" -> result["passed"]
  "error" -> {:error, result["evaluator"]["error"]}
  "unavailable" -> {:unavailable, result["evaluator"]["error"]}
end

Related Pages

• Metric Context
• Rubric Judges
• Repeated Sampling