Metric Context

Metric context gives every evaluation metric the evidence needed to judge more than a generated string. Suite execution builds this context automatically, and direct callers can construct the same stable contract with Aludel.Evals.Metric.Context.

Evaluate With Context

alias Aludel.Evals.AssertionEvaluator
alias Aludel.Evals.Metric.Context

context =
  Context.new("Paris",
    expected: "Paris",
    rendered_input: "What is the capital of France?",
    prompt_template: "What is the capital of {{country}}?",
    variables: %{"country" => "France"},
    metadata: %{"category" => "geography"}
  )

result =
  AssertionEvaluator.evaluate(context, %{
    "type" => "contains",
    "value" => "Paris"
  })

result["passed"]
# => true

Metrics that only need generated text remain compatible with the string form:

Aludel.Evals.AssertionEvaluator.evaluate(
  "Paris",
  %{"type" => "exact_match", "value" => "Paris"}
)

Available Evidence

Field	Meaning	Default
output	Generated text being evaluated	Required
expected	Reference answer or structured expected value	nil
rendered_input	Final input after template rendering	nil
prompt_template	Versioned source template	nil
variables	Template variables	%{}
messages	Ordered multi-turn messages	[]
documents	Normalized attached-document evidence	[]
metadata	Test case and dataset metadata	%{}
provider	Provider identity available to the run	nil
prompt_version	Prompt version identity and number	nil
execution	Normalized execution details and artifacts	%{}

Optional fields use JSON-compatible values so results can be persisted, exported, and replayed consistently without giving metrics direct ownership of Ecto schemas.

Suite runs populate output, rendered input, prompt template, variables, messages, documents, metadata, provider, prompt version, and execution fields. Expected references normally remain in each assertion's configuration; direct callers can use the context's expected field when they need a shared reference value.

Use Context in a Rubric Judge

Model-based rubric judges can use rendered input, expected output, conversation messages, documents, and metadata:

context =
  Context.new("The policy allows returns within 30 days.",
    rendered_input: "Can I return this after 20 days?",
    expected: "Yes, returns are allowed within 30 days.",
    documents: [%{"name" => "returns-policy.txt", "content_type" => "text/plain"}],
    metadata: %{"policy_version" => 3}
  )

Aludel.Evals.AssertionEvaluator.evaluate(context, %{
  "type" => "rubric_judge",
  "template" => "faithfulness",
  "provider_id" => judge_provider_id,
  "threshold" => 85
})

The judge treats evidence as bounded, untrusted data. Assertions control the rubric and pass threshold.

Related Pages

• Evaluator Execution Details
• Rubric Judges
• Evaluation Suites