ExUnit Evaluations

Use Aludel.ExUnit to bring Aludel metrics and persisted quality gates into ordinary application tests. The helpers return normalized results on success and raise ExUnit.AssertionError with concise evidence on failure.

Import the Helpers

Add use Aludel.ExUnit after the application test case:

defmodule MyApp.AnswerTest do
  use ExUnit.Case
  use Aludel.ExUnit
end

Every helper can also be called through Aludel.ExUnit without importing it.

Assert One Response

assert_evaluation/2 runs one Aludel assertion against generated output:

test "returns the canonical city" do
  output = MyApp.answer("What is the capital of France?")

  result =
    assert_evaluation(output, %{
      "type" => "exact_match",
      "value" => "Paris"
    })

  assert result["score"] == 100.0
end

Use any built-in metric: contains, not_contains, resource-bounded regex, exact_match, json_field, json_deep_compare, or rubric_judge. The normalized result remains available for additional application-specific checks. See Regex Assertions for limits and examples across every interface.

Assert Several Metrics

assert_evaluations/2 applies each assertion to the same response and returns results in the same order:

test "returns safe account guidance" do
  output = MyApp.Support.answer("How do I reset my password?")

  results =
    assert_evaluations(output, [
      %{"type" => "contains", "value" => "reset link"},
      %{"type" => "not_contains", "value" => "share your password"},
      %{"type" => "regex", "value" => "(?i)expires? in 15 minutes"}
    ])

  assert length(results) == 3
end

Every assertion runs before the helper fails, so one ExUnit error can identify several non-passing metrics. An empty assertion list fails rather than passing without evidence.

Pass Context to Metrics

Use Aludel.Evals.Metric.Context when an assertion needs the rendered input, expected answer, documents, messages, metadata, provider, prompt version, or execution details:

alias Aludel.Evals.Metric.Context

context =
  Context.new(output,
    expected: "Paris",
    rendered_input: "What is the capital of France?",
    metadata: %{"category" => "geography"}
  )

assert_evaluation(context, %{
  "type" => "contains",
  "value" => "Paris"
})

Model-based rubric assertions use the configured judge provider and retain their evaluator status, model, usage, cost, duration, score, and reasoning in the returned result.

Gate an Existing Persisted Run

assert_suite_run/1 evaluates the effective status of a stored suite result:

suite_run = Aludel.Evals.get_suite_run!(run_id)

assert_suite_run(suite_run)

An immutable stored quality-policy result is authoritative when present. Its status can be passed, failed, invalid, or unavailable. Without a policy, every case must pass and the run must contain at least one case. The original suite-run struct is returned unchanged on success.

Execute, Persist, and Gate

assert_suite/3 runs with default sampling. assert_suite/4 accepts sampling options. Both run a suite through its selected prompt version and provider, persist the complete result, and then apply the same gate:

suite = Aludel.Evals.get_suite!(suite_id)
prompt_version = Aludel.Prompts.get_prompt_version!(prompt_version_id)
provider = Aludel.Providers.get_provider!(provider_id)

suite_run =
  assert_suite(suite, prompt_version, provider,
    samples: 5,
    reducer: :majority
  )

assert suite_run.passed >= 1

A completed failed run remains persisted before the ExUnit assertion is raised. This keeps its outputs, evaluator evidence, cost, latency, sampling attempts, and policy decision available in the workbench. The prompt version must belong to the suite's prompt. Invalid pre-execution configuration does not create a run.

Understand Failures

An inline failure identifies its metric, score, evaluator status, and reason. A suite failure includes aggregate counts, failed case IDs, and non-passing quality-policy rules. Diagnostic text is bounded and control characters are normalized.

Generated model output and expected values are omitted from failure messages. Inspect the returned or persisted result explicitly only when a trusted local workflow needs complete output.

Configure Database and Providers

Inline deterministic metrics need neither a database nor a running Aludel application. Rubric judges require their configured judge provider. The assert_suite helpers use both the configured Aludel repository and provider execution boundary.

Use the host application's normal Ecto SQL sandbox ownership for tests that load or execute persisted suites. Replace provider access only through its documented application boundary, avoid global mocks in asynchronous tests, and set an ExUnit timeout that covers the suite's bounded provider calls and sampling count.

Related Pages

• Evaluation Suites
• Metric Context
• Rubric Judges
• Repeated Sampling
• Quality Policies
• Evaluation Reporters
• Exports and CI