Evaluation Suites

Evaluation suites turn a prompt into a repeatable regression contract. A suite chooses a prompt, contains test cases, and keeps every run tied to one prompt version and provider.

Suite Workflow

1. Create a suite and select a prompt.
2. Add cases manually, import CSV/JSON, or populate from a dataset.
3. Choose a prompt version and provider.
4. Run the suite.
5. Inspect assertion details, score, cost, latency, artifacts, and history.

Suites can be organized into typed suite projects. Metadata and cases can be edited inline after creation.

Test Case Inputs

A test case can contain template variables:

{"question": "Which planet is known as the Red Planet?"}

It can also contain multi-turn messages:

[
  {"role": "user", "content": "Remember that my preferred unit is Celsius."},
  {"role": "assistant", "content": "Understood."},
  {"role": "user", "content": "Which unit should you use?"}
]

Attach PDF, PNG, JPEG, JSON, CSV, or plain-text files for document-aware cases.

Assertions

Use the visual editor or raw JSON.

[
  {"type": "contains", "value": "Mars"},
  {"type": "not_contains", "value": "Jupiter"},
  {"type": "regex", "value": "(?i)confidence"},
  {"type": "exact_match", "value": "Mars"}
]

contains and not_contains are substring checks. regex validates patterns before persistence and enforces size, matching-work, depth, and wall-clock limits during execution. See Regex Assertions for dashboard, CLI, API, ExUnit, and file-suite examples. exact_match compares the full output without normalization.

Typed field assertions read dot-separated JSON paths:

[
  {"type": "json_field", "field": "answer", "expected": "Mars"},
  {"type": "json_field", "field": "confidence", "expected": 1}
]

Expected strings, numbers, booleans, lists, objects, and null values retain their JSON types.

Deep comparison produces a field-level score:

[
  {
    "type": "json_deep_compare",
    "expected": {
      "answer": "Mars",
      "evidence": {"category": "astronomy"}
    },
    "threshold": 75.0
  }
]

Extra fields in the actual output do not reduce the score. The default threshold is 100%. JSON metrics accept plain JSON or JSON inside a Markdown code fence.

Model-based rubric judges

Use a custom rubric when the evaluation depends on meaning rather than an exact string or JSON shape:

[
  {
    "type": "rubric_judge",
    "rubric": "The answer must be factually correct, direct, and supported by the supplied evidence.",
    "provider_id": "00000000-0000-0000-0000-000000000000",
    "threshold": 85
  }
]

Replace the all-zero placeholder with the ID of a configured provider.

See Rubric Judges for the full contract and Judge Catalog for seven versioned templates.

Suite execution supplies every metric with normalized prompt, output, document, provider, and execution evidence. Every result distinguishes completed, errored, and unavailable evaluators. See Metric Context and Evaluator Execution Details.

Evaluate assertions directly

Use the same metric registry outside a persisted suite:

alias Aludel.Evals.AssertionEvaluator
alias Aludel.Evals.Metric.Context

context =
  Context.new("Mars",
    rendered_input: "Which planet is known as the Red Planet?",
    variables: %{"question" => "Which planet is known as the Red Planet?"}
  )

result =
  AssertionEvaluator.evaluate(context, %{
    "type" => "contains",
    "value" => "Mars"
  })

true = result["passed"]
100.0 = result["score"]

Aludel.Evals.Metric.Registry.types/0 lists the built-in assertion identifiers, while fetch/1 exposes their implementing modules for custom orchestration.

File Imports

CSV requires input, expected, and assertion; notes is optional:

input,expected,assertion,notes
"Which planet is red?","Mars","contains","smoke test"
"Name Earth's moon","Moon","exact_match","basic fact"

JSON uses an array of the same records:

[
  {"input": "Which planet is red?", "expected": "Mars", "assertion": "contains"}
]

Imports support contains, not_contains, regex, and exact_match. Aludel validates the complete file and presents accepted/rejected row counts plus row-level errors before confirmation.

Run History and Retry

Every suite run persists aggregate passed/failed counts, average score, average cost, average latency, exact cost and latency totals, and individual results.

Retry a single result when a transient provider failure or nondeterministic answer needs another attempt. Aludel replaces that result, recalculates aggregates, and records retry count and time without rerunning the other cases.

For variable model output, execute each test case more than once and reduce the attempts with :all, :any, strict :majority, or a minimum pass rate. See Repeated Sampling for examples, result fields, and retry semantics.

Attach an immutable quality policy when a suite should enforce overall or metadata-group pass rates, evaluator scores, total cost, or average latency. Suite runs snapshot the latest policy version and single-case retries preserve that original contract. See Quality Policies.

Exports and Automation

Suite results link to a report workspace that previews and downloads console text, versioned JSON, JUnit XML, or GitHub Actions annotations. Suite pages also retain the full raw JSON export with assertion details, callback metadata, artifacts, and retry information. Aludel.ExUnit can assert inline output, gate an existing persisted run, or execute and persist a suite from application tests. mix aludel.eval executes the same suite path for CI using direct identifiers or a versioned JSON/YAML manifest. See ExUnit Evaluations, File-Based Suites, Exports and CI, and Evaluation Reporters.

Related Pages

• Datasets
• Metric Context
• Evaluator Execution Details
• Rubric Judges
• Judge Catalog
• Repeated Sampling
• Quality Policies
• ExUnit Evaluations
• File-Based Suites
• Evaluation Reporters
• Documents and Storage
• Analytics and Prompt Evolution