File Based Suites

Versioned JSON and YAML manifests let a repository, script, or CI job select a persisted Aludel suite without maintaining a second copy of its test cases.

The database remains authoritative for cases, documents, dataset provenance, and quality policies. A manifest contains only target identifiers and optional repeated-sampling settings.

YAML Example

Create evals/support-answer.yaml:

schema_version: 1
suite_id: 9a756a58-eaec-43ca-99e6-f5c016d85d0c
prompt_version_id: e74cf2e1-94b6-4bcb-9ed9-b259661be906
provider_id: e1c60ec0-6d55-419b-b958-7d088055254f
sampling:
  samples: 5
  reducer: majority

Replace the example UUIDs with IDs from the target Aludel installation, then run:

mix aludel.eval --file evals/support-answer.yaml

.yaml and .yml extensions are supported.

JSON Example

JSON uses the same schema:

{
  "schema_version": 1,
  "suite_id": "9a756a58-eaec-43ca-99e6-f5c016d85d0c",
  "prompt_version_id": "e74cf2e1-94b6-4bcb-9ed9-b259661be906",
  "provider_id": "e1c60ec0-6d55-419b-b958-7d088055254f",
  "sampling": {
    "samples": 5,
    "reducer": "minimum_pass_rate",
    "minimum_pass_rate": 0.8
  }
}

mix aludel.eval --file evals/support-answer.json

Every Sampling Reducer

Omit sampling to run every case once with the all reducer.

Require every attempt:

sampling:
  samples: 3
  reducer: all

Accept any successful attempt:

sampling:
  samples: 3
  reducer: any

Require a strict majority:

sampling:
  samples: 5
  reducer: majority

Set an exact minimum pass rate:

sampling:
  samples: 5
  reducer: minimum_pass_rate
  minimum_pass_rate: 0.8

samples must be between 1 and 20. minimum_pass_rate must be between 0.0 and 1.0 and is valid only with its matching reducer.

Reports and CI

Keep report destinations outside the manifest so one committed suite definition can serve several environments:

mix aludel.eval \
  --file evals/support-answer.yaml \
  --format junit \
  --output aludel-junit.xml

- name: Run Aludel evaluation gate
  run: mix aludel.eval --file evals/support-answer.yaml --format github

Reporter options work the same for manifest and ID-based execution. Supported formats are console, json, junit, and github. JSON also supports --pretty; JUnit supports --include-output when the artifact is appropriate for generated responses.

The command exits unsuccessfully for an invalid manifest or target, execution failure, an empty suite, or a non-passing effective quality decision. A suite's latest immutable quality policy is snapshotted at execution time. Without one, all persisted cases must pass.

Elixir API

Load and inspect a manifest before execution:

alias Aludel.Evals.FileSuite

with {:ok, file_suite} <- FileSuite.load("evals/support-answer.yaml"),
     {:ok, suite_run} <- FileSuite.execute(file_suite) do
  {:ok, suite_run}
end

Or use the one-step API:

{:ok, suite_run} =
  Aludel.Evals.FileSuite.load_and_execute("evals/support-answer.json")

load_string/2 accepts in-memory content with :json or :yaml. All loader errors use stable code and message fields.

Validation and Safety

Validation completes before target lookup or provider execution. Aludel rejects:

• files larger than 256 KiB and invalid UTF-8
• unsupported extensions and schema versions
• malformed input, duplicate keys, YAML aliases, explicit tags, and multiple YAML documents
• missing, unknown, or incorrectly typed fields
• invalid UUIDs, reducers, sample counts, and thresholds
• a prompt version owned by another prompt

The manifest never accepts provider credentials, report output paths, inline cases, or quality-policy definitions. Keep credentials in normal runtime configuration or a CI secret store.

UI and File Ownership

Use the dashboard to author prompts, populate suites from datasets, edit cases and assertions, attach documents, configure providers and policies, and inspect completed runs. Commit a manifest only after those persisted records exist in the environment where it will run.

Running a manifest does not import, update, replace, or delete suite data. Changes made later in the dashboard are used by the next manifest execution because the referenced persisted suite remains the source of truth.

Related Pages

• Evaluation Suites
• Repeated Sampling
• Quality Policies
• Evaluation Reporters
• Exports and CI
• Datasets