Repeated Sampling

Repeated sampling makes suite decisions more reliable when model output varies between otherwise identical requests. Aludel executes each test case a bounded number of times, preserves every ordered attempt, and reduces those attempts into one test case result.

Interface	Configure sampling	Inspect results
Dashboard	Suite run panel	Expandable aggregate and attempt history
Mix CLI	JSON or YAML suite manifest	Selected reporter or exported report
ExUnit	Suite assertion options	Failure output and persisted suite run
Elixir API	execute_suite/4 options	SuiteRun.results

Run a Sampled Suite

In the dashboard, open a suite and configure the run panel:

1. Set Attempts per test case from 1 through 20.
2. Choose whether all attempts, any attempt, or a strict majority must pass, or select a minimum pass rate.
3. For a minimum pass rate, enter a percentage from 0 through 100.
4. Run the suite. Expand the sampling summary on a test case result to inspect every ordered attempt.

The dashboard shows the request multiplier before execution. Each sample is one evaluated-provider request per test case, before any model-based assertion requests.

For the Elixir API, pass :samples and :reducer to Aludel.Evals.execute_suite/4:

{:ok, suite_run} =
  Aludel.Evals.execute_suite(suite, prompt_version, provider,
    samples: 5,
    reducer: :majority
  )

samples accepts integers from 1 through 20. The default is one sample, which preserves the standard single-attempt result shape.

The dashboard's percentage is represented as a decimal rate in the API. For example, 80% becomes {:minimum_pass_rate, 0.8}.

Sampling options also work through the supervised suite runner:

{:ok, suite_run} =
  Aludel.Evals.SuiteRunner.execute(suite.id, prompt_version.id, provider.id,
    samples: 5,
    reducer: {:minimum_pass_rate, 0.8}
  )

Invalid options return {:error, {:invalid_sampling, message}} before a suite run is persisted or a model request is made.

For Mix CLI execution, store the same settings in a versioned JSON or YAML suite manifest:

schema_version: 1
suite_id: SUITE_ID
prompt_version_id: PROMPT_VERSION_ID
provider_id: PROVIDER_ID
sampling:
  samples: 5
  reducer: minimum_pass_rate
  minimum_pass_rate: 0.8

mix aludel.eval --file evals/support-answer.yaml

Aludel.ExUnit accepts the Elixir keyword options when gating a persisted suite.

Choose a Reducer

The reducer determines how attempt outcomes become the final pass or fail result.

Reducer	Passes when	Example
:all	Every attempt passes	samples: 3, reducer: :all
:any	At least one attempt passes	samples: 3, reducer: :any
:majority	More than half of the attempts pass	samples: 5, reducer: :majority
{:minimum_pass_rate, rate}	The pass rate meets or exceeds rate	samples: 10, reducer: {:minimum_pass_rate, 0.8}

:majority is strict. Two passes out of four attempts do not pass.

Use :all for invariants that must never fail, :majority for a stable default across variable outputs, and a minimum pass rate when a quality policy has an explicit tolerance.

Inspect the Result

A sampled test case result retains its normal top-level output and assertion details while adding the complete attempt history and reduction evidence:

The dashboard summarizes passed and failed attempts, the reducer, and the final pass rate. Expand that summary to see each attempt's pass state, score, and output. Exports, reporters, and API consumers receive the result map below:

{
  "passed": true,
  "sampling": {
    "schema_version": 1,
    "samples": 3,
    "reducer": "majority",
    "passed_attempts": 2,
    "failed_attempts": 1,
    "pass_rate": 0.6667,
    "representative_attempt": 3
  },
  "attempts": [
    {"attempt": 1, "passed": false},
    {"attempt": 2, "passed": true},
    {"attempt": 3, "passed": true}
  ]
}

The representative attempt is the latest attempt whose pass or fail state matches the reduced outcome. Its output and artifacts remain available at the top level for existing result consumers.

Aludel averages available attempt scores. It sums input tokens, output tokens, cost, and latency across all attempts so suite aggregates reflect the complete work performed.

Retry a Sampled Result

Retrying one sampled test case restores its persisted sampling configuration and executes the complete attempt set again. Aludel replaces the previous aggregate result and recalculates suite totals without counting the previous attempts twice.

Results created before repeated sampling retry as a single attempt. A result with an unsupported sampling schema returns an error instead of silently changing its evaluation contract.

Control Cost and Latency

Each sample is a separate model request. A suite with 20 cases and five samples can make up to 100 evaluated-provider requests, plus any model-based assertion requests configured for those attempts.

Start with three or five samples for unstable cases, keep deterministic checks at the default of one, and use the recorded cost and latency totals to choose a sample count that fits the suite's purpose.

Related Pages

• Evaluation Suites
• Quality Policies
• Runs and Execution
• Analytics and Prompt Evolution
• Exports and CI