Repeated Sampling

Repeated sampling makes suite decisions more reliable when model output varies between otherwise identical requests. Aludel executes each test case a bounded number of times, preserves every ordered attempt, and reduces those attempts into one test case result.

Run a Sampled Suite

Pass :samples and :reducer to Aludel.Evals.execute_suite/4:

{:ok, suite_run} =
  Aludel.Evals.execute_suite(suite, prompt_version, provider,
    samples: 5,
    reducer: :majority
  )

samples accepts integers from 1 through 20. The default is one sample, which preserves the standard single-attempt result shape.

Sampling options also work through the supervised suite runner:

{:ok, suite_run} =
  Aludel.Evals.SuiteRunner.execute(suite.id, prompt_version.id, provider.id,
    samples: 5,
    reducer: {:minimum_pass_rate, 0.8}
  )

Invalid options return {:error, {:invalid_sampling, message}} before a suite run is persisted or a model request is made.

Choose a Reducer

The reducer determines how attempt outcomes become the final pass or fail result.

Reducer	Passes when	Example
:all	Every attempt passes	samples: 3, reducer: :all
:any	At least one attempt passes	samples: 3, reducer: :any
:majority	More than half of the attempts pass	samples: 5, reducer: :majority
{:minimum_pass_rate, rate}	The pass rate meets or exceeds rate	samples: 10, reducer: {:minimum_pass_rate, 0.8}

:majority is strict. Two passes out of four attempts do not pass.

Use :all for invariants that must never fail, :majority for a stable default across variable outputs, and a minimum pass rate when a quality policy has an explicit tolerance.

Inspect the Result

A sampled test case result retains its normal top-level output and assertion details while adding the complete attempt history and reduction evidence:

{
  "passed": true,
  "sampling": {
    "schema_version": 1,
    "samples": 3,
    "reducer": "majority",
    "passed_attempts": 2,
    "failed_attempts": 1,
    "pass_rate": 0.6667,
    "representative_attempt": 3
  },
  "attempts": [
    {"attempt": 1, "passed": false},
    {"attempt": 2, "passed": true},
    {"attempt": 3, "passed": true}
  ]
}

The representative attempt is the latest attempt whose pass or fail state matches the reduced outcome. Its output and artifacts remain available at the top level for existing result consumers.

Aludel averages available attempt scores. It sums input tokens, output tokens, cost, and latency across all attempts so suite aggregates reflect the complete work performed.

Retry a Sampled Result

Retrying one sampled test case restores its persisted sampling configuration and executes the complete attempt set again. Aludel replaces the previous aggregate result and recalculates suite totals without counting the previous attempts twice.

Results created before repeated sampling retry as a single attempt. A result with an unsupported sampling schema returns an error instead of silently changing its evaluation contract.

Control Cost and Latency

Each sample is a separate model request. A suite with 20 cases and five samples can make up to 100 evaluated-provider requests, plus any model-based assertion requests configured for those attempts.

Start with three or five samples for unstable cases, keep deterministic checks at the default of one, and use the recorded cost and latency totals to choose a sample count that fits the suite's purpose.

Related Pages

• Evaluation Suites
• Quality Policies
• Runs and Execution
• Analytics and Prompt Evolution
• Exports and CI