Exports and CI

Aludel exposes result data for debugging, audit, external analysis, and CI quality gates.

Run Result JSON

Open a completed run result and choose Export JSON. The download includes:

• run identity, status, prompt/version, variables, and timestamps
• provider identity, type, and model
• result state, output or error, tokens, latency, and cost
• callback metadata
• normalized execution artifacts

The response is marked no-store and downloaded as run-result-<id>.json.

Suite Run JSON

Suite exports include:

• suite, prompt version, and provider identity
• passed, failed, total, average score, cost, and latency
• every output and assertion result
• token, cost, latency, callback metadata, and artifacts
• retry count and retry time
• immutable quality policy definition, version, outcome, and rule evidence when configured

The response is downloaded as suite-run-<id>.json.

Prompt Evolution Export

The Evolution page exports JSON or CSV with version-level metrics and provider breakdowns. Files use timestamps to avoid accidental overwrites.

Use JSON for structured downstream processing and CSV for spreadsheet analysis. CSV values are quoted safely and formula-like text is neutralized before export.

Headless Suite Execution

Run a versioned JSON or YAML suite manifest:

mix aludel.eval --file evals/support-answer.yaml

The manifest selects the persisted suite, prompt version, provider, and optional sampling policy. See File-Based Suites for the schema, every reducer, validation rules, and Elixir API.

You can also run one suite/version/provider combination directly:

mix aludel.eval \
  --suite-id SUITE_ID \
  --prompt-version-id PROMPT_VERSION_ID \
  --provider-id PROVIDER_ID

The task emits one schema-version-2 JSON envelope by default:

{
  "type": "aludel_eval",
  "schema_version": 2,
  "status": "passed",
  "suite_run_id": "SUITE_RUN_ID",
  "suite_id": "SUITE_ID",
  "prompt_version_id": "PROMPT_VERSION_ID",
  "provider_id": "PROVIDER_ID",
  "quality_policy": {
    "policy_version": 3,
    "status": "passed",
    "passed": true,
    "rules": []
  },
  "summary": {
    "passed": 2,
    "failed": 0,
    "total": 2,
    "pass_rate": 100.0,
    "avg_score": "100.0",
    "avg_cost_usd": "0.0012",
    "avg_latency_ms": 420,
    "total_cost_usd": "0.0024",
    "cost_sample_count": 2,
    "total_latency_ms": 840,
    "latency_sample_count": 2
  },
  "results": []
}

When a suite has a quality policy, the top-level status is the policy's passed, failed, invalid, or unavailable outcome. The task exits unsuccessfully unless that status is passed.

Without a quality policy, it exits unsuccessfully when:

• required or valid arguments are missing
• a target does not exist
• the prompt version belongs to another prompt
• suite execution fails
• the suite has no test cases
• any test case fails

See Quality Policies for rule configuration, missing-evidence behavior, and retry semantics.

ExUnit Quality Gates

Use Aludel.ExUnit when the evaluation should run as part of an application test. It supports deterministic and model-based inline metrics, existing persisted suite runs, and execute-and-persist suite gates:

defmodule MyApp.AnswerTest do
  use ExUnit.Case
  use Aludel.ExUnit

  test "keeps the answer grounded" do
    output = MyApp.answer("Where can I change my password?")

    assert_evaluations(output, [
      %{"type" => "contains", "value" => "account settings"},
      %{"type" => "not_contains", "value" => "send your password"}
    ])
  end
end

The suite helpers use the same effective status as the Mix task and reporters. See ExUnit Evaluations for runnable examples and database/provider setup guidance.

Reporter Formats

Use --format to select console, json, junit, or github. Add --output PATH to write a report to a file, --pretty to pretty-print JSON, and JUnit-only --include-output when the CI artifact is appropriate for generated responses.

mix aludel.eval \
  --suite-id SUITE_ID \
  --prompt-version-id PROMPT_VERSION_ID \
  --provider-id PROVIDER_ID \
  --format junit \
  --output aludel-junit.xml

The task writes a completed evaluation report before returning a nonzero exit for a failed, invalid, or unavailable quality decision. Argument and execution errors remain JSON error envelopes.

See Evaluation Reporters for examples of every format and the custom reporter behavior.

CI Example

- name: Run Aludel regression suite
  env:
    MIX_ENV: test
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: |
    mix ecto.migrate
    mix aludel.eval \
      --suite-id "$ALUDEL_SUITE_ID" \
      --prompt-version-id "$ALUDEL_PROMPT_VERSION_ID" \
      --provider-id "$ALUDEL_PROVIDER_ID"

Keep credentials in the CI platform's secret store and use stable IDs from the target environment.

Related Pages

• Evaluation Suites
• Quality Policies
• ExUnit Evaluations
• File-Based Suites
• Evaluation Reporters
• Analytics and Prompt Evolution
• Runs and Execution