Evaluation Reporters

Aludel normalizes every completed suite run before rendering it. Local tools, CI systems, and custom integrations therefore see the same status, identifiers, aggregate metrics, quality-policy evidence, and case results without depending on persistence details.

Dashboard Workflow

Open a suite result and choose Reports. Select console text, JSON v2, JUnit XML, or GitHub annotations to update the preview, then download the complete rendered report. The preview is limited to 100,000 characters; downloads are complete and use no-store response headers.

Console and GitHub formats omit generated responses. JSON includes them in its stable result schema. JUnit excludes them by default and adds them only after Include generated output is explicitly enabled. The report page also links to the separate full suite-run export when persisted metadata, artifacts, retry details, and the policy definition are needed.

Built-In Reporters

Reporter	Use
console	Concise local and CI logs without ANSI control sequences
json	Schema-version-2 interchange for scripts and APIs
junit	CI test-report viewers and test artifact ingestion
github	GitHub Actions error and notice annotations

All four accept either a persisted Aludel.Evals.SuiteRun or an existing Aludel.Evals.Report:

alias Aludel.Evals.Reporter

console = Reporter.render!(suite_run, :console)
json = Reporter.render!(suite_run, :json, pretty: true)
junit_xml = Reporter.render!(suite_run, :junit)
github_annotations = Reporter.render!(suite_run, :github)

The non-bang render/3 returns {:ok, output} or {:error, reason}. Reporter output is normalized to a binary even when a custom reporter returns iodata.

Console Example

Aludel evaluation FAILED
Suite run: 017f...
Summary: 2 passed, 1 failed, 66.67% pass rate
Averages: score=86.7, cost=0.0041 USD, latency=428 ms
[PASS] test case 018a...
[FAIL] test case 018b...
Policy: failed
  [FAILED] priority: Pass rate was below the configured minimum

Console output omits generated responses and removes control characters from displayed identifiers and policy evidence.

JSON Example

{
  "type": "aludel_eval",
  "schema_version": 2,
  "status": "passed",
  "suite_run_id": "RUN_ID",
  "suite_id": "SUITE_ID",
  "prompt_version_id": "PROMPT_VERSION_ID",
  "provider_id": "PROVIDER_ID",
  "quality_policy": null,
  "summary": {
    "passed": 2,
    "failed": 0,
    "total": 2,
    "pass_rate": 100.0,
    "avg_score": "100.0",
    "avg_cost_usd": "0.0025",
    "avg_latency_ms": 320,
    "total_cost_usd": "0.005",
    "cost_sample_count": 2,
    "total_latency_ms": 640,
    "latency_sample_count": 2
  },
  "results": []
}

Decimal values remain strings so consumers do not silently lose precision. JSON object key order is not part of the contract.

JUnit XML Example

Write JUnit output directly to the artifact path expected by your CI platform:

mix aludel.eval \
  --suite-id SUITE_ID \
  --prompt-version-id PROMPT_VERSION_ID \
  --provider-id PROVIDER_ID \
  --format junit \
  --output aludel-junit.xml

Each evaluation case becomes a <testcase>, and a failed case contains its failed assertion reasons in <failure>. Generated responses are omitted by default. When all cases pass but the active quality policy does not, Aludel adds a failed quality-policy case. An empty run becomes a failed evaluation case. These synthetic cases keep the CI test view from showing a false success.

Identifiers and evaluator reasons are XML-escaped. Characters forbidden by XML 1.0 are removed. Add --include-output only when the destination artifact is appropriate for generated responses; enabled output is escaped inside <system-out>. Library callers use include_output: true.

GitHub Actions Example

- name: Run Aludel evaluation gate
  run: >-
    mix aludel.eval
    --suite-id "$ALUDEL_SUITE_ID"
    --prompt-version-id "$ALUDEL_PROMPT_VERSION_ID"
    --provider-id "$ALUDEL_PROVIDER_ID"
    --format github

Failed cases and non-passing policy rules become error annotations. A passing run emits one notice. Titles and messages escape workflow-command control characters, and messages are bounded before rendering.

Pretty JSON and File Output

JSON is the CLI default. Add --pretty for readable output:

mix aludel.eval \
  --suite-id SUITE_ID \
  --prompt-version-id PROMPT_VERSION_ID \
  --provider-id PROVIDER_ID \
  --pretty

--output PATH works with every reporter. The task writes the report before returning a nonzero exit for a failed, unavailable, or invalid quality result. Invalid arguments, missing records, and execution failures remain machine-readable JSON error envelopes.

Custom Reporter Example

Implement the reporter behavior when a downstream system needs another format:

defmodule MyApp.MarkdownReporter do
  @behaviour Aludel.Evals.Reporter

  alias Aludel.Evals.Report

  @impl true
  def render(%Report{} = report, _options) do
    {:ok, ["# Evaluation ", String.upcase(report.status), "\n"]}
  end
end

markdown =
  Aludel.Evals.Reporter.render!(
    suite_run,
    MyApp.MarkdownReporter
  )

Custom reporters should treat identifiers, metadata, generated output, and evaluator reasons as untrusted data and escape them for the target format.

Related Pages

• Exports and CI
• Evaluation Suites
• Quality Policies