Rubric Judges

Rubric judges score meaning, correctness, or policy compliance when deterministic string and JSON assertions are not expressive enough. A judge uses a separately configured Aludel provider and records its score, reasoning, identity, usage, cost, latency, and lifecycle status with the assertion result.

Interface	Support
Dashboard	Visually configure built-in or custom judges while creating or editing suite test cases
Mix CLI	Run persisted rubric assertions with mix aludel.eval
ExUnit	Evaluate inline assertions or gate a persisted suite with Aludel.ExUnit
Elixir API and files	Use the same assertion map in API calls and JSON/YAML suite manifests

Configure a Judge in the Dashboard

1. Create a suite or edit one of its test cases.
2. Add an assertion and choose rubric judge.
3. Choose Built-in judge and a versioned template, or choose Custom rubric and enter up to 4,000 characters of criteria.
4. Select the configured provider that should act as judge.
5. Optionally change the 0–100 pass threshold, add a reference answer, and add grounding context.
6. Save the suite or test case.

The threshold defaults to 80. The judge provider is separate from the provider whose output the suite evaluates. Reference answers and grounding context become untrusted evaluation evidence; they do not replace the rubric.

Add a Custom Rubric

The visual controls persist the following assertion map. You can also author it directly in the JSON assertion editor, a file-based suite, or the Elixir API:

[
  {
    "type": "rubric_judge",
    "rubric": "The answer must be factually correct, directly answer the question, and avoid unsupported claims.",
    "provider_id": "00000000-0000-0000-0000-000000000000",
    "threshold": 80,
    "expected": "Optional reference answer",
    "context": "Optional grounding evidence"
  }
]

Replace the all-zero placeholder with the ID of a configured provider. threshold accepts 0 through 100 and defaults to 80. expected and context are optional evidence fields.

Custom rubric text must be non-blank and no longer than 4,000 characters. A rubric and a built-in template cannot appear in the same assertion.

Evaluate Programmatically

alias Aludel.Evals.AssertionEvaluator
alias Aludel.Evals.Metric.Context

context =
  Context.new("Paris is the capital of France.",
    rendered_input: "What is the capital of France?",
    expected: "Paris",
    metadata: %{"category" => "geography"}
  )

result =
  AssertionEvaluator.evaluate(context, %{
    "type" => "rubric_judge",
    "rubric" => "The answer must identify the correct capital and contain no unsupported claims.",
    "provider_id" => judge_provider_id,
    "threshold" => 85
  })

Suite execution constructs the same context automatically.

Result Contract

The judge must return a JSON object with a numeric score from 0 through 100 and non-empty reasoning. Aludel derives passed from the configured threshold; any pass or fail verdict returned by the model is ignored.

{
  "type": "rubric_judge",
  "passed": true,
  "score": 92.0,
  "reason": "The response gives the correct capital without unsupported additions.",
  "metadata": {
    "schema_version": 1,
    "threshold": 85.0,
    "rubric": "The answer must identify the correct capital and contain no unsupported claims.",
    "truncated_fields": []
  },
  "evaluator": {
    "status": "completed",
    "provider": "openai",
    "model": "judge-model",
    "input_tokens": 120,
    "output_tokens": 24,
    "cost_usd": 0.00042,
    "duration_ms": 431
  }
}

Malformed judge output, request failures, and missing providers produce isolated failed results with error or unavailable evaluator status. The suite continues and retains stable error categories.

Evidence and Safety Boundaries

Aludel sends the rubric as the only evaluation criteria. Rendered input, generated output, expected answer, context, messages, documents, and metadata are encoded as untrusted JSON evidence and cannot replace the output contract.

Each evidence field is bounded to 50,000 characters. Judge reasoning is bounded to 4,000 characters. Judge requests use temperature 0 and a 500-token response limit. Raw malformed responses and provider failure details are not persisted.

Every judge attempt is a separate model request. Review the recorded evaluator cost and token usage when choosing where to use model-based assertions or repeated sampling.

Use a Built-In Template

For common evaluation criteria, replace rubric with a versioned template:

[
  {
    "type": "rubric_judge",
    "template": "faithfulness",
    "provider_id": "00000000-0000-0000-0000-000000000000",
    "threshold": 85,
    "context": "Grounding evidence for the response"
  }
]

See Judge Catalog for all templates and examples.

Related Pages

• Judge Catalog
• Metric Context
• Evaluator Execution Details
• Repeated Sampling