Judge Catalog

Aludel includes seven versioned rubric templates for common model-based evaluation tasks. Templates provide stable criteria while each assertion chooses its own judge provider and pass threshold.

Configure a Template

In the dashboard, create a suite or edit a test case, add an assertion, choose rubric judge, keep Built-in judge selected, and choose one of the templates below. Select a configured judge provider and optionally set the threshold, reference answer, and grounding context.

The dashboard stores the same assertion map used by the Mix CLI, ExUnit, file-based suites, and the Elixir API:

[
  {
    "type": "rubric_judge",
    "template": "correctness",
    "provider_id": "00000000-0000-0000-0000-000000000000",
    "threshold": 85,
    "expected": "Paris"
  }
]

Replace the all-zero placeholder with the ID of a configured provider. The shorter examples below use the same placeholder.

Aludel resolves the template when the assertion runs and stores its ID, version, and resolved rubric in result metadata. Historical results therefore retain the exact criteria they used even after the catalog evolves.

Templates and Examples

Template	Use it when	Example evidence
correctness	The answer must agree with a reference answer or supplied facts	expected: "Paris" for a capital-city question
relevance	The response must directly address the requested task without tangents	A support question where the response should explain only the requested return step
faithfulness	Every material claim must be grounded in supplied context or documents	context: "Returns are allowed within 30 days"
safety	The response must avoid enabling harmful or dangerous action	A dual-use request where safe redirection is expected
refusal	A refusal must be appropriate, proportionate, and useful	A harmful request that should receive a clear refusal and safe alternative
pii	The response must avoid unauthorized disclosure or inference of personal data	Context containing a fictional placeholder that must not be expanded into real identifiers
hallucination	Claims, citations, entities, tool results, and outcomes must not be fabricated	A grounded answer expected to cite only supplied sources

Correctness

{"type":"rubric_judge","template":"correctness","provider_id":"00000000-0000-0000-0000-000000000000","threshold":90,"expected":"Paris"}

Use an expected answer, context, or documents whenever correctness depends on reference evidence.

Relevance

{"type":"rubric_judge","template":"relevance","provider_id":"00000000-0000-0000-0000-000000000000","threshold":80}

The rendered input provides the task the response should address.

Faithfulness

{"type":"rubric_judge","template":"faithfulness","provider_id":"00000000-0000-0000-0000-000000000000","threshold":90,"context":"Returns are allowed within 30 days."}

Supply grounding context or documents. Without grounding evidence, the template scores conservatively.

Safety

{"type":"rubric_judge","template":"safety","provider_id":"00000000-0000-0000-0000-000000000000","threshold":95}

The template distinguishes benign discussion from actionable harmful assistance using the rendered input and available context.

Refusal Quality

{"type":"rubric_judge","template":"refusal","provider_id":"00000000-0000-0000-0000-000000000000","threshold":85}

The template penalizes unsafe compliance and unnecessary refusal of benign requests.

PII Protection

{"type":"rubric_judge","template":"pii","provider_id":"00000000-0000-0000-0000-000000000000","threshold":95}

Use metadata or context to express what disclosure is authorized. Clearly fictional placeholders are not treated as real personal data.

Hallucination

{"type":"rubric_judge","template":"hallucination","provider_id":"00000000-0000-0000-0000-000000000000","threshold":90,"expected":"Use only the supplied release notes."}

Provide the expected answer, grounding context, or documents needed to verify material claims.

Inspect the Catalog in Elixir

Aludel.Evals.JudgeCatalog.ids()
# => ["correctness", "relevance", "faithfulness", "safety", "refusal", "pii", "hallucination"]

{:ok, template} = Aludel.Evals.JudgeCatalog.fetch("faithfulness")
template.version
# => 1

Unknown IDs return :error during direct lookup and fail suite assertion validation before execution.

Related Pages

• Rubric Judges
• Metric Context
• Evaluation Suites