Datasets

Datasets are reusable, ordered collections of evaluation examples. They separate test inputs from a particular suite so the same corpus can seed multiple experiments.

Dataset Shape

A dataset has a name, optional description, and JSON metadata. Each ordered entry contains:

• name
• variable values
• messages
• assertions
• metadata
• position

An entry must contain variables, messages, or both.

Single-Turn Example

Variables:

{"question": "Which planet is known as the Red Planet?"}

Assertions:

[
  {"type": "json_field", "field": "answer", "expected": "Mars"}
]

Metadata:

{"language": "en", "category": "astronomy", "priority": "smoke"}

Multi-Turn Example

Messages:

[
  {"role": "system", "content": "Answer concisely."},
  {"role": "user", "content": "Remember that my preferred unit is Celsius."},
  {"role": "assistant", "content": "Understood."},
  {"role": "user", "content": "What unit should the forecast use?"}
]

Each message needs a supported role and content. Aludel validates message order and payload shape before saving.

Filter by Metadata

The dataset detail page applies PostgreSQL JSON containment filters. To find English smoke cases, enter:

{"language": "en", "priority": "smoke"}

Only entries whose metadata contains every supplied pair are shown.

Populate a Suite

Open a suite, choose a dataset, and select Add entries. Aludel:

1. locks the suite during population
2. copies dataset entries in position order
3. preserves variables, messages, assertions, and metadata
4. records the source dataset entry on each test case
5. skips entries already imported into that suite

The operation is idempotent for one dataset-entry/suite pair. Editing the copied test case later does not mutate the dataset.

Context API

{:ok, dataset} =
  Aludel.Datasets.create_dataset(%{
    name: "Core regression",
    description: "High-value examples",
    metadata: %{"owner" => "platform"}
  })

{:ok, entry} =
  Aludel.Datasets.create_entry(dataset, %{
    name: "Red planet",
    variable_values: %{"question" => "Which planet is red?"},
    messages: [],
    assertions: [%{"type" => "contains", "value" => "Mars"}],
    metadata: %{"priority" => "smoke"}
  })

Aludel.Datasets.list_entries(dataset, metadata: %{"priority" => "smoke"})
Aludel.Datasets.populate_suite(dataset, suite)

Entries without an explicit position are appended under a dataset lock. Repeating populate_suite/2 returns only newly copied entries because existing source links are skipped.

Invalid or cross-dataset entry IDs return nil instead of raising:

case Aludel.Datasets.get_entry(dataset, entry_id) do
  nil -> :not_found
  entry -> {:ok, entry}
end

Related Pages

• Red-Team Datasets
• Generated Red-Team Cases
• Evaluation Suites
• Data Model
• Demo Data