Red Team Datasets

Aludel ships a stable, versioned catalog of adversarial inputs that can be materialized into ordinary reusable datasets. The resulting entries use the same dashboard, suite, policy, reporter, CLI, and ExUnit workflows as manually authored cases.

The catalog categories align with the OWASP Top 10 for LLM Applications and the NIST Generative AI Profile. The included cases are regression seeds, not a complete security assessment or compliance certification.

Interface Availability

Interface	Red-team workflow
Elixir API	Browse the catalog and materialize all or selected cases
Dashboard	Inspect and edit materialized entries, populate suites, run suites, and review results
Mix CLI	Run the resulting persisted suite with mix aludel.eval; there is no separate catalog command
ExUnit	Execute and gate the resulting suite with Aludel.ExUnit

Catalog Cases

Case ID	Category	Severity	Technique	Judge
direct-instruction-override	prompt_injection	critical	direct	refusal
indirect-document-override	prompt_injection	critical	indirect	refusal
system-prompt-extraction	system_prompt_leakage	high	direct	refusal
private-record-disclosure	sensitive_information_disclosure	critical	direct	PII
unapproved-destructive-action	excessive_agency	critical	direct	safety
fabricated-citation	misinformation	high	direct	hallucination
concealed-security-disablement	unsafe_assistance	high	direct	safety

Browse without changing the database:

Aludel.RedTeam.all()
Aludel.RedTeam.categories()
Aludel.RedTeam.fetch("direct-instruction-override")

Each returned case includes its prompt, ID and version, category, severity, technique, risk reference, recommended judge, canary assertion, and SHA-256 template checksum.

Materialize the Full Catalog

{:ok, dataset} =
  Aludel.Datasets.create_dataset(%{
    name: "Security regressions",
    description: "Adversarial checks for the support assistant"
  })

{:ok, %{created: entries, skipped: []}} =
  Aludel.RedTeam.materialize(dataset)

The default variable is input, so use {{input}} in the target prompt. Every entry has a deterministic not_contains assertion for a unique canary.

Materialize Selected Categories

{:ok, %{created: injection_entries}} =
  Aludel.RedTeam.materialize(dataset,
    categories: [:prompt_injection, :system_prompt_leakage]
  )

Unknown categories return {:error, {:unknown_categories, values}} without creating entries.

Materialize Individual Cases

{:ok, %{created: [entry]}} =
  Aludel.RedTeam.materialize(dataset,
    case_ids: ["fabricated-citation"],
    variable: "request"
  )

The variable option lets the entry target prompts that use placeholders other than {{input}}. Category and case-ID filters are combined when both are present.

Add the Recommended Judge

{:ok, %{created: entries}} =
  Aludel.RedTeam.materialize(dataset,
    judge_provider_id: judge_provider.id,
    judge_threshold: 90
  )

This retains the deterministic canary and adds the recommended built-in rubric judge for each case. The threshold defaults to 80 and accepts values from 0 through 100.

Provenance and Deduplication

Each entry records these fields beneath metadata["red_team"]:

• catalog and catalog version
• case ID and case version
• category, severity, and technique
• risk reference and recommended judge
• source provenance
• template and materialization checksums
• stable deduplication key

Materialization is transactional and locks the dataset so entry positions remain ordered. Repeating identical input returns the existing entries in skipped:

{:ok, %{created: created, skipped: []}} = Aludel.RedTeam.materialize(dataset)
{:ok, %{created: [], skipped: skipped}} = Aludel.RedTeam.materialize(dataset)

The key identifies the catalog case version and destination variable. If the same key already has different content or judge configuration, Aludel returns {:error, {:deduplication_conflict, key}} instead of replacing it. Materialization never updates or deletes existing entries.

Run the Cases

Populate a suite from the dashboard or API:

{:ok, imported_cases} = Aludel.Datasets.populate_suite(dataset, suite)

Then run the suite from the dashboard, mix aludel.eval, Aludel.ExUnit, or Aludel.Evals.execute_suite/4. Policies and reporters work without red-team-specific configuration.

Related Pages

• Datasets
• Evaluation Suites
• Judge Catalog
• Quality Policies
• Exports and CI