Generated Red Team Cases

Aludel can use an existing provider to propose product-specific adversarial cases. Generation is bounded, schema-validated, and separate from persistence: candidates are returned for inspection without creating or executing dataset entries.

Interface Availability

Interface	Generated-case workflow
Elixir API	Generate, inspect, explicitly approve, and atomically import candidates
Dashboard	Generate with explicit bounds, inspect the review receipt, approve exact candidates, and atomically import them
Mix CLI	No generation or import command; run a suite populated from imported entries with mix aludel.eval
ExUnit	No generation or import helper; gate a persisted suite with Aludel.ExUnit after API import

Generate Candidates

In the dashboard, open a dataset and choose Generate cases. Select a configured provider and risk categories, enter optional target context, and review every request, case-count, token, cost, and timeout limit before generating. The request runs asynchronously, and no dataset entry is written.

{:ok, generation} =
  Aludel.RedTeam.generate(generator_provider.id,
    categories: [:prompt_injection, :sensitive_information_disclosure],
    target_context: "A support assistant may read account history but must not reveal credentials",
    cases_per_category: 2,
    max_requests: 2,
    max_output_tokens: 1_200,
    max_total_tokens: 8_000,
    max_cost_usd: 1.00,
    request_timeout_ms: 30_000
  )

The default request generates two prompt-injection candidates. Supported categories are returned by Aludel.RedTeam.categories/0.

Generation Limits

Option	Default	Allowed value
categories	[:prompt_injection]	Non-empty unique subset of the catalog categories
cases_per_category	2	1 to 5
target_context	empty	Up to 10,000 characters
max_requests	6	1 to 6
max_output_tokens	1,200	100 to 4,000 per request
max_total_tokens	20,000	At least the per-request limit, up to 100,000
max_cost_usd	5.00	Greater than 0, up to 100
request_timeout_ms	30,000	100 to 120,000 per request

Request count, cases per category, context size, response size, output tokens, and timeout are bounded directly. Total tokens and cost are provider-reported stop thresholds checked before Aludel starts the next category. The final in-flight request can report usage above a threshold, bounded by the context-size and per-request output limits. Failed or timed-out external requests may still incur provider-side usage that Aludel cannot observe because no usage response was returned. Calling generate/2 again starts new requests; categories are not retried automatically.

Review Successes and Failures

Generation runs once per category and returns :completed, :partial_failure, or :failed. A valid category remains reviewable when another category times out, has a provider failure, reaches a budget before starting, or returns invalid structured output.

generation.status
generation.usage
generation.limits
generation.failures

Enum.each(generation.cases, fn candidate ->
  IO.inspect(%{
    id: candidate.id,
    category: candidate.category,
    severity: candidate.severity,
    technique: candidate.technique,
    prompt: candidate.prompt,
    rationale: candidate.rationale,
    recommended_judge: candidate.recommended_judge
  })
end)

Failures contain a category, stable type, and safe message. Raw provider errors and malformed model output are not retained. The raw target context is represented by a checksum rather than copied into later dataset metadata.

Review Boundary

Generation deliberately stops after returning the review record. It does not create, update, delete, or execute dataset entries. Review the prompt, rationale, classification, recommended judge, failures, and recorded usage before approving any candidate.

Each candidate and the complete generation have checksums for stable review. The generation also records its provider and model, schema and prompt versions, timestamp, requested categories, observed requests, tokens and cost, applied limits, and a checksum of the target context.

The dashboard shows those receipt fields alongside safe category failures and each candidate's complete prompt, rationale, classification, recommended judge, ID, and checksum. No candidate is pre-approved. Unimported candidates remain only in the connected page session, so leaving or reconnecting requires a new bounded generation request.

Approve and Import

In the dashboard, check the exact candidates you approve, select the prompt variable, judge provider, and threshold, then choose Import approved cases. The result reports created and already-present entries separately. The generation receipt and every candidate are revalidated before the atomic write.

Pass the stable IDs of at least one explicitly approved candidate:

approved_case_ids =
  generation.cases
  |> Enum.filter(&approved_by_reviewer?/1)
  |> Enum.map(& &1.id)

{:ok, %{created: created, skipped: skipped}} =
  Aludel.RedTeam.import_generated(dataset, generation,
    approved_case_ids: approved_case_ids,
    variable: "input",
    judge_provider_id: generator_provider.id,
    judge_threshold: 80
  )

The judge provider defaults to the provider used for generation. Each imported candidate receives its recommended built-in rubric judge. The threshold defaults to 80 and accepts values from 0 through 100.

Before locking the dataset, Aludel revalidates the generation checksum, outcome accounting, and every candidate checksum. Approved IDs must be non-empty, unique, and present in that generation. Imported metadata records the generation status and failures, provider and model, observed usage, limits, target-context checksum, candidate rationale and classification, explicit review evidence, and import checksums. It does not copy the raw target context.

The complete approved selection is written atomically in generation order. Repeating the exact import returns its entries in skipped. Changed content, provenance, review data, variable, or judge configuration under the same deduplication key causes a conflict and rolls back every new entry from that call.

Imported cases use the normal Datasets, Evaluation Suites, Judge Catalog, Quality Policies, and Exports and CI workflows.