Generated Red Team Cases

Aludel can use an existing provider to propose product-specific adversarial cases. Generation is bounded, schema-validated, and separate from persistence: candidates are returned for inspection without creating or executing dataset entries.

Interface Availability

Interface	Generated-case workflow
Elixir API	Generate candidates and inspect partial failures, usage, and limits
Dashboard	No generation form; manually authored cases use the normal dataset workflow
Mix CLI	No generation command
ExUnit	No generation helper; generated candidates are inert review values

Generate Candidates

{:ok, generation} =
  Aludel.RedTeam.generate(generator_provider.id,
    categories: [:prompt_injection, :sensitive_information_disclosure],
    target_context: "A support assistant may read account history but must not reveal credentials",
    cases_per_category: 2,
    max_requests: 2,
    max_output_tokens: 1_200,
    max_total_tokens: 8_000,
    max_cost_usd: 1.00,
    request_timeout_ms: 30_000
  )

The default request generates two prompt-injection candidates. Supported categories are returned by Aludel.RedTeam.categories/0.

Generation Limits

Option	Default	Allowed value
categories	[:prompt_injection]	Non-empty unique subset of the catalog categories
cases_per_category	2	1 to 5
target_context	empty	Up to 10,000 characters
max_requests	6	1 to 6
max_output_tokens	1,200	100 to 4,000 per request
max_total_tokens	20,000	At least the per-request limit, up to 100,000
max_cost_usd	5.00	Greater than 0, up to 100
request_timeout_ms	30,000	100 to 120,000 per request

Request count, cases per category, context size, response size, output tokens, and timeout are bounded directly. Total tokens and cost are provider-reported stop thresholds checked before Aludel starts the next category. The final in-flight request can report usage above a threshold, bounded by the context-size and per-request output limits. Failed or timed-out external requests may still incur provider-side usage that Aludel cannot observe because no usage response was returned. Calling generate/2 again starts new requests; categories are not retried automatically.

Review Successes and Failures

Generation runs once per category and returns :completed, :partial_failure, or :failed. A valid category remains reviewable when another category times out, has a provider failure, reaches a budget before starting, or returns invalid structured output.

generation.status
generation.usage
generation.limits
generation.failures

Enum.each(generation.cases, fn candidate ->
  IO.inspect(%{
    id: candidate.id,
    category: candidate.category,
    severity: candidate.severity,
    technique: candidate.technique,
    prompt: candidate.prompt,
    rationale: candidate.rationale,
    recommended_judge: candidate.recommended_judge
  })
end)

Failures contain a category, stable type, and safe message. Raw provider errors and malformed model output are not retained. The raw target context is represented by a checksum rather than copied into later dataset metadata.

Review Boundary

Generation deliberately stops after returning the review record. It does not create, update, delete, or execute dataset entries. Review the prompt, rationale, classification, recommended judge, failures, and recorded usage before manually authoring any evaluation case.

Each candidate and the complete generation have checksums for stable review. The generation also records its provider and model, schema and prompt versions, timestamp, requested categories, observed requests, tokens and cost, applied limits, and a checksum of the target context.

Manually accepted cases can use the normal Datasets, Evaluation Suites, Judge Catalog, Quality Policies, and Exports and CI workflows.