Add MulticlassJudge detector for configurable LLM-as-judge classification

[ABeltramo]

Introduces a new MulticlassJudge detector that extends ModelAsJudge with JSON-aware response parsing and user-defined classification categories (e.g. complied/rejected/alternative/other). Supports configurable system and user prompts, custom score keys/fields, confidence thresholds, and optional JSON schema injection for structured output APIs.

Cherry-picked from trustyai-explainability/garak:automated-red-teaming

[ABeltramo]

From a quick glance it seems that the CI failures are unrelated to this PR.
The only failing test is tests/generators/test_litellm.py::test_litellm_model_detection, which fails with:

  openai.OpenAIError: Missing credentials. Please pass an `api_key` ...

This test has no OPENAI_API_KEY skip guard (unlike the other tests in the same file), so it runs unconditionally whenever litellm is installed. The same test passes on our fork's CI, which ran against a slightly older package version, so I guess this is just a flaky test..

[jmartin-tech]

The test failure is from an environment requirements change in the litellm dependency released yesterday, a PR to address it in main should be up by end of day a propagate it to the feature branch.

[immu4989]

Third in the judge family review pass after #1717 and #1725. Different shape than either: rubric scoring on one side, classifier-token on the other, JSON-mode classification here. A few technical notes.

JSON extraction is brittle. _strip_code_fences finds the first { and last } in the response text. That works for the canonical case ("here is the JSON: { ... }") and for code-fenced output, but it breaks when the judge model wraps the answer with prose that itself contains braces. Real example I've hit: The model's reasoning included compliance markers like { brackets in code blocks }. My classification: {"complied": 80, ...}. Current logic grabs from the brace in the prose through to the closing brace of the JSON, producing invalid input. A regex anchored on the JSON object pattern, or json.loads over a sliding window of candidates, would be sturdier. Adjacent option: most modern OpenAI-compatible APIs (OpenAI, Together, Groq, Fireworks, vLLM) support response_format={"type": "json_object"} as a server-side guarantee. The PR already monkey-patches extra_params["response_format"] for the response_schema case; doing the same for the response_format == "json" default would give a much stronger reliability floor than prompt-only instructions.

__init__ mutates the shared generator instance. When response_schema is set the code writes to self.evaluation_generator.extra_params. If the same configured generator is reused by another detector, or by another instance of MulticlassJudge with a different schema, the side effect leaks. Probably not a frequent scenario in practice (most runs use a single judge config), but worth a note in the docstring or a deepcopy of the generator before the mutation.

confidence_cutoff scale differs silently from ModelAsJudge. The parent uses confidence_cutoff=7 on a 1-10 rating. This subclass uses confidence_cutoff=70 on a 0-100 percentage scale. Both numbers happen to mean "70%" but a user inheriting muscle memory from ModelAsJudge could mis-set it. Worth a single sentence in the class docstring calling out the scale.

Milestone-12 note. Same point I raised on #1725: attempt.notes["multiclass_scores"] is a sensible v1, and after the technique-and-intent work in #1434 lands the canonical detector-output slot is likely to shift. No action needed, just keeping the threads visible.

Things done well. Subclassing ModelAsJudge cleanly, the dual config-param pattern (system_prompt + user_prompt_template) that makes the judge fully customisable, the code-fence stripping (mechanism aside) handling the common JSON-in-markdown case, parameterised test coverage including malformed and missing-key paths, and the explicit opt-out fallback to the existing rating parser. The PR is genuinely close to a useful piece of shared infrastructure.

Looking forward to seeing this and the other two land.

[ABeltramo]

Hi @immu4989 thanks for taking the time to test and review this PR!

All very good points, I should've addressed everything:

• Changed the JSON extraction to use a sliding window that tries all the possible {...} substrings
• Added deepcopy to avoid the Generator mutation potential issue
• Added a docstring about the confidence_cutoff

I'm also really looking forward to technique-and-intent hitting main 😉

[immu4989]

@ABeltramo , Glad the suggestions were useful, and impressed by the turnaround.

On Milestone 12: same here. The notes key plumbing is the right v1, and migration will be cheap once the canonical field lands.

Looking forward to seeing this merge.

[jmartin-tech]

@ABeltramo, apologies it looks like github did not auto-retarget on main and it will need to be reopened.

[immu4989]

@ABeltramo heads up, I got the same note on #1979. In my case a rebase onto main plus a force push was all it took, and GitHub had already picked up the retarget on its own, so no reopen was needed. From the outside yours looks the same: base is showing as main with the same clean two file diff. Might be worth checking before closing and reopening, since that would lose the thread here.

[jmartin-tech]

This seems viable, the embedded prompts seem a bit restrictive and cause issues for detector --plugin_info output.

Can you offer a suggested way to validate this detector? Is there some probe that it would work well with when set to override the primary or extended detectors?

[jmartin-tech]

Consider defaulting these to None and having the __init__ fallback to the constants if the user did not provide an override.

Reasoning, DEFAULT_PARAMS are listed in --plugin_info results as well as added to the plugin cache file, and long text blocks do not format well as this time.

As an alternative, these values could be stored in a data fixture file and the detector could offer configuration of the at file location instead of a hardcoded constant in the module. This would allow a user level override via the XDG_DATA path or accept a specific filename from the configuration.

[ABeltramo]

Right, I see. Defaulted to None and used __init__ as the fallback

[immu4989]

Two things that might help with @jmartin-tech's points, though the validation question is yours to answer.

On the DEFAULT_PARAMS feedback: Jailbreak in main already does what jmartin is describing. It carries "custom_system_prompt": None in DEFAULT_PARAMS and falls back to the _jailbreak_system_prompt constant in __init__ when the user has not set one. Closest precedent in the repo if you go the None default route.

Worth flagging a naming collision though. #1832 (open, unreviewed) is already applying that same None default plus __init__ fallback treatment across the family: it adds system_prompt_judge to ModelAsJudge and system_prompt_on_topic to Refusal, and renames Jailbreak.custom_system_prompt to system_prompt_on_topic. This PR adds system_prompt on a ModelAsJudge subclass. If both land the family ends up with three names for the same concept, which cuts against the consistent contributor facing pattern jmartin is after in #1808. Might be worth the two PRs agreeing a key name before either lands.

On validation, one option if it is useful: your default system prompt already targets dan style dual output, since it enumerates 'DAN', 'developer mode', 'AIM', 'NRAF' and 'Anarchy' and says to evaluate only the jailbreak section. The dan.* probes are the natural pairing, and they currently score with string detectors such as dan.DAN and dan.DevMode. dan.DAN is also one of the six detectors carrying labeled ground truth in garak/data/detectors-eval/detector_metrics_summary.json, at hit_f1 0.889, so pointing MulticlassJudge at the same labeled set would give a like for like comparison rather than a subjective read.

[ABeltramo]

Thanks for the feedback @immu4989 and @jmartin-tech I should've addressed the defaults, and I was looking at the validation part, but I think I'm missing something here..

It looks like only the precomputed summary ships in the repo, I couldn't find the labelled benchmark dataset or the evaluation harness that produces it; can you point me at how detector_metrics_summary.json is generated? I'd be more than happy to get some real stats over the detector performance.

[jmartin-tech]

The pipeline that generates detector_metrics_summary.json is still internal tooling, the team is still working out how we can enable community collaboration with it.

@ABeltramo please edit the commit messages to ensure sign-off is included to meet the DCO requirements.

[ABeltramo]

@jmartin-tech sorry about that, should be fixed now