FEAT Ship a reusable LlamaGuard parser and system prompt for CallableResponseHandler

[immu4989]

The scorer composition series (#2125, #2146, #2157) landed CallableResponseHandler as the escape hatch for classifier targets whose output is not PyRIT's default JSON shape, and its docstring names LlamaGuard's safe / unsafe\nS1,S2 format as the motivating case. The pilot in tests/unit/score/test_llamaguard_pilot.py proves the pattern end to end.

The pilot is deliberately a demonstration: its docstring notes the system prompt is "a compact stand-in" because "the real prompt is long and static," and its parser handles only the two happy-path verdicts. Nothing reusable ships with it. A user who wants LlamaGuard scoring today has to reconstruct two things by hand: the full canonical system prompt with the MLCommons S1 through S14 taxonomy, and a parser robust enough for real classifier output.

This proposes shipping those two pieces as bundled, tested helpers so LlamaGuard scoring works out of the box on the composition API, with no new scorer class.

Proposed

1. A parse_llamaguard_response helper in pyrit/score/, public and tested. Beyond the pilot stub it handles case-insensitive verdicts, surrounding whitespace, a missing or empty category line, and a non-verdict response (raising InvalidJsonException so the existing retry applies). It returns the same score_value / rationale / metadata dict the handler already consumes, so it drops straight into CallableResponseHandler(parser=parse_llamaguard_response).
2. The canonical Llama-Guard-3 system prompt as a bundled YAML asset under the scorer datasets, so users do not hand-paste it. A short header notes it can be overridden for other LlamaGuard versions.
3. A short usage example in the scorer docs showing the SelfAskTrueFalseScorer plus CallableResponseHandler wiring, mirroring the pilot.

No new scorer subclass. This stays entirely within the composition model the series established.

Why it is worth having in-tree

LlamaGuard is one of the most widely used open-weight safety classifiers, and the handler was built with it as the reference case. Shipping the parser and prompt turns "here is the hook, write your own adapter" into "point a target at a LlamaGuard endpoint and score." It also gives PyRIT a template that ShieldGemma and WildGuard can follow, each as a parser plus a prompt on the same handler, without further core changes.

Open questions

1. Do you want a bundled classifier parser and prompt in-tree at all, or do you consider that adapter layer user territory now that the handler exists? The pilot living only in tests left this ambiguous, so worth settling before I write anything.
2. If in-tree, where should the parser live: a small pyrit/score/llamaguard.py helper module, or somewhere you would prefer?
3. Scope: bundle only Llama-Guard-3 now, or set up the parser and prompt layout so ShieldGemma and WildGuard slot in as follow-ups?

Happy to implement once you confirm the direction, especially question 1.