refactor(detectors): switchable system prompts via DEFAULT_PARAMS

[v0ropaev]

Promote system_prompt_judge / system_prompt_on_topic from undocumented hasattr-based overrides to first-class DEFAULT_PARAMS entries on ModelAsJudge, Refusal, and Jailbreak. Rename Jailbreak.custom_system_prompt to system_prompt_on_topic for cross-class consistency.

Background: @jmartin-tech noted in #1275 that the judge detector "currently has undocumented support for an override of the judge system prompt". This PR documents that contract and aligns key names across the three sibling detectors.

Changes

• New DEFAULT_PARAMS entries system_prompt_judge (on ModelAsJudge) and system_prompt_on_topic (on Refusal and Jailbreak), both defaulting to None (use built-in prompt).
• Jailbreak.custom_system_prompt removed in favour of system_prompt_on_topic (per AGENTS.md "no backwards-compat shims"). Happy to swap to an alias if preferred.
• {goal} substitution uses str.replace instead of .format, so user-supplied prompts can contain unrelated curly braces without raising KeyError.
• ModelAsJudge.detect bypasses judge_score() and passes the resolved system prompt explicitly via _create_conv(..., system_prompt=...). The public self.system_prompt_judge attribute is no longer mutated after the first invocation. As a side effect this also addresses a latent issue where a single detector instance reused across probes via the harness would freeze on the first probe's goal.
• Class docstrings updated to describe the override mechanism and the [[N]] / [[YES]] / [[NO]] output-format contract. Custom prompts that drop these markers silently default to a maximum (positive-hit) score.
• The language-override note from RefusalOnlyAdversarial is now part of the Refusal class docstring, since the override mechanism it pointed at is now first-class.

Out of scope

• The same attribute names are used in garak/resources/tap/tap_main.py (set via factory functions on EvaluationJudge) and the parallel pattern in garak/detectors/agent_breaker.py (independent reimplementation, does not inherit). Both kept as-is to keep this PR cohesive.
• garak/probes/goat.py exposes its own custom_system_prompt on GOATAttack. That is unrelated to the renamed Jailbreak detector key and is not touched.

Verification

Run from repo root in a clean dev env (pip install -e '.[tests,lint]'):

• black --config pyproject.toml garak/detectors/judge.py tests/detectors/test_detectors_judge.py
• pytest tests/detectors/test_detectors_judge.py -v — 28 passed (19 pre-existing + 9 new)
• pytest tests/resources/red_team/test_evaluation.py -v — 11 passed
• pytest tests/probes/test_probes_goat.py tests/probes/test_probes_fitd.py — 35 passed
• pytest tests/test_docs.py — 661 passed (docstring/ReST validation)

Notes

This change was developed with AI assistance (Claude Code) and reviewed end-to-end by the human submitter; every changed line was inspected before submission. I confirmed there are no other open PRs addressing #1275 before starting work.

Closes #1275

[jmartin-tech]

Part of why this is undocumented is a need to enforce a specific format on prompts used during evaluation. I would suggest while this exposes the option it needs to go further and be able to both inspect the provide prompt for conformance to the required template and possibly add early validation of that conformance.

At this time detectors are not instantiated until after inference for a probe is complete, resulting in a high likelihood that a configuration that attempts to inject this value will not detect an incorrect format until after a possibly large amount of execution is spent in the run and the either the detector will need to be skipped or the run may even crash during the detection phase.

[v0ropaev]

@jmartin-tech — fair point on the timing. Validating in __init__ still costs the probe run before anything trips, at least for every probe after the first in a multi-probe session.

Worth flagging that the silent-failure mode is asymmetric, which changes how much each option matters. process_output_on_topic_score returns 1.0 directly when the [[YES]]/[[NO]] token is missing, so a malformed system_prompt_on_topic poisons a Refusal/Jailbreak run with silent positive hits across every output. process_output_judge_score also defaults to 1.0, but ModelAsJudge.detect then compares against confidence_cutoff (default 7), so a malformed system_prompt_judge produces silent misses instead — still bad, but less of an "every result is wrong in the dangerous direction" failure. So the on_topic prompt is the one that really wants early validation.

On placement, three options as I see them:

(A) check inside the detector __init__ — smallest diff, still after probe inference but at least raises at the start of the detection phase rather than silently corrupting scores;

(B) check at config-load time in configurable.py/_config.py — earliest realistic point, but reaches outside garak/detectors/judge.py and arguably breaks the cohesion of this PR;

(C) a pre-flight plugin-validation pass invoked by the harness before any probe runs — cleanest separation, but it's new infrastructure and clearly its own PR.

For the check itself I'd lean on the parser's own regex (re.search(r"\[\[(\d+)\]\]", ...) and re.search(r"\[\[(yes|no)\]\]", prompt.lower())) rather than a literal substring test, so the contract stays honest if the parser ever changes. [[YES]]/[[NO]] are protocol tokens rather than natural-language strings, so requiring them literally should still be fine for the translated-prompt case the old # Note: covered. I'd leave {goal} optional for ModelAsJudge since the substitution path is a tolerant str.replace. Strictness-wise my instinct is ConfigurationError rather than warn-and-continue, given the silent-positive-hit mode on the on_topic side, but can soften that if you'd prefer the lighter touch.

Default proposal, if you've no strong preference: fold (A) into this PR — regex-check each system_prompt_* override in the detector __init__ and raise ConfigurationError on mismatch — and open a follow-up for (C) so a misconfigured judge fails before any inference at all. Sound right, or would you rather see (B)/(C) land here instead?