FEAT: Add PAIRAttack as a TAP alias with PAIR-definitional defaults pinned#1822
Merged
Merged
Conversation
PAIR (Chao et al. 2023, arXiv:2310.08419) is a structural special case of TreeOfAttacksWithPruningAttack (Mehrotra et al. 2023, arXiv:2312.02119): parallel single-branch iterative refinement with no tree expansion and no off-topic pruning. PyRIT already shipped TAP but had no way for users to search/discover/register PAIR, leading to misconfigured "PAIR" runs that were really TAP. This adds PAIRAttack as a thin TreeOfAttacksWithPruningAttack subclass that pins branching_factor=1 and on_topic_checking_enabled=False at the super().__init__() call site and removes them from its own signature, so those PAIR-definitional structural parameters cannot drift. tree_width and tree_depth remain user-tunable with TAP's defaults (3 and 5) for benchmark-cost comparability between PAIR and TAP. The TAP adversarial system prompt YAML is reused unchanged (matching garak's choice and the paper's prompt structure). - New pyrit/executor/attack/multi_turn/pair.py - Exported from pyrit.executor.attack[.multi_turn] - Registered as "pair" in SCENARIO_TECHNIQUES so AdversarialBenchmark picks it up alongside "tap" - Unit tests covering structural-default pinning and the signature-exclusion contract - Registry test asserting the spec points at PAIRAttack - New doc/code/executor/attack/pair_attack.py jupytext notebook with PAIR<->TAP parameter mapping, wired into doc/myst.yml - 0_attack.md mermaid diagram updated to add the PAIRAttack node Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…drop unreadable docstring table
- Remove doc/code/executor/attack/pair_attack.{py,ipynb}; mention PAIR
as the structural degenerate case of TAP in tap_attack.py instead.
Unregister pair_attack.ipynb from doc/myst.yml.
- Sort __all__ alphabetically in both
pyrit/executor/attack/__init__.py and
pyrit/executor/attack/multi_turn/__init__.py.
- Replace the ASCII grid table in PAIRAttack's class docstring with a
short inline description -- the table didn't render readably in
plaintext / IDE tooltips.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit regenerated the notebook from .py via jupytext, which dropped persisted execution outputs. Use jupytext --update instead so the PAIR-mention markdown edit applies without nuking outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PyRIT uses plain Google-style docstrings with bare class names (per the style guide); drop the :class:\...\ and double-backtick \\...\\ reST conventions I introduced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hannahwestra25
approved these changes
May 28, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PyRIT already ships
TreeOfAttacksWithPruningAttack(TAP, Mehrotra et al. 2023, arXiv:2312.02119), which is a strict generalization of PAIR (Chao et al. 2023, arXiv:2310.08419). PAIR is the parallel-streams, single-branch, no-pruning special case of TAP. Today users searching for "PAIR" find nothing, so anyone trying to use PAIR has to know the TAP-to-PAIR mapping themselves and remember to manually disable branching/pruning, which silently turns a "PAIR" run into a misconfigured TAP run. Garak takes the same modeling stance: itsPAIRprobe is aTAPsubclass with PAIR-specific defaults.This adds
PAIRAttackas a thin subclass ofTreeOfAttacksWithPruningAttackthat hides and hardcodes the two PAIR-definitional structural parameters (branching_factor=1,on_topic_checking_enabled=False) at thesuper().__init__()call site. Removing them from the subclass signature means they cannot drift via kwargs orset_default_value. The remaining knobs (tree_width,tree_depth, scoring/converter/adversarial configs) stay exposed with TAP's defaults (tree_width=3,tree_depth=5) so PAIR and TAP have comparable runtime cost for benchmarking. The TAP adversarial system prompt YAML is reused unchanged -- the behavioural difference between PAIR and TAP is purely structural-search, not prompt text.PAIR-to-TAP mapping:
tree_widthtree_depthbranching_factoron_topic_checking_enabledThe new attack is registered as
"pair"inSCENARIO_TECHNIQUESwith the same["core", "multi_turn"]tags as"tap", soAdversarialBenchmarkpicks it up automatically alongside TAP via the existing_get_benchmarkable_specsfilter.Decisions worth a careful look
on_topic_checking_enabledis hidden + hardcoded toFalseon the same definitional reasoning asbranching_factor. Happy to expose it as a user-tunable kwarg if reviewers prefer.tree_width=3/tree_depth=5preserve TAP's defaults rather than matching the PAIR paper'sN=20, K=3. This trades paper-fidelity for benchmark-cost parity with TAP.Tests and Documentation
Tests (
tests/unit/executor/attack/multi_turn/test_pair.py, new): 10 focused tests covering structural-default pinning, the signature-exclusion contract forbranching_factorandon_topic_checking_enabled(these are absent fromPAIRAttack.__init__so they cannot be set via kwargs orset_default_value), parameter overrides, subclass relationship, sharedTAPAttackContext, and inherited TAP validations (adversarial-target capabilities,FloatScaleThresholdScorerrequirement). A registry test was added totests/unit/registry/test_attack_technique_registry.pyasserting the"pair"spec points atPAIRAttackand has the expected tags. The existing parametrizedSCENARIO_TECHNIQUEStests automatically cover spec well-formedness and name uniqueness for the new entry.Full regression sweep:
uv run pytest tests/unit/executor/attack tests/unit/scenario tests/unit/registry -q-> 1535 passed, no regressions to TAP.Documentation: The existing
doc/code/executor/attack/tap_attack.pyjupytext notebook gains a short paragraph explaining PAIR as the structural degenerate case of TAP and pointing atPAIRAttack. Decided against a standalone notebook on review feedback since the configuration is otherwise identical. The mermaid diagram indoc/code/executor/attack/0_attack.mdwas updated to add aPAIRAttacknode under TAP. The notebook was synced viauv run jupytext --update --to ipynb doc/code/executor/attack/tap_attack.pyto preserve persisted cell outputs.Pre-commit clean: ruff, ruff-format, ty type-check, nbstripout, validate-docs all pass on the touched files.