Skip to content

FEAT: Add PAIRAttack as a TAP alias with PAIR-definitional defaults pinned#1822

Merged
romanlutz merged 6 commits into
microsoft:mainfrom
romanlutz:romanlutz/pair-attack-alias
May 29, 2026
Merged

FEAT: Add PAIRAttack as a TAP alias with PAIR-definitional defaults pinned#1822
romanlutz merged 6 commits into
microsoft:mainfrom
romanlutz:romanlutz/pair-attack-alias

Conversation

@romanlutz
Copy link
Copy Markdown
Contributor

Description

PyRIT already ships TreeOfAttacksWithPruningAttack (TAP, Mehrotra et al. 2023, arXiv:2312.02119), which is a strict generalization of PAIR (Chao et al. 2023, arXiv:2310.08419). PAIR is the parallel-streams, single-branch, no-pruning special case of TAP. Today users searching for "PAIR" find nothing, so anyone trying to use PAIR has to know the TAP-to-PAIR mapping themselves and remember to manually disable branching/pruning, which silently turns a "PAIR" run into a misconfigured TAP run. Garak takes the same modeling stance: its PAIR probe is a TAP subclass with PAIR-specific defaults.

This adds PAIRAttack as a thin subclass of TreeOfAttacksWithPruningAttack that hides and hardcodes the two PAIR-definitional structural parameters (branching_factor=1, on_topic_checking_enabled=False) at the super().__init__() call site. Removing them from the subclass signature means they cannot drift via kwargs or set_default_value. The remaining knobs (tree_width, tree_depth, scoring/converter/adversarial configs) stay exposed with TAP's defaults (tree_width=3, tree_depth=5) so PAIR and TAP have comparable runtime cost for benchmarking. The TAP adversarial system prompt YAML is reused unchanged -- the behavioural difference between PAIR and TAP is purely structural-search, not prompt text.

PAIR-to-TAP mapping:

PAIR concept TAP parameter PAIRAttack handling
N parallel streams tree_width exposed, default 3
K refinement iterations tree_depth exposed, default 5
no tree expansion branching_factor hardcoded to 1
no off-topic pruning on_topic_checking_enabled hardcoded to False

The new attack is registered as "pair" in SCENARIO_TECHNIQUES with the same ["core", "multi_turn"] tags as "tap", so AdversarialBenchmark picks it up automatically alongside TAP via the existing _get_benchmarkable_specs filter.

Decisions worth a careful look

  • on_topic_checking_enabled is hidden + hardcoded to False on the same definitional reasoning as branching_factor. Happy to expose it as a user-tunable kwarg if reviewers prefer.
  • tree_width=3 / tree_depth=5 preserve TAP's defaults rather than matching the PAIR paper's N=20, K=3. This trades paper-fidelity for benchmark-cost parity with TAP.

Tests and Documentation

Tests (tests/unit/executor/attack/multi_turn/test_pair.py, new): 10 focused tests covering structural-default pinning, the signature-exclusion contract for branching_factor and on_topic_checking_enabled (these are absent from PAIRAttack.__init__ so they cannot be set via kwargs or set_default_value), parameter overrides, subclass relationship, shared TAPAttackContext, and inherited TAP validations (adversarial-target capabilities, FloatScaleThresholdScorer requirement). A registry test was added to tests/unit/registry/test_attack_technique_registry.py asserting the "pair" spec points at PAIRAttack and has the expected tags. The existing parametrized SCENARIO_TECHNIQUES tests automatically cover spec well-formedness and name uniqueness for the new entry.

Full regression sweep: uv run pytest tests/unit/executor/attack tests/unit/scenario tests/unit/registry -q -> 1535 passed, no regressions to TAP.

Documentation: The existing doc/code/executor/attack/tap_attack.py jupytext notebook gains a short paragraph explaining PAIR as the structural degenerate case of TAP and pointing at PAIRAttack. Decided against a standalone notebook on review feedback since the configuration is otherwise identical. The mermaid diagram in doc/code/executor/attack/0_attack.md was updated to add a PAIRAttack node under TAP. The notebook was synced via uv run jupytext --update --to ipynb doc/code/executor/attack/tap_attack.py to preserve persisted cell outputs.

Pre-commit clean: ruff, ruff-format, ty type-check, nbstripout, validate-docs all pass on the touched files.

romanlutz and others added 4 commits May 27, 2026 05:10
PAIR (Chao et al. 2023, arXiv:2310.08419) is a structural special case of
TreeOfAttacksWithPruningAttack (Mehrotra et al. 2023, arXiv:2312.02119):
parallel single-branch iterative refinement with no tree expansion and no
off-topic pruning. PyRIT already shipped TAP but had no way for users to
search/discover/register PAIR, leading to misconfigured "PAIR" runs that
were really TAP.

This adds PAIRAttack as a thin TreeOfAttacksWithPruningAttack subclass
that pins branching_factor=1 and on_topic_checking_enabled=False at the
super().__init__() call site and removes them from its own signature, so
those PAIR-definitional structural parameters cannot drift. tree_width
and tree_depth remain user-tunable with TAP's defaults (3 and 5) for
benchmark-cost comparability between PAIR and TAP. The TAP adversarial
system prompt YAML is reused unchanged (matching garak's choice and the
paper's prompt structure).

- New pyrit/executor/attack/multi_turn/pair.py
- Exported from pyrit.executor.attack[.multi_turn]
- Registered as "pair" in SCENARIO_TECHNIQUES so AdversarialBenchmark
  picks it up alongside "tap"
- Unit tests covering structural-default pinning and the
  signature-exclusion contract
- Registry test asserting the spec points at PAIRAttack
- New doc/code/executor/attack/pair_attack.py jupytext notebook with
  PAIR<->TAP parameter mapping, wired into doc/myst.yml
- 0_attack.md mermaid diagram updated to add the PAIRAttack node

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…drop unreadable docstring table

- Remove doc/code/executor/attack/pair_attack.{py,ipynb}; mention PAIR
  as the structural degenerate case of TAP in tap_attack.py instead.
  Unregister pair_attack.ipynb from doc/myst.yml.
- Sort __all__ alphabetically in both
  pyrit/executor/attack/__init__.py and
  pyrit/executor/attack/multi_turn/__init__.py.
- Replace the ASCII grid table in PAIRAttack's class docstring with a
  short inline description -- the table didn't render readably in
  plaintext / IDE tooltips.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit regenerated the notebook from .py via jupytext, which dropped persisted execution outputs. Use jupytext --update instead so the PAIR-mention markdown edit applies without nuking outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PyRIT uses plain Google-style docstrings with bare class names (per the style guide); drop the :class:\...\ and double-backtick \\...\\ reST conventions I introduced.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread doc/code/executor/attack/tap_attack.py
Comment thread pyrit/executor/attack/multi_turn/pair.py Outdated
@hannahwestra25 hannahwestra25 self-assigned this May 28, 2026
romanlutz and others added 2 commits May 28, 2026 11:42
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@romanlutz romanlutz enabled auto-merge May 29, 2026 13:04
@romanlutz romanlutz added this pull request to the merge queue May 29, 2026
Merged via the queue into microsoft:main with commit 087e35b May 29, 2026
48 checks passed
@romanlutz romanlutz deleted the romanlutz/pair-attack-alias branch May 29, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants