Skip to content

FEAT add StrategySequenceAttack compound attack primitive#1819

Merged
hannahwestra25 merged 8 commits into
microsoft:mainfrom
hannahwestra25:hawestra/strategy_sequence_extraction
May 28, 2026
Merged

FEAT add StrategySequenceAttack compound attack primitive#1819
hannahwestra25 merged 8 commits into
microsoft:mainfrom
hannahwestra25:hawestra/strategy_sequence_extraction

Conversation

@hannahwestra25
Copy link
Copy Markdown
Contributor

@hannahwestra25 hannahwestra25 commented May 27, 2026 •
edited
Loading

What does this PR do?

Adds SequentialAttack — a compound AttackStrategy that chains multiple inner attacks against a single objective with a configurable completion policy. Think "try Crescendo first, fall back to PromptSending" without leaving the AttackStrategy layer.

Why?

Right now, if you want to try strategy A then B on the same objective, you either push that logic up to the Scenario layer (which breaks the one-objective → one-result invariant) or write custom glue code. SequentialAttack keeps that branching where it belongs — at the attack level — and stays composable in notebooks.

What's in the box

Component Description
SequentialAttack The compound strategy. Runs child attacks in order, returns one SequentialAttackResult.
SequentialChildAttack Bundles a strategy with per-child config (seed group, adversarial chat, scorer, labels).
SequentialAttackResult Extends AttackResult; exposes child_attack_results (live) and child_attack_result_ids (ID-only, for DB round-trip).
SequenceCompletionPolicy FIRST_SUCCESS (default), FIRST_DECISIVE, STRICT_ALL, EXHAUSTIVE, or LAST_RESULT.

Outcome aggregation

SUCCESS if any child succeeded, ERROR if all errored, UNDETERMINED if all undetermined, otherwise FAILURE.

Files changed

  • Implementation: pyrit/executor/attack/compound/sequential_attack.py + __init__.py exports
  • Docs: New notebook (4_sequential_attack.ipynb / .py), updated attack overview (0_attack.md, framework.md), cross-link from Crescendo notebook
  • Tests: 21 unit tests in test_sequential_attack.py covering completion policies, per-child overrides, outcome aggregation, label stamping, result shape, and edge cases

What this enables later

  • Replace max_attempts_on_failure with SequentialAttack([same] * N, completion_policy=FIRST_SUCCESS)
  • Adaptive scenario rewrite on top of this primitive
  • Mixed-pipeline patterns (TAP → Crescendo) as config, not code

rlundeen2
rlundeen2 reviewed May 27, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/strategy_sequence.py Outdated
rlundeen2
rlundeen2 reviewed May 27, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/strategy_sequence.py Outdated
rlundeen2
rlundeen2 reviewed May 27, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/strategy_sequence.py Outdated
rlundeen2
rlundeen2 reviewed May 27, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/strategy_sequence.py Outdated
@hannahwestra25 hannahwestra25 force-pushed the hawestra/strategy_sequence_extraction branch from d281a26 to 1557e09 Compare May 27, 2026 22:53
@hannahwestra25 hannahwestra25 changed the title FEAT extract StrategySequenceAttack compound attack (stacked on #1760) FEAT add StrategySequenceAttack compound attack primitive May 27, 2026
@hannahwestra25 hannahwestra25 mentioned this pull request May 27, 2026
Open
@hannahwestra25 hannahwestra25 force-pushed the hawestra/strategy_sequence_extraction branch 2 times, most recently from 444d727 to 236f80d Compare May 28, 2026 00:17
@hannahwestra25 hannahwestra25 marked this pull request as ready for review May 28, 2026 00:32
@hannahwestra25
feat: add SequentialAttack compound primitive
cd690e6 
Adds a thin AttackStrategy that runs a sequence of inner attacks against
one objective, controlled by a single SequenceMode.

- SequentialAttack chains AttackStrategy items via AttackExecutor and
  returns one envelope SequentialAttackResult, preserving the
  one-objective to one-AttackResult invariant.
- SequentialAttackItem bundles per-item strategy + seed_group +
  adversarial_chat + objective_scorer + memory_labels.
- SequentialAttackResult(AttackResult) exposes metadata-backed
  attempt_result_ids listing each inner attempt id in dispatch order
  (mirrors TAPAttackResult / CrescendoAttackResult pattern: no new
  dataclass fields, safe with to_dict()).
- SequenceMode collapses iteration + outcome aggregation into a single
  intent-named knob:
    FIRST_SUCCESS  - stop on SUCCESS; resilient past ERROR/FAILURE (default)
    FIRST_DECISIVE - stop on SUCCESS or ERROR; fail-fast adaptive
    STRICT_ALL     - stop on first non-SUCCESS; required pipeline
    EXHAUSTIVE     - run all; any-success aggregation
    LAST_RESULT    - run all; inherit final item's outcome

Splits the compound primitive out of microsoft#1760 so the adaptive scenario
rewrite can sit on top of it. Uses AttackContext[AttackParameters]
directly per review feedback (no thin context/params subclasses).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25 hannahwestra25 force-pushed the hawestra/strategy_sequence_extraction branch from 236f80d to cd690e6 Compare May 28, 2026 00:36
hannahwestra25 and others added 4 commits May 28, 2026 10:23
@hannahwestra25
Merge branch 'main' into hawestra/strategy_sequence_extraction
d5d85dc
@hannahwestra25
refactor: rename SequentialAttackItem -> SequentialAttackStep
36f47b4 
The Item suffix was generic. Step reads naturally for an ordered
sequence, pairs cleanly with the existing StopPolicy/SequenceMode
vocabulary, and has no class-name collisions in the codebase.

Cascade renames for internal consistency:
- constructor kwarg `items=` -> `steps=`
- internal `self._items` -> `self._steps` and loop vars
- private method `_run_item_async` -> `_run_step_async`
  (and its keyword-only `item` parameter -> `step`)
- docstring/example/comment vocabulary updated throughout
- test helpers (`_patch_run_item` -> `_patch_run_step`) and the
  affected test method names

Intentionally left alone: the `attempt_result_ids` property,
`ATTEMPT_RESULT_IDS_KEY` constant, and `metadata[attempt_result_ids]`
- each step still produces one attempt, and renaming would also break
  any persisted-metadata readers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
refactor: rename SequenceMode -> SequencePolicy
d8d7568 
Each enum value encodes a *rule* (stop condition + outcome aggregation)
governing the sequence, which Policy describes more precisely than
the generic Mode. Also harmonizes with the PR description's original
`StopPolicy` terminology.

Cascade renames for internal consistency:
- constructor kwarg `mode=` -> `policy=`
- attribute `self._mode` -> `self._policy`
- all `self._mode is SequenceMode.X` comparisons updated
- docstrings / code-block / comment vocabulary updated where `mode`
  referred to the SequencePolicy concept
- test parametrize tuple name, function parameter, `mode=mode` kwarg
  in the dispatch call, and `test_default_mode_is_first_success` ->
  `test_default_policy_is_first_success`

Intentionally left alone: enum member names (FIRST_SUCCESS,
FIRST_DECISIVE, STRICT_ALL, EXHAUSTIVE, LAST_RESULT) and their
string-value backings (`first_success` etc.) -- changing the
string values would break any persisted metadata using them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
docs: add SequentialAttack notebook and cross-references
717744c 
- new 4_sequential_attack.{py,ipynb} demonstrating a Crescendo -> PromptSending fallback chain, per-step memory_labels, inspection of inner attempts via attempt_result_ids, and a SequencePolicy reference table

- 0_attack.md gains a Compound Attacks bullet and updates the AttackStrategy mermaid diagram with SequentialAttack

- myst.yml registers the new notebook in the docs nav

- framework.md mentions compound strategies alongside single/multi-turn attacks

- 3_crescendo_attack.{py,ipynb} adds a tip pointing at the new Sequential notebook

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
@hannahwestra25
docs: use American spelling 'behavior' in sequential attack notebook
4bafd79 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py
rlundeen2
rlundeen2 reviewed May 28, 2026
View reviewed changes
Comment thread pyrit/executor/attack/compound/sequential_attack.py Outdated
@hannahwestra25 hannahwestra25 force-pushed the hawestra/strategy_sequence_extraction branch from 37a1703 to 4bafd79 Compare May 28, 2026 18:45
hannahwestra25 and others added 2 commits May 28, 2026 15:11
@hannahwestra25
refactor: apply rlundeen2 review feedback to SequentialAttack
4a818d2 
Addresses review threads on PR microsoft#1819:

- Rename SequentialAttackStep -> SequentialChildAttack; kwarg steps= ->
  child_attacks=; helper _run_step_async -> _run_child_attack_async.
- Rename SequencePolicy -> SequenceCompletionPolicy; kwarg policy= ->
  completion_policy=; saved on the result as both a typed completion_policy
  field and metadata['completion_policy'] (string) for DB round-trip.
- Add SequentialAttackResult.child_attack_results: list[AttackResult],
  populated at execute time. child_attack_result_ids (renamed from
  attempt_result_ids) now derives from it when populated, falling back to
  metadata['child_attack_result_ids'] for envelopes loaded from the DB.
  Constant renamed to CHILD_ATTACK_RESULT_IDS_KEY.
- Forward context._attribution through _run_child_attack_async to
  AttackExecutor.execute_attack_from_seed_groups_async so inner rows carry
  parent linkage when the compound is nested under a Scenario.
- Envelope now has conversation_id='', last_response=None, last_score=None
  -- the wrapper owns no conversation; callers use child_attack_results for
  per-child detail. executed_turns is the sum across child attacks that ran.
- Notebook and 0_attack.md updated to reflect the renamed surface and the
  new child_attack_results/child_attack_result_ids views.

Six new tests cover: envelope no-conversation invariant, dispatch-order
child results, completion-policy round-trip, metadata-fallback on
child_attack_result_ids, summed executed_turns, and attribution forwarding.
41/41 pass; pre-commit clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25
Merge branch 'main' into hawestra/strategy_sequence_extraction
3c52ebc
rlundeen2
rlundeen2 approved these changes May 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@rlundeen2 rlundeen2 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

May want to update the pr description to match

@hannahwestra25 hannahwestra25 added this pull request to the merge queue May 28, 2026
Merged via the queue into microsoft:main with commit 38b7ab8 May 28, 2026
48 checks passed
@hannahwestra25 hannahwestra25 deleted the hawestra/strategy_sequence_extraction branch May 28, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rlundeen2 rlundeen2 rlundeen2 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hannahwestra25 @rlundeen2