Skip to content

FIX: pyrit_shell run fails for every scenario with a flag-collision error - #2303

Merged
jsong468 merged 2 commits into
microsoft:mainfrom
jsong468:pyrit_shell_flag_collision_bug
Aug 3, 2026
Merged

FIX: pyrit_shell run fails for every scenario with a flag-collision error#2303
jsong468 merged 2 commits into
microsoft:mainfrom
jsong468:pyrit_shell_flag_collision_bug

Conversation

@jsong468

@jsong468 jsong468 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Bug Description

Running any scenario from pyrit_shell failed before argument parsing with:

Error: Scenario parameter flag '--memory-labels' collides with a built-in flag. Rename the parameter to avoid the collision.

Root cause: The shell registers each of a scenario's parameters as a CLI flag, using the full list from the API (scenario_meta.supported_parameters). That list combines the scenario's custom parameters with the framework's common parameters (memory_labels, max_concurrency, max_retries), which normalize to flags that already exist as built-in shell flags. The collision guard _validate_scenario_flag_collisions raised on the first such clash (memory_labels) instead of skipping it. Since every scenario carries the common parameters, run was broken universally.

pyrit_scan was unaffected: its _add_scenario_params_from_api silently skips any scenario flag that already exists as a built-in. The inconsistency between the two entry points was the bug.

Fix

Make the shell mirror pyrit_scan. Replaced _validate_scenario_flag_collisions with _resolve_scenario_flag_collisions in pyrit/cli/_cli_args.py, which silently drops any scenario spec whose flag is already taken — by a built-in flag or an earlier scenario param — so the earlier owner keeps the flag (first wins). This matches pyrit_scan._add_scenario_params_from_api, which skips any flag already registered on the parser, so both entry points now accept the same inputs.

parse_run_arguments now uses the returned filtered specs rather than relying on a validator side effect.

Testing

  • Added tests/unit/cli/test_cli_args.py::test_parse_run_arguments_skips_scenario_params_colliding_with_builtins — reproduces the airt.scam case (common params present) and asserts run parses successfully while the scenario's custom --max-turns flag still works.
  • Added tests/unit/cli/test_cli_args.py::test_parse_run_arguments_first_wins_on_scenario_vs_scenario_collision — confirms two scenario params normalizing to the same flag resolve first-wins (parity with pyrit_scan), not an error.
  • New tests pass. Full test_cli_args.py + test_pyrit_shell.py suites pass (86/86), no regressions.

@rlundeen2 rlundeen2 self-assigned this Jul 31, 2026
Comment thread pyrit/cli/_cli_args.py Outdated
@jsong468
jsong468 enabled auto-merge August 3, 2026 16:44
@jsong468
jsong468 added this pull request to the merge queue Aug 3, 2026
Merged via the queue into microsoft:main with commit 3911980 Aug 3, 2026
54 checks passed
@jsong468
jsong468 deleted the pyrit_shell_flag_collision_bug branch August 3, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants