FIX: Include adversarial config in attack and eval identity - #1959
Merged
rlundeen2 merged 6 commits intoJun 9, 2026
Merged
Conversation
Add a get_attack_adversarial_config() hook (mirroring get_attack_scoring_config()) so the adversarial chat target plus its system/seed prompts flow into both the component identity hash and the atomic-attack eval hash. Previously the adversarial_chat eval rule existed but nothing ever emitted an adversarial_chat child, so it was dead config. - Add inline system_prompt support to AttackAdversarialConfig and deprecate system_prompt_path (removed_in 0.17.0); add resolve_adversarial_system_prompt. - Emit adversarial_chat child + adversarial_system_prompt/adversarial_seed_prompt params from the base attack strategy identity. - Wire the hook through RedTeaming, Crescendo, TAP/PAIR, RolePlay, and ContextCompliance. - Update AttackTechniqueFactory to take adversarial_system_prompt (not path). - Migrate internal callsites off the deprecated system_prompt_path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hannahwestra25
self-requested a review
June 9, 2026 15:25
…mpts Inline system-prompt strings are trusted and declare the required parameters themselves, so the missing-parameter check was a no-op for them. Restructure resolve_adversarial_system_prompt so validation runs only for explicitly provided SeedPrompts, making the trust-vs-validate split explicit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
enabled auto-merge
June 9, 2026 20:45
Covers the previously-untested branches flagged by diff coverage: the both-set deprecation warning and explicit-SeedPrompt validation in resolve_adversarial_system_prompt; the hook return-None guards across the five adversarial strategies; RedTeaming's default seed-prompt fallback; and the factory's YAML system_prompt_path resolution, baked-prompt conflict, and SeedPrompt serialization paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace a :class: reST role with double-backticks in attack_config (MyST renders docstrings, not Sphinx), and apply ruff format + import sorting to feature files flagged by the CI pre-commit run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-config-eval-identifier
main dropped the MessagePiece.scores attribute in favor of CentralMemory.get_prompt_scores(). Update TAP's _get_score_for_response to use the new API so ty type-checks and scores resolve correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a
get_attack_adversarial_config()hook (mirroring get_attack_scoring_config()) so the adversarial chat target plus its system/seed prompts flow into both the component identity hash and the atomic-attack eval hash. Previously the adversarial_chat eval rule existed but nothing ever emitted an adversarial_chat child, so it was dead config.So before this fix, two attacks that differed only by adversarial model, adversarial system prompt, or adversarial seed prompt produced identical eval hashes (and identical component hashes). Now, any changes to these values produces different eval hashes and identity hashes.