Skip to content

FEAT: SeedSimulatedConversation to generate simulated conversations in attacks#1304

Merged
rlundeen2 merged 34 commits intomicrosoft:mainfrom
rlundeen2:users/rlundeen/2026_01_02_simulated_attack_param
Jan 8, 2026
Merged

FEAT: SeedSimulatedConversation to generate simulated conversations in attacks#1304
rlundeen2 merged 34 commits intomicrosoft:mainfrom
rlundeen2:users/rlundeen/2026_01_02_simulated_attack_param

Conversation

@rlundeen2
Copy link
Copy Markdown
Contributor

@rlundeen2 rlundeen2 commented Jan 3, 2026

Refactor Seed Models, allowing Seeds to configure simulating conversation in Attacks.

  • Separated SeedGroup and SeedAttackGroup into distinct files, where SeedAttackGroup is a subclass - specifically a SeedGroup with exactly one objective. I imagine in the future there will be more groups (e.g. SeedBenchmarkGroup).
  • Added SeedSimulatedConversation to be able to configure simulating conversations in datasets. In the future I expect there will be more seed types (e.g. SeedExpectedResponse that might have an expected return value and required for benchmarks)
  • Added code in AttackExecutor and Attack's FromSeedGroups to generate simulated conversations based on SeedSimulatedConversation

Multiple small refactors to clearly separate responsibilities along the way. In the future

To document, I added examples in 2_seed_programming that shows how Seeds translate to attack parameters, and included the new SeedSimulatedConversation

DB Migration Notes

The following failed before migration:

pytest .\tests\integration\memory\test_azure_sql_memory_integration.py::test_get_seeds_with_metadata_filter

Migration in test and prod:

ALTER TABLE [SeedPromptEntries] ADD [seed_type] NVARCHAR(50) NULL;

Then the test passes

@rlundeen2 rlundeen2 changed the title FEAT: SeedSimulatedConversation to generate simulated conversations in attacks Draft FEAT: SeedSimulatedConversation to generate simulated conversations in attacks Jan 4, 2026
@rlundeen2 rlundeen2 changed the title Draft FEAT: SeedSimulatedConversation to generate simulated conversations in attacks FEAT: SeedSimulatedConversation to generate simulated conversations in attacks Jan 6, 2026
Comment thread pyrit/models/seeds/seed_group.py Outdated
Comment thread pyrit/models/seeds/seed_attack_group.py
Comment thread pyrit/models/seeds/seed_attack_group.py
Comment thread pyrit/models/seeds/seed_attack_group.py
Comment thread pyrit/models/seeds/seed_attack_group.py
@rlundeen2 rlundeen2 force-pushed the users/rlundeen/2026_01_02_simulated_attack_param branch from 4737f02 to 17541d8 Compare January 7, 2026 00:15
@rlundeen2 rlundeen2 merged commit 520257e into microsoft:main Jan 8, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants