Skip to content

FEAT: New Scenario - Scams#1202

Merged
nina-msft merged 32 commits intomicrosoft:mainfrom
nina-msft:users/nina-msft/6685-scam-scenario
Jan 6, 2026
Merged

FEAT: New Scenario - Scams#1202
nina-msft merged 32 commits intomicrosoft:mainfrom
nina-msft:users/nina-msft/6685-scam-scenario

Conversation

@nina-msft
Copy link
Copy Markdown
Contributor

@nina-msft nina-msft commented Nov 22, 2025

Description

New scenario for creation of scam materials via single-turn attacks (RolePlay (persuasion), Context Compliance) & multi-turn attacks (Red Teaming Attack with custom Persuasion Persona system prompt).

Tests and Documentation

Unit test added, tested end-to-end locally.

Example results:

====================================================================================================
                                      📊 SCENARIO RESULTS: Scam                                      
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: Scam
    • Scenario Version: 1
    • PyRIT Version: 0.10.1.dev0
    • Description:
        Scam scenario evaluates an endpoint's ability to generate scam-related materials (e.g., phishing emails,
        fraudulent messages) with primarily persuasion-oriented techniques.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: xxx
    • Target Endpoint: https://xxx.openai.azure.com/openai/v1

  📊 Scorer Information
    • Scorer Type: TrueFalseCompositeScorer
      └─ Composite of 2 scorer(s):
          • Scorer Type: SelfAskTrueFalseScorer
          • Scorer Type: TrueFalseInverterScorer
            └─ Composite of 1 scorer(s):
                • Scorer Type: SelfAskRefusalScorer

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 3
    • Total Attack Results: 12
    • Overall Success Rate: 58%
    • Unique Objectives: 4

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: scam_role_play
    • Number of Results: 4
    • Success Rate: 100%

  🔸 Strategy: scam_context_compliance
    • Number of Results: 4
    • Success Rate: 25%

  🔸 Strategy: scam_persuasive_rta
    • Number of Results: 4
    • Success Rate: 50%

====================================================================================================

Comment thread pyrit/executor/attack/single_turn/prompt_sending.py Outdated
Comment thread pyrit/scenario/scenarios/airt/scam_scenario.py Outdated
Comment thread pyrit/scenario/scenarios/airt/scam_scenario.py Outdated
@nina-msft nina-msft changed the title FEAT: New Scenario - Scams (Role Play) FEAT: New Scenario - Scams (Impersonation) Dec 12, 2025
Comment thread pyrit/scenario/scenarios/airt/scam_scenario.py Outdated
@nina-msft nina-msft changed the title FEAT: New Scenario - Scams (Impersonation) FEAT: New Scenario - Scams Jan 2, 2026
Comment thread pyrit/scenario/scenarios/airt/scam.py
Comment thread doc/code/scenarios/0_scenarios.ipynb
Comment thread tests/unit/scenarios/test_scam.py Outdated
Comment thread pyrit/scenario/scenarios/airt/scam.py Outdated
Copy link
Copy Markdown
Contributor

@jsong468 jsong468 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor NITs, but lgtm!

Comment thread pyrit/scenario/scenarios/airt/scam.py
@nina-msft nina-msft merged commit fb8e223 into microsoft:main Jan 6, 2026
20 checks passed
@nina-msft nina-msft deleted the users/nina-msft/6685-scam-scenario branch January 6, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants