Skip to content

FEAT: Use registry-based default objective scorer in scenarios#1528

Merged
rlundeen2 merged 6 commits intomicrosoft:mainfrom
rlundeen2:users/rlundeen/2026_03_20_scenario_default_scorer
Mar 24, 2026
Merged

FEAT: Use registry-based default objective scorer in scenarios#1528
rlundeen2 merged 6 commits intomicrosoft:mainfrom
rlundeen2:users/rlundeen/2026_03_20_scenario_default_scorer

Conversation

@rlundeen2
Copy link
Copy Markdown
Contributor

@rlundeen2 rlundeen2 commented Mar 21, 2026

Centralizes the default objective scorer into Scenario._get_default_objective_scorer(), replacing duplicated hardcoded scorer construction across ContentHarms, Jailbreak, and RedTeamAgent. The base class method checks the ScorerRegistry for a scorer tagged DEFAULT_OBJECTIVE_SCORER (the best F1 scorer from initialization), falling back to TrueFalseInverterScorer(SelfAskRefusalScorer(OpenAIChatTarget())).

Note, we also run scorer metrics from the registry. Because of this, there was a mismatch between the Scenario default scorers and the metrics. Now, these match so metrics are reported.

@nina-msft
Copy link
Copy Markdown
Contributor

nina-msft commented Mar 23, 2026

Assuming we're only updating content_harms, jailbreaks, and foundry scenarios to use the default objective scorer from core because they most closely resemble that type of scorer - but all other scenarios have custom handling for the objective scorer that may rely on the user having a "gpt-4o-unsafe" model.

I see value in removing duplicate code from the changes in this PR and also making it so that we can use metrics-informed scorers from the registry when possible for these scenarios...but for other scenarios we're not getting that same benefit. Are there follow-up stories we need to make so all scenarios are covered?

Copy link
Copy Markdown
Contributor

@nina-msft nina-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a net-positive change, we may want to consider follow up stories for other scenarios (based off of my comment below) so that we can offer this advantage across the board for scenarios as it makes sense.

Most of the comments look like nits but all are small improvements - please take if no objections :)

@rlundeen2 rlundeen2 merged commit d0d90fa into microsoft:main Mar 24, 2026
35 checks passed
jbolor21 pushed a commit to jbolor21/jbolor-PyRIT that referenced this pull request Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants