Report harm scorer accuracy split by inter-rater agreement (a third of violence gold labels are single-vote)

[WatchTree-19]

Is your feature request related to a problem?

A harm scorer is evaluated against one gold label per response. Where that label is the mean of several human raters, the mean hides how much the raters disagreed, and the eval reports a single MAE/accuracy as if the label were certain. On the multi-rater gold sets in pyrit/datasets/scorer_evals/harm, a large fraction of the binary harmful/not labels come down to a single rater's vote.

Measured on the current files (Krippendorff alpha computed with PyRIT's own pyrit.score.scorer_evaluation.krippendorff.krippendorff_alpha, ordinal, so it matches the *_metrics.jsonl values):

file	n	raters	ordinal alpha	exact 3-way agreement	binary label is a single-vote (2-1) split
violence_multi_score.csv	187	3	0.766	32.1%	31.0%
hate_speech_multi_score.csv	199	3	0.861	42.2%	17.6%
mini_hate_speech.csv	15	3	0.659	20.0%	26.7%

On violence, nearly a third of the binary gold labels are 2-1 splits, so a scorer marked wrong on those rows was outvoted by one human rather than overruled by a consensus. A point metric spends the scorer's error budget partly on rows the humans themselves split on, and a scorer that is well calibrated to that ambiguity (for example one that abstains where humans divide) is penalised for it rather than credited.

The other eight harm gold sets (exploits, hate_speech, info_integrity, privacy, representational, self_harm, sexual, violence) ship a single rating, so their disagreement cannot be observed at all and their labels are treated as ground truth with no uncertainty signal.

Describe the solution you'd like

1. In HarmScorerEvaluator, when a gold set has more than one rater, additionally report accuracy/MAE split by agreement level: unanimous rows vs contested (non-unanimous) rows, from the same eval run. A scorer at 0.87 overall that is 0.95 on agreed rows and near chance on contested ones is behaving correctly, and the split makes that visible where the aggregate hides it.
2. 2. Optionally mark the single-rater gold sets as provisional in the harm-definition metadata, so a low scorer number on them is read as possible label uncertainty rather than a scorer defect.

Describe alternatives you've considered

Down-weighting contested rows in the aggregate metric, or dropping them. Reporting the split is less opinionated: it changes nothing about the existing numbers and adds a second view, so no current metric moves.

Additional context

I have a small standalone audit script (numpy plus PyRIT's Krippendorff) that produces the table above and emits the contested row indices per file, which is what recommendation 1 would filter on. Happy to attach it or open a PR for the reporting change if this direction is wanted. Flagging Roman Lutz (@romanlutz) since this feeds directly into the dataset-curation work.

[WatchTree-19]

Following up on this with an actual measurement rather than just the proposal.

I trained the ViolenceClassifierScorer from #2626 and scored the multi-rater violence gold set (violence_multi_score.csv, 187 rows carrying three ratings each), then split the scorer's accuracy by how much the three raters agreed.

On the raters: Krippendorff alpha (ordinal) is 0.766, all three agree exactly on 32.1% of rows, and 31.0% of the binary labels are contested.

On the scorer: 100% accuracy on the 129 rows where all three raters agreed, and 67.2% on the 58 rows where they split, against 89.8% overall. That is in-sample, since these rows are also the scorer's training rows, so the levels are optimistic. The gap between the two buckets is the part I would rely on, not the absolute numbers.

That gap is the argument for reporting accuracy split by agreement level rather than as one figure. A scorer can look strong overall while sitting close to chance on exactly the items human raters found hard, and those are usually the items a red teaming verdict turns on. Happy to put the agreement-split computation behind a small helper if that would be useful to the evaluation code.

Separately, on the agent-led red teaming area: I have a working prototype of an escalation controller layered over the existing strategies, which tries single-turn first and escalates through red teaming and crescendo (with TAP optional) based on what the objective scorer returns, stopping as soon as the objective is met. I have run it end to end against a live target. Glad to share it if that shape is of interest.