FEAT: Policy Scorer Compatibility (phase 2.5)

doc/code/framework.md

@@ -258,7 +258,7 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Any decision an attack makes should be based on a scorer result
 - A scorer is not limited to a message, it could be anything (e.g. was this tool called or was this file written). It receives a `Scorable`, which identifies that evidence, and an optional `ScoringExpectation`.
 - `TrueFalseScorer` and `FloatScaleScorer` define result families. `MessageScorer` adds message resolution and message-only policy on top of them.
-- `Score.status` marks a verdict complete or undetermined, and the attack decides how to branch on it.
+- A scorer declares which evidence it reads, rather than the caller filtering evidence for it. A `MessageScorer` states the conversation roles and data types it reads on its `ScorerPromptValidator`.
 - **Does not own**: acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job, and aggregating scores across runs is analytics'. It may call a target to evaluate, but it doesn't send the attack's objective prompt or manage the conversation.
 
 **Framework Plans**:

doc/code/scoring/0_scoring.ipynb

@@ -219,11 +219,24 @@
     "storage and stores its SHA-256 digest. The score remains resolvable after the source file is\n",
     "removed.\n",
     "\n",
-    "A complete score has `status=\"complete\"` and a typed value. An undetermined score has\n",
-    "`status=\"undetermined\"` and no value. A fully blocked response is a complete negative result\n",
-    "by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.\n",
-    "`SelfAskRefusalScorer` is the intentional exception because a content-filter block is a\n",
-    "refusal, so it returns `True`. Other response errors remain undetermined."
+    "Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the\n",
+    "evidence, such as a message with no supported role or data type. A non-empty list contains\n",
+    "completed or undetermined scores.\n",
+    "\n",
+    "A complete score has `status=\"complete\"` and a typed domain verdict. An undetermined score\n",
+    "has `status=\"undetermined\"` and no value because supported evidence failed to load. A fully\n",
+    "blocked response is a complete negative result by default: `False` for message true/false\n",
+    "scorers and `0.0` for message float-scale scorers. `SelfAskRefusalScorer` is the intentional\n",
+    "exception because a content-filter block is a refusal, so it returns `True`.\n",
+    "\n",
+    "A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.\n",
+    "A message scorer names the conversation roles it reads with `supported_roles` on its\n",
+    "`ScorerPromptValidator`. Prepended (`simulated_assistant`) turns are fabricated history, so a\n",
+    "scorer must opt in to read them. Every scorer still receives a failed response, because a\n",
+    "scorer whose evidence never came from the response must run even when the response failed.\n",
+    "Explicit `role_filter` and `skip_on_error_result` values remain supported until removal.\n",
+    "Callers that rely on their historical defaults must now pass them explicitly. New code\n",
+    "should use `supported_roles` and the scorer's unreadable-evidence fallback instead."
    ]
   },
   {

doc/code/scoring/0_scoring.py

@@ -121,11 +121,24 @@
 # storage and stores its SHA-256 digest. The score remains resolvable after the source file is
 # removed.
 #
-# A complete score has `status="complete"` and a typed value. An undetermined score has
-# `status="undetermined"` and no value. A fully blocked response is a complete negative result
-# by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.
-# `SelfAskRefusalScorer` is the intentional exception because a content-filter block is a
-# refusal, so it returns `True`. Other response errors remain undetermined.
+# Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the
+# evidence, such as a message with no supported role or data type. A non-empty list contains
+# completed or undetermined scores.
+#
+# A complete score has `status="complete"` and a typed domain verdict. An undetermined score
+# has `status="undetermined"` and no value because supported evidence failed to load. A fully
+# blocked response is a complete negative result by default: `False` for message true/false
+# scorers and `0.0` for message float-scale scorers. `SelfAskRefusalScorer` is the intentional
+# exception because a content-filter block is a refusal, so it returns `True`.
+#
+# A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.
+# A message scorer names the conversation roles it reads with `supported_roles` on its
+# `ScorerPromptValidator`. Prepended (`simulated_assistant`) turns are fabricated history, so a
+# scorer must opt in to read them. Every scorer still receives a failed response, because a
+# scorer whose evidence never came from the response must run even when the response failed.
+# Explicit `role_filter` and `skip_on_error_result` values remain supported until removal.
+# Callers that rely on their historical defaults must now pass them explicitly. New code
+# should use `supported_roles` and the scorer's unreadable-evidence fallback instead.
 # %% [markdown]
 # ## Scoring directly
 #

doc/code/scoring/3_combining_scorers.ipynb

@@ -100,6 +100,12 @@
     "text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer\n",
     "kind as its input.\n",
     "\n",
+    "An empty child result means that the scorer did not apply. A composite scorer ignores empty\n",
+    "child results and aggregates the remaining results. It returns an empty list if every child\n",
+    "result is empty. Inverter and threshold wrappers pass an empty result through unchanged.\n",
+    "A conversation wrapper returns an empty result when it finds no applicable conversation\n",
+    "evidence or its child returns no score. Any outer wrapper then applies the rules above.\n",
+    "\n",
     "Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not\n",
     "project those APIs from their children. Score wrappers through the canonical `Scorable` API.\n",
     "\n",

doc/code/scoring/3_combining_scorers.py

@@ -83,6 +83,12 @@
 # text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer
 # kind as its input.
 #
+# An empty child result means that the scorer did not apply. A composite scorer ignores empty
+# child results and aggregates the remaining results. It returns an empty list if every child
+# result is empty. Inverter and threshold wrappers pass an empty result through unchanged.
+# A conversation wrapper returns an empty result when it finds no applicable conversation
+# evidence or its child returns no score. Any outer wrapper then applies the rules above.
+#
 # Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not
 # project those APIs from their children. Score wrappers through the canonical `Scorable` API.
 #

pyrit/executor/attack/multi_turn/crescendo.py

@@ -713,9 +713,7 @@ async def _score_response_async(self, *, context: CrescendoAttackContext) -> Sco
                 response=context.last_response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=context.objective,
-                skip_on_error_result=False,
             )
 
         objective_score = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/multi_prompt_sending.py

@@ -408,9 +408,7 @@ async def _evaluate_response_async(self, *, response: Message, objective: str) -
                 response=response,
                 auxiliary_scorers=self._auxiliary_scorers,
                 objective_scorer=self._objective_scorer if self._objective_scorer else None,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         objective_scores = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/tree_of_attacks.py

@@ -804,12 +804,11 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
         and any auxiliary scorers (which provide additional metrics). The scoring results are
         used by the TAP algorithm to decide which branches to explore further.
 
-        Blocked or errored responses are scored via the scorer's unified default behavior:
-        ``TrueFalseScorer`` returns
-        ``Score(False)`` and ``FloatScaleScorer``
-        returns ``Score(0.0)`` whenever no supported pieces remain after validator filtering
-        (the normal outcome for a blocked piece). This keeps blocked branches at the bottom
-        of the priority queue without needing attack-level error mapping.
+        Scorers apply their own unreadable-response policy. A fully blocked response uses the
+        scorer family's neutral fallback unless the scorer overrides it. An unreadable transport
+        or protocol response produces an undetermined score. A response with no supported role
+        or data type makes the scorer return ``[]``, so this method raises ``RuntimeError``. Tree
+        of Attacks does not map these outcomes to ``False`` or ``0.0``.
 
         Args:
             response (Message): The response from the objective target to evaluate.
@@ -841,9 +840,7 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=False,
             )
 
         # Extract objective score

pyrit/executor/attack/single_turn/prompt_sending.py

@@ -367,9 +367,7 @@ async def _evaluate_response_async(
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         if not self._objective_scorer:

pyrit/score/__init__.py

@@ -39,7 +39,7 @@
     from pyrit.score.float_scale.system_prompt_extraction_scorer import SystemPromptExtractionScorer
     from pyrit.score.float_scale.video_float_scale_scorer import VideoFloatScaleScorer
     from pyrit.score.message_scorable_resolver import MessageScorableResolver
-    from pyrit.score.message_scorer import MessageScorer, MessageScoringOptions
+    from pyrit.score.message_scorer import MessageScorer
     from pyrit.score.response_handler import CallableResponseHandler, JsonSchemaResponseHandler, ResponseHandler
     from pyrit.score.scorable import ContentScorable, MessageScorable, Scorable
     from pyrit.score.scorer import Scorer
@@ -176,7 +176,6 @@
     "MessageScorableResolver": "pyrit.score.message_scorable_resolver",
     "MessageScorable": "pyrit.score.scorable",
     "MessageScorer": "pyrit.score.message_scorer",
-    "MessageScoringOptions": "pyrit.score.message_scorer",
     "MethKeywordScorer": "pyrit.score.true_false.regex.meth_keyword_scorer",
     "MetricsType": "pyrit.score.scorer_evaluation.metrics_type",
     "NerveAgentKeywordScorer": "pyrit.score.true_false.regex.nerve_agent_keyword_scorer",

pyrit/score/conversation_scorer.py

@@ -9,6 +9,7 @@
     ContentScorable,
     Message,
     MessagePiece,
+    Scorable,
     Score,
     ScoringExpectation,
 )
@@ -60,14 +61,35 @@ def _build_scoring_message(self, *, message: Message) -> Message | None:
         Keep the trigger that identifies the conversation to acquire.
 
         The trigger content is not sent to the child scorer. ``_score_prepared_message_async``
-        replaces it with a text view of the full conversation. The base class applies
-        ``skip_on_error_result`` before this hook.
+        replaces it with a text view of the full conversation. Overriding this hook keeps an
+        unreadable trigger, because the conversation behind it is still there to read.
 
         Returns:
             Message | None: The trigger message, or None if it has no pieces.
         """
         return message if message.message_pieces else None
 
+    def _reads_any_role(self, *, message: Message, anchor: Scorable | None) -> bool:
+        """
+        Defer role policy until the conversation locator has acquired its evidence.
+
+        Returns:
+            bool: True because the trigger identifies history; it is not the evidence itself.
+        """
+        return True
+
+    def _build_fallback_score(self, *, message: Message, objective: str | None) -> list[Score]:
+        """
+        Return ``[]`` when the conversation trigger does not yield applicable evidence.
+
+        Returns:
+            list[Score]: Always ``[]``.
+        """
+        return []
+
+    def _validate_scoring_message(self, *, message: Message, objective: str | None) -> None:
+        """Skip message validation because the trigger is only a conversation locator."""
+
     async def _score_prepared_message_async(
         self,
         *,
@@ -83,7 +105,7 @@ async def _score_prepared_message_async(
         error JSON when ``should_score_blocked_content`` is turned off). This ensures the wrapped
         scorer's text-only validator accepts the synthetic message and scores the full
         conversation, even when the triggering turn was blocked or errored; the wrapped
-        scorer's fallback only fires when the rendered conversation is genuinely unscoreable.
+        scorer returns ``[]`` when the rendered conversation is not applicable.
 
         The wrapped scorer is invoked through its non-persisting nested path. The outer
         ``Scorer.score_async`` persists the returned scores exactly once, anchored to the
@@ -95,7 +117,8 @@ async def _score_prepared_message_async(
             expectation (ScoringExpectation | None): What the wrapped scorer should look for.
 
         Returns:
-            list[Score]: List of Score objects from the underlying scorer
+            list[Score]: The wrapped scorer's completed or undetermined results, or ``[]``
+                when no applicable conversation evidence or child score exists.
 
         Raises:
             ValueError: If conversation with the given ID is not found in memory.
@@ -123,7 +146,7 @@ async def _score_prepared_message_async(
         for conv_message in conversation:
             for piece in conv_message.message_pieces:
                 # Only include user and assistant messages in the conversation text
-                if piece.api_role in ["user", "assistant", "tool"]:
+                if piece.api_role in ["user", "assistant", "tool"] and self._validator.is_role_supported(piece):
                     role_display = "Assistant (simulated)" if piece.is_simulated else piece.api_role.capitalize()
                     # For blocked pieces with partial content, use the partial content
                     # instead of the error JSON when should_score_blocked_content is enabled
@@ -137,6 +160,9 @@ async def _score_prepared_message_async(
                         text = piece.converted_value
                     conversation_text += f"{role_display}: {text}\n"
 
+        if not conversation_text:
+            return []
+
         wrapped_scorer = self._get_wrapped_scorer()
         scores = await wrapped_scorer._score_nested_async(
             scorable=ContentScorable(value=conversation_text),
@@ -193,7 +219,7 @@ def create_conversation_scorer(
         scorer (Scorer): The true/false or float-scale scorer to wrap for
             conversation-level evaluation. It must support text ``ContentScorable`` evidence.
         validator (ScorerPromptValidator | None): Optional validator override.
-            If not provided, uses the wrapped scorer's validator.
+            If not provided, uses the conversation scorer's default text validator.
 
     Returns:
         Scorer: A ConversationScorer instance that is also an instance of the wrapped scorer's type.

pyrit/score/float_scale/azure_content_filter_scorer.py

@@ -342,67 +342,40 @@ async def _score_piece_async(self, message_piece: MessagePiece, *, objective: st
 
     def _build_fallback_score(self, *, message: Message, objective: str | None) -> list[Score]:
         """
-        Build one neutral ``0.0`` fallback score per configured harm category.
+        Build one fallback score per configured harm category.
 
         AzureContentFilterScorer's normal output is one score per category in
-        ``self._harm_categories``. To preserve that shape on blocked / error / filtered
-        input, this override emits one neutral ``0.0`` score per configured category
-        (each tagged with the category name and matching the normal-path metadata),
-        instead of the single category-less score produced by the base
-        ``MessageFloatScaleScorer._build_fallback_score``.
-
-        Inspects the first message piece to tailor the rationale/description for
-        blocked, error, and filtered cases.
+        ``self._harm_categories``. To preserve that shape for blocked and unreadable
+        responses, this override emits one result per configured category. Unsupported
+        evidence returns ``[]``.
 
         Args:
-            message (Message): The message whose first piece is inspected for status.
+            message (Message): The message whose fallback result is expanded.
             objective (str | None): The objective associated with this scoring call.
 
         Returns:
-            list[Score]: One ``0.0`` ``float_scale`` score per configured harm category,
-                each attributed to the first piece.
+            list[Score]: ``[]`` for non-applicable evidence; otherwise, one completed or
+                undetermined result per configured harm category.
 
         Raises:
             ValueError: If the first message piece has no ``id`` or ``original_prompt_id``.
         """
-        first_piece = message.message_pieces[0]
-        piece_id = first_piece.id or first_piece.original_prompt_id
-        if piece_id is None:
-            raise ValueError("Cannot create score: message piece has no id or original_prompt_id")
-
-        if first_piece.is_blocked():
-            status = "The response was blocked with no content to score"
-            description = "Blocked response; returning 0.0 per configured category."
-        elif first_piece.has_error():
-            # A transport or protocol failure is not the target's answer, so there is no verdict.
-            return [
-                self._build_undetermined_score(
-                    rationale=f"Response had an error: {first_piece.response_error}; no verdict was reachable.",
-                    description="Error response; no verdict was reachable.",
-                    message_piece_id=piece_id,
-                    objective=objective,
-                    score_category=[category.value],
-                    score_metadata={"azure_severity": 0},
-                )
-                for category in self._harm_categories
-            ]
-        else:
-            status = "No supported pieces to score after filtering"
-            description = "No pieces to score after filtering; returning 0.0 per configured category."
-
-        rationale = f"{status}; returning 0.0 for each configured harm category."
-        metadata: dict[str, str | int | float] = {"azure_severity": 0}
+        fallback_scores = super()._build_fallback_score(message=message, objective=objective)
+        if not fallback_scores:
+            return []
+        fallback = fallback_scores[0]
 
         return [
             Score(
-                score_value="0.0",
-                score_value_description=description,
+                score_value=fallback.score_value,
+                status=fallback.status,
+                score_value_description=fallback.score_value_description,
                 score_type="float_scale",
                 score_category=[category.value],
-                score_metadata=metadata,
-                score_rationale=rationale,
+                score_metadata={"azure_severity": 0},
+                score_rationale=fallback.score_rationale,
                 scorer_class_identifier=self.get_identifier(),
-                message_piece_id=piece_id,
+                message_piece_id=fallback.message_piece_id,
                 objective=objective,
             )
             for category in self._harm_categories