FEAT: Chain-of-Thought Rendering for Attack Results

[ValbuenaVC]

Description

Adds opt-in rendering of OpenAI Responses API reasoning summaries across PyRIT’s output layer.

This PR:

• Adds include_reasoning_summaries=False to conversation and attack-result APIs.
• Renders provider-generated reasoning summaries while keeping them hidden by default.
• Validates persisted reasoning pieces against the OpenAI Responses reasoning-item shape.
• Prevents reasoning content from leaking after conversion to another data type.
• Renders reasoning summaries in subdued gray for Pretty output.
• Renders escaped reasoning headers in Markdown.
• Supports objective, pruned, and adversarial attack conversations.
• Clearly labels summaries as provider-generated summaries, not raw hidden chain-of-thought.

No breaking behavior is introduced because reasoning remains opt-in.

Background and Scope

The original story also requested investigation into JSON Schema adoption for scorers and attacks.

That infrastructure and adoption already landed through:

• [FEAT] Support JSON Schema in Responses #1177 — OpenAI Chat/Responses JSON Schema support
• FEAT: Adding Json Schema Pipeline to SeedPrompts #1432 — SeedPrompt schema pipeline, persistence, capability adaptation, and refusal-scorer adoption
• FEAT: Adding json schema to all scorer datasets #2017 — schema adoption across additional scorer datasets
• MAINT: Unify adversarial-chat JSON schema across Crescendo, TAP, and PAIR #2039 and MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback #2053 — shared adversarial-chat schema across Crescendo, TAP/PAIR, Red Teaming, and related attacks

Schemas remain domain-specific; this PR does not introduce a universal response schema or change target/scorer/attack semantics.

CoPyRIT already maps persisted reasoning pieces into its existing Reasoning panel and has mapper/component test coverage. This PR adds the missing PyRIT output parity. CoPyRIT currently has no scenario-results UI, so scenario frontend rendering is outside this PR.

Tests and Documentation

Added or expanded unit coverage for:

• Pretty and Markdown conversation rendering
• Attack objective, pruned, and adversarial conversations
• Empty and malformed reasoning summaries
• Exact OpenAI Responses payload validation
• Original/converted data-type leakage prevention
• Exact header literals, spacing, and Pretty colors
• Public helper argument forwarding

Local results:

• 171 passed
• pyrit.output: 98% statement coverage
• Ruff and formatting checks pass
• All required PR checks currently pass, including frontend unit, E2E, lint, and type-check workflows

Updated the paired output documentation with conversation, attack, and scenario examples and clarification that OpenAI exposes summaries rather than raw chain-of-thought.

[Copilot]

Pull request overview

Adds opt-in provider reasoning-summary rendering across conversation, attack-result, and scenario-result output.

Changes:

• Parses and validates OpenAI reasoning-summary payloads.
• Adds pretty and Markdown rendering with reasoning hidden by default.
• Propagates reasoning options through helpers and adds unit coverage.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyrit/output/conversation/base.py	Adds reasoning filtering and parsing.
pyrit/output/conversation/pretty.py	Renders pretty reasoning blocks.
pyrit/output/conversation/markdown.py	Renders Markdown reasoning blocks.
pyrit/output/attack_result/base.py	Extends the rendering contract.
pyrit/output/attack_result/pretty.py	Propagates reasoning through pretty output.
pyrit/output/attack_result/markdown.py	Propagates reasoning through Markdown output.
pyrit/output/scenario_result/base.py	Extends the scenario rendering contract.
pyrit/output/scenario_result/pretty.py	Adds scenario-level reasoning output.
pyrit/output/helpers.py	Exposes reasoning options in helpers.
tests/unit/output/conftest.py	Adds shared reasoning fixtures.
tests/unit/output/conversation/test_reasoning.py	Tests parsing, visibility, and formatting.
tests/unit/output/attack_result/test_pretty.py	Tests pretty attack reasoning.
tests/unit/output/attack_result/test_markdown.py	Tests Markdown attack reasoning.
tests/unit/output/scenario_result/test_pretty.py	Tests scenario reasoning output.
tests/unit/output/test_helpers.py	Tests helper argument forwarding.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pyrit/output/scenario_result/pretty.py:341

• When scenario reasoning is enabled, this nested attack printer renders each full objective conversation, and PrettyConversationMemoryPrinter displays every image_path piece in notebooks. Because this printer is always created with its default blur_images=False and output_scenario_async exposes no blur option, requesting reasoning summaries can unexpectedly display unblurred attack images with no way for callers to opt into the safety control available on output_attack_async. Please either render only the reasoning blocks or plumb the image-blur settings through the scenario API and nested printer.

        attack_result_printer = PrettyAttackResultMemoryPrinter(
            sink=sink,
            width=width,
            indent_size=indent_size,
            enable_colors=enable_colors,
        )

[behnam-o]

Looks great! I just have these two comments, one about whether scenario results should even expose this feature and another about how we seem to silently just not print a reasoning section if the model hasn't "reasoned" (I think in that case, it's good to expose/render this fact rather than treating it the same as if the model does not "support/generate" reasoning at all)

[behnam-o]

it might also be a good idea to include a sample prompt and output that triggers a reasoning piece and show how it gets rendered. for example, I used this snippet to see what the final output looks like, I think it's worth having a cell in the output notebook in the docs

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

[hannahwestra25]

plans to add this to the scanner ?

[ValbuenaVC]

plans to add this to the scanner ?

Not unless you feel it belongs there. Do users want to see the full reasoning trace during a scanner run? I'm not opposed to it, but I assumed the answer was no, though I may be wrong.