FEAT: Block GUI sends when target doesn't support the modality by romanlutz · Pull Request #4 · romanlutz/PyRIT

romanlutz · 2026-05-06T13:10:46Z

Summary

The GUI previously didn't tell users when a target doesn't support a modality they're sending (e.g., attaching an image to a text-only target, or applying a text-to-image converter against a text-only target). This resulted in cryptic backend errors.

This PR exposes target input modality capabilities to the frontend and blocks sends with a clear warning when there's a mismatch.

Changes

Backend

pyrit/backend/models/targets.py — Added supported_input_data_types: list[str] to TargetInstance DTO
pyrit/backend/mappers/target_mappers.py — Flatten input_modalities from TargetCapabilities into the new field

Frontend — Modality validation

frontend/src/types/index.ts — Added supported_input_data_types to TargetInstance type
frontend/src/components/Chat/converterTypes.ts — Added file → binary_path mapping; added outputDataType to PieceConversion
frontend/src/components/Chat/ChatInputArea.tsx — Validates attachment types AND converter output types against target capabilities; disables send button with warning when mismatched
frontend/src/components/Chat/ChatWindow.tsx — Passes converterOutputDataTypes to ChatInputArea
frontend/src/components/Chat/ConverterPanel/ConverterPanel.tsx — Passes selected converter's output type to ConverterPreview
frontend/src/components/Chat/ConverterPanel/ConverterPreview.tsx — Includes outputDataType in PieceConversion

Frontend — Input area styling

Textarea auto-grows with content, capped at 60vh (solo) or 30vh each (when conversion active)
Both textareas scroll with matching custom scrollbar styling
Clear conversion button moved below send button with tooltip
Attach/converter buttons have round borders for consistency
Consistent badge indentation between Original and Converted

Tests

3 new backend mapper tests for supported_input_data_types
9 new frontend tests covering: text-only target + image, image-capable target, no target, audio on text+image, multiple unsupported types, send blocked, file/binary_path, converter output blocking, converter output supported

…oft#1389) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…icrosoft#1397) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…crosoft#1402) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…icrosoft#1403)

Co-authored-by: Anandan Sundar <205880232+anasundar_microsoft@users.noreply.github.com>

…alerts (microsoft#1405)

…osoft#1394) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…t#1408) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…crosoft#1390) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ions (microsoft#1395) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…dule init (microsoft#1428) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ntication (microsoft#1404) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…#1420) Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Victor Valbuena <50061128+ValbuenaVC@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…icrosoft#1429) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…LOG, PLE, Q, RSE, SLOT (microsoft#1412) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com>

…h labels (microsoft#1670) Co-authored-by: Copilot <copilot@github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Tejas Saubhage <tsaubhage0007@gmail.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…rosoft#1653) Co-authored-by: thirteeneight <purple.format.2o@icloud.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

…crosoft#1665) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…crosoft#1645)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…eTarget with GPT Image endpoints (microsoft#1661)

…soft#1676) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ry lecture, journalist interview) (microsoft#1677) Signed-off-by: precognitivem0nk <rextedgorman@gmail.com>

Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com> Co-authored-by: Behnam <60201681+behnam-o@users.noreply.github.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: varunj-msft <vjoginpalli@microsoft.com> Co-authored-by: Maifee Ul Asad <29339330+maifeeulasad@users.noreply.github.com> Co-authored-by: tejas_0007 <102517877+tejas0077@users.noreply.github.com> Co-authored-by: Tejas Saubhage <tsaubhage0007@gmail.com> Co-authored-by: Miroslav Horacek <115625094+thirteeneight@users.noreply.github.com> Co-authored-by: thirteeneight <purple.format.2o@icloud.com> Co-authored-by: Richard Lundeen <137218279+rlundeen2@users.noreply.github.com> Co-authored-by: jsong468 <songjustin@microsoft.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Behnam Ousat <behnamousat@microsoft.com> Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com>

…on tests, and endpoint target deprecation (microsoft#1672) Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

) Co-authored-by: Behnam Ousat <behnamousat@microsoft.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

Expose supported_input_data_types from TargetCapabilities through the backend DTO and mapper so the frontend knows which input modalities a target accepts. ChatInputArea now shows a non-blocking warning when the user attaches a file whose type (image, audio, video, file) is not in the target's supported input data types. Changes: - Add supported_input_data_types field to TargetInstance backend model - Flatten input_modalities in target_object_to_instance mapper - Add supported_input_data_types to frontend TargetInstance type - Add file->binary_path mapping in converterTypes - Derive unsupported attachment types in ChatInputArea and show warning - Add backend mapper tests and frontend component tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Change the unsupported modality warning from non-blocking to blocking: - Disable the send button when unsupported attachment types are present - Guard handleSend against unsupported types - Update warning text to instruct user to remove the attachment - Update tests to verify send is disabled Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extend modality validation to also check converter output data types, not just raw attachment types. When a converter transforms text into an unsupported type (e.g., text-to-image on a text-only target), the send button is now disabled with a warning. Changes: - Add outputDataType to PieceConversion interface - Pass output type from ConverterPanel through ConverterPreview - Pass converterOutputDataTypes from ChatWindow to ChatInputArea - Validate converter outputs against target supported_input_data_types - Update all affected tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Increase textInput and convertedTextarea maxHeight from 96px/80px to 30vh each, allowing each to use up to ~30% of viewport height - Remove JS-side 96px cap on auto-resize so CSS maxHeight controls it - Add matching webkit scrollbar styling to convertedTextarea - Use alignItems flex-start on both rows so they grow naturally Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When a conversion is active, both textareas now cap at 15vh each (half of the 30vh total) via inline style overrides. Without a conversion, the original textarea gets the full 30vh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Wrap both original and converted textareas in a single scrollable container (textScrollArea) with max-height 60vh and resize: vertical - Textareas grow naturally inside it with overflow hidden (no per-textarea scrollbars), the outer container handles scrolling - Move clear-conversion button to columnRight below send with tooltip - Both textareas share space equally within the scrollable area Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove overflow:hidden from inputWrapper that was blocking scroll - Add clearConversionButton style with circular border (matches send button shape but uses subtle neutral colors for discoverability) - Increase columnRight gap to spacingVerticalS for more breathing room between info icon and send button - Add marginRight to convertedBadge matching originalBadge for consistent indentation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Textarea sizing: - Each textarea auto-resizes via JS (height = scrollHeight) and is capped by CSS max-height with overflowY: auto for scrolling - Original textarea: 60vh when solo, 30vh when conversion active (via textInputShared class toggled with mergeClasses) - Converted textarea: 30vh with matching scrollbar styling - Both use identical webkit scrollbar styles - Converted textarea also auto-resizes via its own ref + useLayoutEffect Button styling: - Attach and converter buttons get round border (iconButton) - Clear conversion button has distinct circular style with subtle border - More gap between attach/converter buttons (spacingVerticalS) - More gap in columnRight between info/send/clear (spacingVerticalS) Badge alignment: - convertedBadge now has marginRight matching originalBadge Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

romanlutz and others added 30 commits February 25, 2026 05:49

MAINT Enable ruff UP (pyupgrade) rules and fix all violations (micros…

ac61249

…oft#1389) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable ruff SIM (flake8-simplify) rules and fix all violations (m…

0ab0b3d

…icrosoft#1397) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Regenerate uv.lock for dependabot security fixes (microsoft#1401)

54a4efb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX Mock tokenizer in unit test to avoid HuggingFace network call (mi…

e89ccfe

…crosoft#1402) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX: Ensuring _score_value_with_llm works with reasoning targets (m…

8a85254

…icrosoft#1403)

FEAT Add PromptIntel remote dataset loader (microsoft#1400)

0dbff8d

Co-authored-by: Anandan Sundar <205880232+anasundar_microsoft@users.noreply.github.com>

MAINT Update cryptography>=46.0.5 and werkzeug>=3.1.6 for dependabot …

69a85fa

…alerts (microsoft#1405)

MAINT Enable ruff PIE (flake8-pie) rules and fix all violations (micr…

8d9b7cd

…osoft#1394) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Bump rollup to >=4.59.0 for dependabot security alert (microsof…

c82b1ec

…t#1408) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable ruff B (flake8-bugbear) rules and fix all violations (mi…

a07283d

…crosoft#1390) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable ruff TCH (flake8-type-checking) rules and fix all violat…

7c65439

…ions (microsoft#1395) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT add MacOS CI (microsoft#1407)

358b1b0

FEAT: Generalize Colloquial Wordswap Attack Converter (microsoft#1348)

07fd38b

FIX circular import making pyrit.__version__ unavailable during submo…

1353ad4

…dule init (microsoft#1428) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>

FEAT Add SimpleSafetyTests dataset loader (microsoft#1426)

cae1cda

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Migrate Azure Cognitive Services from API key to Entra ID authe…

6059991

…ntication (microsoft#1404) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT Expand memory interface and models for attack results (microsoft…

dfbfbf6

…#1420) Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT Add SALAD-Bench dataset loader (microsoft#1425)

1f6fb87

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable ruff rule E (pycodestyle errors) (microsoft#1413)

e1353c4

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable ruff PGH rule for pygrep-hooks linting (microsoft#1416)

03156c0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT Add OR-Bench dataset loader (microsoft#1423)

b265532

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Victor Valbuena <50061128+ValbuenaVC@users.noreply.github.com>

MAINT Enable ruff PERF rule for performance linting (microsoft#1415)

9b98e20

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX address deprecation warnings (microsoft#1430)

2658a5a

MAINT Enable ruff rules: DTZ, N, T10, TID, YTT (microsoft#1414)

0f3a83a

FIX replace api key in scenarios (microsoft#1435)

769bb24

MAINT Add pre-commit hook to sanitize user paths in notebook outputs (m…

afa4315

…icrosoft#1429) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT Enable low-hanging ruff rules: A, ERA, EXE, FA, FLY, ICN, ISC, …

3986851

…LOG, PLE, Q, RSE, SLOT (microsoft#1412) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com>

MAINT/FIX: Generalizing eval-hash (microsoft#1434)

17de38a

FIX: Add arm64 Support for devcontainer (microsoft#1409)

bed086e

FEAT Add BeaverTails dataset loader (microsoft#1424)

799e981

behnam-o and others added 29 commits April 30, 2026 02:09

MAINT: Add deprecation warnings when Message Piece is constructed wit…

55fd7bc

…h labels (microsoft#1670) Co-authored-by: Copilot <copilot@github.com>

DOC: Deployment guide for GUI instances (microsoft#1658)

5f38a88

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

FEAT: Deployment automation for isolated GUI instances (microsoft#1655)

40ec403

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

MAINT Migration from mypy to ty (microsoft#1319)

4f4d2e0

Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

TEST add unit tests for build scripts (microsoft#1642)

063ae4f

Co-authored-by: Tejas Saubhage <tsaubhage0007@gmail.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX Surface AttackResultEntry.timestamp on hydrated AttackResult (mic…

9ad5a22

…rosoft#1653) Co-authored-by: thirteeneight <purple.format.2o@icloud.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

FEAT: Add single_turn_crescendo technique with adversarial_config (mi…

5eaa0c3

…crosoft#1665) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT: Lazy Imports for perf (microsoft#1668)

7065339

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT Migrate consumers to TargetConfiguration capability checks (mi…

e635ee9

…crosoft#1645)

MAINT FIX: Fix ReadTheDocs config validation error (microsoft#1673)

96a9510

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT: Deprecating Dall-E specific parameters and aligning OpenAIImag…

fd195bf

…eTarget with GPT Image endpoints (microsoft#1661)

FIX: RTD build - install deps explicitly in build.jobs.install (micro…

0817489

…soft#1676) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT: pyproject update (microsoft#1674)

7e214ea

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FEAT: Add scenario listing API endpoints to backend (microsoft#1666)

c70b7c3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX set editable_history in 2_red_teaming_attack (microsoft#1675)

644b45c

FEAT add crescendo persona scenario techniques (movie director, histo…

585c009

…ry lecture, journalist interview) (microsoft#1677) Signed-off-by: precognitivem0nk <rextedgorman@gmail.com>

MAINT: Simplifying Initializers (microsoft#1679)

2e952a6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MAINT: Add labels to attack results (microsoft#1624)

39031dd

Co-authored-by: Behnam Ousat <behnamousat@microsoft.com> Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com>

FEAT: HuggingFace reproducibility params, OpenAI-compatible integrati…

a7e9c80

…on tests, and endpoint target deprecation (microsoft#1672) Co-authored-by: Roman Lutz <romanlutz@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

FIX Return no memory results for empty prompt ID filters (microsoft#1499

2934127

) Co-authored-by: Behnam Ousat <behnamousat@microsoft.com> Co-authored-by: Roman Lutz <romanlutz13@gmail.com>

romanlutz closed this May 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT: Block GUI sends when target doesn't support the modality#4

FEAT: Block GUI sends when target doesn't support the modality#4
romanlutz wants to merge 587 commits into
mainfrom
gui-modality-check

romanlutz commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

romanlutz commented May 6, 2026

Summary

Changes

Backend

Frontend — Modality validation

Frontend — Input area styling

Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants