Is your feature request related to a problem? Please describe.
Following #511, I found a potential feature-parity gap with Garak's Simple Assistive Task Linkage (SATA) probe.
SATA masks selected terms in an objective with [MASK] tokens, then presents the masked objective as a simple assistive infill task. I could not find an equivalent SATA or POS-aware mask-and-infill technique in PyRIT's current converters, issues, or pull requests.
PyRIT already has the HarmBench dataset and TaskFramingConverter, so the missing functionality appears to be the masking and word-selection step rather than a new dataset or scorer.
Describe the solution you'd like
Add a reusable text transformation that can:
- select a configurable number of candidate words or phrases;
- replace them with a configurable mask token such as
[MASK];
- compose with
TaskFramingConverter to reproduce the SATA masked-infill technique;
- support deterministic behavior for testing; and
- include unit tests and a short example using an existing dataset such as HarmBench.
One possible implementation would be a part-of-speech word-selection strategy used with SelectiveTextConverter. A dedicated SATA converter is another option if that better fits the project architecture.
I would appreciate maintainer guidance on the preferred API and NLP dependency before implementing this, particularly whether to use an optional POS tagger or keep the selection logic dependency-free.
Describe alternatives you've considered, if relevant
An exact port of Garak's NLTK-based probe would be straightforward, but it would introduce model-data download behavior and be less reusable.
A complete SATA-specific scenario also seems unnecessarily broad because PyRIT already provides the relevant dataset, task-framing, attack, and scoring components.
Additional context
Is your feature request related to a problem? Please describe.
Following #511, I found a potential feature-parity gap with Garak's Simple Assistive Task Linkage (SATA) probe.
SATA masks selected terms in an objective with
[MASK]tokens, then presents the masked objective as a simple assistive infill task. I could not find an equivalent SATA or POS-aware mask-and-infill technique in PyRIT's current converters, issues, or pull requests.PyRIT already has the HarmBench dataset and
TaskFramingConverter, so the missing functionality appears to be the masking and word-selection step rather than a new dataset or scorer.Describe the solution you'd like
Add a reusable text transformation that can:
[MASK];TaskFramingConverterto reproduce the SATA masked-infill technique;One possible implementation would be a part-of-speech word-selection strategy used with
SelectiveTextConverter. A dedicated SATA converter is another option if that better fits the project architecture.I would appreciate maintainer guidance on the preferred API and NLP dependency before implementing this, particularly whether to use an optional POS tagger or keep the selection logic dependency-free.
Describe alternatives you've considered, if relevant
An exact port of Garak's NLTK-based probe would be straightforward, but it would introduce model-data download behavior and be less reusable.
A complete SATA-specific scenario also seems unnecessarily broad because PyRIT already provides the relevant dataset, task-framing, attack, and scoring components.
Additional context