FEAT Add adaptive jailbreaking

[romanlutz]

Is your feature request related to a problem? Please describe.

The technique from this repo is not captured in PyRIT yet https://github.com/tml-epfl/llm-adaptive-attacks

Describe the solution you'd like

Investigate the best way to integrate, e.g., as an attack, and propose a high-level plan here. Then, maintainers will provide feedback and help with questions during the implementation phase.

Describe alternatives you've considered, if relevant

Additional context

[donebydan]

I'd be keen to explore this Open Issue! 🙏

[romanlutz]

Looks like endpoints and the corresponding APIs don't provide logprobs anymore nowadays so I think we can pretty much close this. Wdyt @donebydan?

[donebydan]

Happy to close. I don't have permission to do so - could you? 🙏

[romanlutz]

Actually, I found a couple models that still do provide logprobs according to the documentation so maybe still worth doing? We would need someone to investigate a bit on this.

[immu4989]

@romanlutz — taking up the investigation request. Quick survey of logprob exposure across mainstream APIs and open-model serving as of today, plus the relevant PyRIT integration points.

Logprob availability (2026)

Provider	Logprobs?	Notes
OpenAI Chat Completions	Yes	logprobs: true + top_logprobs (1–20) on standard chat models
Anthropic Messages API	No	No logprobs parameter; usage only carries token counts (docs)
Google Gemini generateContent	No	No logprobs / responseLogprobs parameter in the current API reference
Self-served open models (vLLM, TGI, llama.cpp)	Yes	Native; top-k logprobs configurable per-request
Aggregated open-model APIs (Together, Fireworks, OpenRouter)	Yes for most	Pass-through to vLLM/TGI under the hood

Implication for the Andriushchenko-style attack

The technique requires logprob feedback for suffix optimization, so it's no longer universal — but coverage is still meaningful: works against OpenAI's main chat models and against any open model served via vLLM/TGI (locally hosted Llama, Qwen, Mistral, etc. — exactly the targets red-teamers most need to evaluate). It's a partial-coverage attack now, not an all-targets attack.

Relevant PyRIT integration points

A few things I noticed reading the current code:

1. PromptChatTarget is deprecated in v0.14.0 in favor of PromptTarget + TargetConfiguration capabilities (per pyrit/prompt_target/common/prompt_chat_target.py). New work should target the PromptTarget interface.
2. Neither openai_chat_target.py nor openai_completion_target.py currently passes logprobs=True or processes logprob arrays — target-side opt-in would be needed.
3. MessagePiece.prompt_metadata is typed dict[str, Union[str, int]] — too restrictive for logprob arrays. Integration would need either a new field on MessagePiece (e.g., response_logprobs), or a separate carrier model.

One question before going further

Given the partial coverage, is the team open to landing this as an attack class that's explicit about target compatibility — checks for a supports_logprobs capability on the target and raises clearly when not supported — rather than requiring universal support?

If yes, I'd be happy to put up a small spike: minimal target-side opt-in (return_logprobs on OpenAIChatTarget), the MessagePiece extension to carry the data, and a stub attack that consumes it. ~200–300 LOC, additive, easy to review — with credit to @donebydan's [DRAFT] PR #537 where applicable. The Andriushchenko adaptive search itself would be a follow-up once the plumbing is in place.

If the team would rather close this as "not viable enough to warrant the plumbing," also a clean answer — happy to file the closing rationale as a comment for future readers.

[sajisanchu1913-source]

Hi @romanlutz I recently had PR #1831 merged in PyRIT (SALT-NLP MIC dataset loader). I've been reading through this issue and @immu4989's excellent investigation above.
I'd love to take this on. My proposed approach:

1. Add return_logprobs opt-in to OpenAIChatTarget
2. Extend MessagePiece to carry logprob data
3. Implement the adaptive attack class with explicit target compatibility checks
   Happy to coordinate with @immu4989 if they're already working on it, or take it independently if not. Could you assign this to me?

[romanlutz]

We have TargetCapabilities now so this would just be one more! I can assign the item to you.

[romanlutz]

Actually @immu4989 has the first right to pick it up I suppose 🙂 I think it's still relevant for targets that expose the logprobs.