MCP client should scan tool responses for prompt injection attackes

[pelikhan]

A MCP tool may return a string that contains a prompt injection attack. If the prompt injection is successful, the behavior of the client LLM is undefined and/or compromised.

This can also happen from a trusted server, like microsoft/playwright-mcp, if that server gets compromised by the prompt injection. For example, a web browsing MCP server using playwright could read a page on internet instrumented with prompt injection and pass it on through the tool result.

Recommendation

Run all tool output through an AI prompt detection filter.

Example of such protection is implemented in genaiscript - https://microsoft.github.io/genaiscript/blog/mcp-tool-validation/

[connor4312]

This is a nice idea, I think practicality will vary. Every tool round will be slower due to filtering logic and we increase cost to users of each tool call round; people are very sensitive to LLM latency and we put a ton of effort into improving it. But I could see it as a good opt-in feature especially for enterprise.

[pelikhan]

This could/should be an optin. For example, a tool downloading data from internet is subject to prompt injection. However, a tool doing some computation on a network-isolated VM is safe.

[digitarald]

I think the first step is having a benchmark to show GPT-as-a-judge actually works and can't be easily jailbroken.

[pelikhan]

Tool output should be sanitized through the Azure AI Content Safety filter before the llm-as-judge stage.