How mcp response are handled by models with base64 image

[menardorama]

Hi
I am running a FLux2 dev on a sglang diffusion instance. I have vibe coded a simple mcp server exposed as a streamable http on librehat.

From my understanding, things are getting complicated as it is not as standard as it should be.... In Librechat, OpenAI and Anthropic seems to handle images coming from mcp differrently. i got success with openai at least....

The thing is we are selfhosting some models such as kimi k2, qwen coder, gpt oss... and I wanted to have a full on premise agent that don"t use any cloud resources.

So does anyone got such issue or confirm my thought ?

@danny-avila Can you clarify please how is it supposed to work ? I had a look at the puppeteer mcp server and tried to do the same but it seems that due to mcp parser files it's hardcoded to specific providers.

Thanks anyway

[menardorama]

@danny-avila I have digged a lit bit on my issue and from my understanding you enabled image parsing on some cloud providers (I see openai, anthropic, azureai and bedrock).

I am not sure to look at the right place but in main/packages/api/src/mcp/parsers.ts I can see that RECOGNIZED_PROVIDERS don't include custom endpoints.

Is this limitation intentional ? Is there a way to enable in a simple way ?

Tanks a lot