Should the results after integrating svelte mcp be included in the ranking?

[muqiuhan]

• https://svelte.dev/docs/mcp/overview

The Svelte MCP (Model Context Protocol) server can help your LLM or agent of choice write better Svelte code. It works by providing documentation relevant to the task at hand, and statically analysing generated code so that it can suggest fixes and best practices.

[maxffarrell]

@khromov I haven't played with MCP much, should we start testing models with MCP and show them alongside existing "v2" results?

[khromov]

@maxffarrell I've been thinking about this for a while and I think we need a separate leaderboard for MCP-enabled models, because MCP makes a huge difference, Paolo has been testing it out and seeing big improvements with even local models like gpt-oss-20b: https://bsky.app/profile/paolo.ricciuti.me/post/3lzluxcsoi224

There's a few things that need to be in place though:

1. Figure out how to add the https://mcp.svelte.dev/mcp or npx -y @sveltejs/mcp (remote/local) MCP for each provider
2. Modify the default prompt for instructions to include the Svelte MCP prompt: https://svelte.dev/docs/mcp/overview#Usage

For OpenRouter specifically, while they do support MCP servers I suspect it's a little bit more complicated because different models are trained differently on how to do tool calls, so we need to read up to see what sort of magic OpenRouter does - do they "normalize" the tool calls, do they inject their own tool calling conventions, etc. I think this can make a big difference in results quality.

[maxffarrell]

@khromov seems like AI SDK supports MCP in some capacity

https://ai-sdk.dev/cookbook/node/mcp-tools

I started a branch to experiment with a migration for maintainability and making it easier to add new providers, especially after I noticed Kimi-k2-turbo was only on Moonshots api. Seems to me like ai sdk might be the best way forward but it hinges on how well their mcp implementation works, would love to hear your thoughts

[paoloricciuti]

The MCP integration of AI SDK is not super-duper great in general, but it should work for our use case. They only support tools but given we need to do it programmatically that's all we really need so it could be a good idea.

[khromov]

Work in progress PR:
#52