[Idea] Smart MCP Router – Semantic Tool Routing to Reduce Context Bloat in High-MCP Environments

[lucmuss]

💡 Smart MCP Router – Semantic Tool Routing for Nanobot

The Problem

When running many MCP servers simultaneously (20, 30, 40+), every single tool definition gets injected into the LLM's context on every request. In a real-world setup with 38 MCP servers and 220+ registered tools, this means:

• Thousands of tokens wasted just on tool descriptions before the user's actual query is even processed
• The LLM has to evaluate 220+ tools on every single turn to decide which one to call
• Slower responses, higher API costs, worse tool selection accuracy
• Startup time increases significantly (60s+ just for MCP registration)

This is not a RAM or CPU problem — it's a context window pollution problem.

Real-World Numbers (from my setup)

Metric	Value
Active MCP servers	38
Registered tools	220+
Estimated tool-definition tokens per request	~4,000–6,000
MCP startup registration time	~63 seconds
Useful tools per typical request	3–8

~97% of the tool context is irrelevant on any given request.

---

The Idea: Smart MCP Router

Instead of exposing all 220+ tools directly to the LLM, introduce a lightweight Python-based MCP proxy that:

1. Indexes all tool descriptions into a local vector store (file-based, no external DB needed)
2. Exposes only 2 meta-tools to the LLM (smart_route and list_tools)
3. On each request, performs a semantic similarity search to find the top-K most relevant tools
4. Forwards the call to the actual MCP server transparently

The LLM never sees 220 tools. It sees 2. The router handles the rest.

---

Architecture / Flow Diagram

┌──────────────────────────────────────────────────────────┐
│                      USER QUERY                          │
│            "What's the weather in Leipzig?"               │
└──────────────────────┬───────────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────────┐
│                       LLM                                │
│                                                          │
│  Sees only 2 tools:                                      │
│    • smart_route(query, arguments)                       │
│    • list_tools(query)                                   │
│                                                          │
│  Calls: smart_route(                                     │
│    query="weather in Leipzig",                           │
│    arguments={query: "Leipzig weather"}                  │
│  )                                                       │
└──────────────────────┬───────────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────────┐
│               SMART MCP ROUTER (Python)                  │
│                                                          │
│  1. Embed the query string                               │
│  2. Search local vector index (numpy .npz file)          │
│  3. Find top-8 matching tools by cosine similarity       │
│  4. Select best match: brave_web_search (score: 0.91)    │
│  5. Forward call to actual "brave-search" MCP server     │
│  6. Return result to LLM                                 │
└────────┬──────────────────┬──────────────┬───────────────┘
         │                  │              │
         ▼                  ▼              ▼
   ┌───────────┐    ┌────────────┐   ┌──────────┐
   │  brave-   │    │  email-    │   │  caldav   │   ... 38 MCP
   │  search   │    │  mcp       │   │          │   servers running
   │  (stdio)  │    │  (stdio)   │   │ (stdio)  │   in background
   └───────────┘    └────────────┘   └──────────┘

---

Why File-Based Instead of Qdrant/ChromaDB?

For most setups (even with 500+ tools), a full vector database is overkill. The tool index is small:

Tools	Index File Size	Search Time
100	~150 KB	<5ms
220	~350 KB	<10ms
500	~800 KB	<15ms
1000	~1.5 MB	<25ms

A simple numpy .npz file with precomputed embeddings + cosine similarity search is:

• Zero dependencies beyond numpy and sentence-transformers
• No Docker container needed
• No network calls
• No database to maintain
• Loads in <100ms

If you later scale to 1000+ tools across distributed systems, you can swap in Qdrant/ChromaDB with minimal code changes.

---

Implementation Outline

Project Structure

smart-mcp-router/
├── sync_tools.py          # Scans config.json, discovers tools, builds index
├── smart_router_mcp.py    # The actual MCP server (stdio)
├── tool_index.npz         # Auto-generated vector index (numpy)
├── tool_metadata.json     # Auto-generated tool registry
├── requirements.txt       # sentence-transformers, numpy
└── README.md

Step 1: sync_tools.py – Automatic Tool Discovery & Indexing

This script reads your existing config.json, starts each MCP server briefly, asks it for its tool list via the MCP protocol (tools/list), computes embeddings, and saves everything to local files.

#!/usr/bin/env python3
"""
sync_tools.py
Scans all MCP servers from config.json, discovers their tools,
generates embeddings, and saves a local vector index.

Usage: python3 sync_tools.py [--config /path/to/config.json]
Run this once after setup and again whenever you add/remove MCP servers.
"""

import json
import subprocess
import os
import argparse
import numpy as np
from sentence_transformers import SentenceTransformer

DEFAULT_CONFIG = os.path.expanduser(
    "~/.nanobot/config.json"
)
INDEX_DIR = os.path.dirname(os.path.abspath(__file__))
INDEX_FILE = os.path.join(INDEX_DIR, "tool_index.npz")
META_FILE = os.path.join(INDEX_DIR, "tool_metadata.json")
MODEL_NAME = "all-MiniLM-L6-v2"  # 80MB, fast, runs on CPU


def load_config(path: str) -> dict:
    with open(path) as f:
        return json.load(f)


def discover_tools_stdio(name: str, cfg: dict) -> list:
    """Start an MCP server via stdio, send tools/list, parse response."""
    cmd = [cfg["command"]] + cfg.get("args", [])
    env_vars = {**os.environ, **cfg.get("env", {})}

    try:
        proc = subprocess.Popen(
            cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.DEVNULL,
            env=env_vars,
        )
        request = json.dumps({
            "jsonrpc": "2.0",
            "id": 1,
            "method": "tools/list",
            "params": {}
        }) + "\n"

        stdout, _ = proc.communicate(
            input=request.encode(), timeout=20
        )

        for line in stdout.decode().strip().split("\n"):
            try:
                resp = json.loads(line)
                if "result" in resp and "tools" in resp["result"]:
                    return [
                        {
                            "tool_name": t["name"],
                            "description": t.get("description", ""),
                            "mcp_server": name,
                            "input_schema": t.get("inputSchema", {}),
                        }
                        for t in resp["result"]["tools"]
                    ]
            except json.JSONDecodeError:
                continue
    except (subprocess.TimeoutExpired, FileNotFoundError, OSError) as e:
        print(f"  ⚠  {name}: {e}")
        if 'proc' in locals():
            proc.kill()
    return []


def discover_tools_http(name: str, cfg: dict) -> list:
    """For streamableHttp MCP servers, send tools/list via HTTP POST."""
    import urllib.request

    url = cfg["url"]
    headers = cfg.get("headers", {})
    headers["Content-Type"] = "application/json"

    payload = json.dumps({
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/list",
        "params": {}
    }).encode()

    try:
        req = urllib.request.Request(url, data=payload, headers=headers)
        with urllib.request.urlopen(req, timeout=15) as resp:
            data = json.loads(resp.read().decode())
            if "result" in data and "tools" in data["result"]:
                return [
                    {
                        "tool_name": t["name"],
                        "description": t.get("description", ""),
                        "mcp_server": name,
                        "input_schema": t.get("inputSchema", {}),
                    }
                    for t in data["result"]["tools"]
                ]
    except Exception as e:
        print(f"  ⚠  {name} (HTTP): {e}")
    return []


def discover_all_tools(config: dict) -> list:
    servers = config.get("mcpServers", {})
    all_tools = []

    for name, cfg in servers.items():
        print(f"🔍 Scanning: {name}...")
        if cfg.get("type") in ("streamableHttp",):
            tools = discover_tools_http(name, cfg)
        elif "command" in cfg:
            tools = discover_tools_stdio(name, cfg)
        else:
            print(f"  ⚠  {name}: unknown transport, skipping")
            continue

        print(f"   → {len(tools)} tools found")
        all_tools.extend(tools)

    return all_tools


def build_index(tools: list):
    print(f"\n📦 Loading embedding model ({MODEL_NAME})...")
    model = SentenceTransformer(MODEL_NAME)

    texts = [
        f"{t['tool_name']}: {t['description']}" for t in tools
    ]

    print(f"🧮 Computing embeddings for {len(texts)} tools...")
    embeddings = model.encode(texts, show_progress_bar=True)

    # Save numpy index
    np.savez_compressed(
        INDEX_FILE,
        embeddings=embeddings.astype(np.float32),
    )
    print(f"💾 Index saved: {INDEX_FILE} "
          f"({os.path.getsize(INDEX_FILE) / 1024:.0f} KB)")

    # Save metadata
    meta = []
    for t in tools:
        meta.append({
            "tool_name": t["tool_name"],
            "description": t["description"],
            "mcp_server": t["mcp_server"],
            "input_schema": t["input_schema"],
        })

    with open(META_FILE, "w") as f:
        json.dump(meta, f, indent=2, ensure_ascii=False)
    print(f"💾 Metadata saved: {META_FILE}")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--config", default=DEFAULT_CONFIG,
        help="Path to nanobot config.json"
    )
    args = parser.parse_args()

    config = load_config(args.config)
    tools = discover_all_tools(config)

    print(f"\n📊 Total: {len(tools)} tools from "
          f"{len(config.get('mcpServers', {}))} servers")

    if tools:
        build_index(tools)
        print("\n✅ Done! Run smart_router_mcp.py to use the router.")
    else:
        print("\n❌ No tools discovered. Check your config.json.")


if __name__ == "__main__":
    main()

Step 2: smart_router_mcp.py – The Router MCP Server

#!/usr/bin/env python3
"""
smart_router_mcp.py
A lightweight MCP server that acts as a semantic router.
Exposes 2 tools to the LLM, routes to the real MCP servers.

Requires: tool_index.npz and tool_metadata.json (from sync_tools.py)
"""

import json
import sys
import os
import subprocess
import numpy as np
from sentence_transformers import SentenceTransformer

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
INDEX_FILE = os.path.join(SCRIPT_DIR, "tool_index.npz")
META_FILE = os.path.join(SCRIPT_DIR, "tool_metadata.json")
CONFIG_PATH = os.path.expanduser("~/.nanobot/config.json")
MODEL_NAME = "all-MiniLM-L6-v2"
TOP_K = 8

# Load everything at startup
model = SentenceTransformer(MODEL_NAME)
index_data = np.load(INDEX_FILE)
embeddings = index_data["embeddings"]

with open(META_FILE) as f:
    tool_metadata = json.load(f)

with open(CONFIG_PATH) as f:
    nanobot_config = json.load(f)


def cosine_search(query: str, top_k: int = TOP_K) -> list:
    """Find most relevant tools by cosine similarity."""
    q_emb = model.encode(query).astype(np.float32)

    # Cosine similarity = dot product of normalized vectors
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    normed = embeddings / (norms + 1e-10)
    q_norm = q_emb / (np.linalg.norm(q_emb) + 1e-10)

    scores = normed @ q_norm
    top_indices = np.argsort(scores)[::-1][:top_k]

    results = []
    for idx in top_indices:
        t = tool_metadata[idx]
        results.append({
            "tool_name": t["tool_name"],
            "mcp_server": t["mcp_server"],
            "description": t["description"],
            "score": float(scores[idx]),
        })
    return results


def call_backend_mcp(
    mcp_name: str, tool_name: str, arguments: dict
) -> dict:
    """Forward a tool call to the actual MCP server."""
    cfg = nanobot_config["mcpServers"][mcp_name]

    if cfg.get("type") in ("streamableHttp",):
        return call_backend_http(cfg, tool_name, arguments)

    cmd = [cfg["command"]] + cfg.get("args", [])
    env_vars = {**os.environ, **cfg.get("env", {})}

    proc = subprocess.Popen(
        cmd,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.DEVNULL,
        env=env_vars,
    )

    request = json.dumps({
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/call",
        "params": {"name": tool_name, "arguments": arguments},
    }) + "\n"

    stdout, _ = proc.communicate(
        input=request.encode(), timeout=30
    )

    for line in stdout.decode().strip().split("\n"):
        try:
            resp = json.loads(line)
            if "result" in resp:
                return resp["result"]
        except json.JSONDecodeError:
            continue

    return {"content": [{"type": "text", "text": "No response"}]}


def call_backend_http(
    cfg: dict, tool_name: str, arguments: dict
) -> dict:
    """Forward a tool call to an HTTP-based MCP server."""
    import urllib.request

    headers = cfg.get("headers", {})
    headers["Content-Type"] = "application/json"

    payload = json.dumps({
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/call",
        "params": {"name": tool_name, "arguments": arguments},
    }).encode()

    req = urllib.request.Request(
        cfg["url"], data=payload, headers=headers
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.loads(resp.read().decode())
        return data.get("result", {})


def handle(request: dict) -> dict:
    rid = request.get("id")
    method = request.get("method")

    if method == "initialize":
        return {
            "jsonrpc": "2.0", "id": rid,
            "result": {
                "protocolVersion": "2024-11-05",
                "capabilities": {"tools": {"listChanged": False}},
                "serverInfo": {
                    "name": "smart-mcp-router",
                    "version": "0.1.0"
                },
            },
        }

    if method == "tools/list":
        return {
            "jsonrpc": "2.0", "id": rid,
            "result": {
                "tools": [
                    {
                        "name": "smart_route",
                        "description":
                            "Routes your request to the best matching "
                            "MCP tool. Describe what you need in natural "
                            "language. The router finds and calls the "
                            "right tool automatically.",
                        "inputSchema": {
                            "type": "object",
                            "properties": {
                                "query": {
                                    "type": "string",
                                    "description":
                                        "What do you want to do? "
                                        "Natural language.",
                                },
                                "tool_name": {
                                    "type": "string",
                                    "description":
                                        "Optional: specific tool name "
                                        "if you already know it.",
                                },
                                "arguments": {
                                    "type": "object",
                                    "description":
                                        "Arguments for the target tool.",
                                },
                            },
                            "required": ["query"],
                        },
                    },
                    {
                        "name": "list_tools",
                        "description":
                            "Search available tools by description. "
                            "Returns the top matches for your query.",
                        "inputSchema": {
                            "type": "object",
                            "properties": {
                                "query": {
                                    "type": "string",
                                    "description":
                                        "What kind of tool are you "
                                        "looking for?",
                                },
                                "top_k": {
                                    "type": "integer",
                                    "description":
                                        "Number of results (default 8).",
                                },
                            },
                            "required": ["query"],
                        },
                    },
                ]
            },
        }

    if method == "tools/call":
        params = request.get("params", {})
        name = params.get("name")
        args = params.get("arguments", {})

        if name == "list_tools":
            query = args.get("query", "")
            top_k = args.get("top_k", TOP_K)
            results = cosine_search(query, top_k)
            return {
                "jsonrpc": "2.0", "id": rid,
                "result": {
                    "content": [{
                        "type": "text",
                        "text": json.dumps(
                            results, indent=2, ensure_ascii=False
                        ),
                    }]
                },
            }

        if name == "smart_route":
            query = args.get("query", "")
            hint = args.get("tool_name")
            tool_args = args.get("arguments", {})

            candidates = cosine_search(query)

            if hint:
                target = next(
                    (c for c in candidates
                     if c["tool_name"] == hint),
                    next(
                        (t for t in tool_metadata
                         if t["tool_name"] == hint),
                        candidates[0],
                    ),
                )
                if isinstance(target, dict) and "score" not in target:
                    target = {
                        "tool_name": target["tool_name"],
                        "mcp_server": target["mcp_server"],
                    }
            else:
                target = candidates[0]

            result = call_backend_mcp(
                target["mcp_server"],
                target["tool_name"],
                tool_args,
            )

            # Include routing info for transparency
            routing_info = (
                f"[Routed to: {target['mcp_server']}/"
                f"{target['tool_name']}]"
            )

            if "content" in result:
                result["content"].insert(0, {
                    "type": "text",
                    "text": routing_info,
                })

            return {
                "jsonrpc": "2.0", "id": rid,
                "result": result,
            }

    if method == "notifications/initialized":
        return None  # no response needed for notifications

    return {
        "jsonrpc": "2.0", "id": rid,
        "error": {"code": -32601, "message": "Method not found"},
    }


def main():
    for line in sys.stdin:
        line = line.strip()
        if not line:
            continue
        try:
            req = json.loads(line)
            resp = handle(req)
            if resp is not None:
                sys.stdout.write(json.dumps(resp) + "\n")
                sys.stdout.flush()
        except Exception as e:
            err = {
                "jsonrpc": "2.0",
                "id": None,
                "error": {"code": -32603, "message": str(e)},
            }
            sys.stdout.write(json.dumps(err) + "\n")
            sys.stdout.flush()


if __name__ == "__main__":
    main()

Step 3: Integration in config.json

Replace your 38 MCP server entries with a single one:

{
  "mcpServers": {
    "smart-router": {
      "command": "python3",
      "args": ["/path/to/smart-mcp-router/smart_router_mcp.py"],
      "toolTimeout": 35
    }
  }
}

All 38 original servers stay installed and runnable — the router calls them on demand. They just don't get registered with the LLM directly anymore.

requirements.txt

sentence-transformers>=2.2.0
numpy>=1.24.0

---

Before & After Comparison

Metric	Before (Direct)	After (Smart Router)
Tools visible to LLM	220+	2
Token overhead per request	~4,000–6,000	~200
MCP startup registration	~63 seconds	<3 seconds
Tool selection accuracy	LLM guesses from 220	Semantic top-8 match
Adding a new MCP server	Edit config → restart (tools bloat more)	Edit config → python3 sync_tools.py → restart
External dependencies	None	sentence-transformers, numpy (pure Python)
Database required	No	No (file-based .npz)
Index size (220 tools)	N/A	~350 KB

---

Limitations & Future Work

• Cold start: sentence-transformers model load takes ~2-3 seconds on first request (stays cached after)
• Argument mapping: The LLM must still know what arguments to pass; list_tools helps here
• Multi-tool chains: If the LLM needs to call 3 different tools in sequence, it currently makes 3 smart_route calls (works fine, just slightly more verbose)
• Future: Could add a batch_route tool for multi-step workflows
• Future: Could add tool-usage statistics to improve ranking over time

---

Prompt for Coding AI (Copy-Paste Ready)

If you want an AI assistant to implement or extend this, use this prompt:

System Prompt for Implementation:

You are building a "Smart MCP Router" — a Python-based MCP server that acts as a semantic proxy in front of multiple MCP backend servers.

Context:

• MCP (Model Context Protocol) servers expose tools via JSON-RPC over stdio or HTTP
• The problem: Too many tools (220+) bloat the LLM context window
• The solution: A router that indexes all tools as embeddings and exposes only 2 meta-tools (smart_route and list_tools)

Architecture:

1. sync_tools.py reads config.json, starts each MCP server, calls tools/list, computes embeddings with all-MiniLM-L6-v2, saves to tool_index.npz + tool_metadata.json
2. smart_router_mcp.py is itself an MCP server (stdio). On tools/list it returns only 2 tools. On smart_route calls, it embeds the query, does cosine similarity search against the index, picks the best matching real tool, forwards the call to the real MCP server, and returns the result.

Requirements:

• Python 3.10+
• Only dependencies: sentence-transformers, numpy
• No external database (file-based vector index)
• Must handle both stdio and streamableHttp MCP backends
• Must be transparent: include [Routed to: server/tool] in responses
• Must support --config flag for custom config paths

Your task: Review the existing code, find bugs, add error handling, add unit tests, and make it production-ready. Focus on reliability — this will run 24/7 as a personal assistant backend.

---

How to Test

# 1. Install dependencies
pip install sentence-transformers numpy

# 2. Build the index
python3 sync_tools.py --config /path/to/config.json

# 3. Test the search manually
python3 -c "
from smart_router_mcp import cosine_search
results = cosine_search('send an email')
for r in results[:5]:
    print(f'{r[\"score\"]:.3f}  {r[\"mcp_server\"]}/{r[\"tool_name\"]}')
"

# 4. Run as MCP server (Nanobot will do this automatically)
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | \
  python3 smart_router_mcp.py

---

Looking for feedback on:

1. Does this approach make sense for your setup?
2. Any edge cases I'm missing?
3. Interest in making this a standalone package (pip install smart-mcp-router)?

Would love to hear from anyone running 10+ MCP servers — what's your experience with tool context bloat?

[Re-bin]

This idea is great!

However, I want to know which model will be used for sentence encoding and whether it will be resource-intensive. Can we use a cloud API for embedding instead of local sentence encoding? Additionally, if the semantics of MCP change, how can we quickly re-encode?

[lucmuss]

Great question — let me break it down:

---

Embedding Model

The current implementation uses all-MiniLM-L6-v2 from the sentence-transformers library [5]. Here's why:

Metric	Value
Model size	~80 MB
Embedding dimension	384
Speed (220 tools)	~2-3 seconds on CPU
RAM usage	~200-300 MB during encoding
Quality	Excellent for short texts like tool descriptions

This runs entirely local — no API calls, no tokens burned, no latency. For our use case (encoding short tool descriptions like "brave_web_search: Search the web using Brave") this model is more than sufficient and keeps the whole system self-contained and offline-capable.

Why not a Cloud API for embeddings?

You could use OpenAI's text-embedding-3-small or similar, but:

• It adds an external dependency for a core routing function
• Every re-index would cost API credits (small, but unnecessary)
• Adds network latency to what should be a <5ms local operation
• If the API is down, your entire routing breaks

The whole point of this project is to reduce complexity, not add more moving parts. Local encoding with MiniLM keeps it lightweight and reliable [1].

That said — if someone absolutely wants cloud embeddings (e.g., running on a very constrained device), the architecture supports it. You'd just swap out the model.encode() call in sync_tools.py for an API call. The .npz index format stays the same regardless of where the embeddings come from.

---

Re-encoding when MCP semantics change

This is the key question. Two approaches:

Approach 1: Manual re-sync (simplest)

# After adding/changing any MCP server:
python3 sync_tools.py --config /path/to/config.json
# Done. Takes ~5-10 seconds for 220 tools.

Approach 2: Automated periodic re-sync (set and forget)

This is what I'd recommend for production setups. A simple cron job or systemd timer that re-indexes every 6 hours:

# Cron (inside the container or on host)
0 */6 * * * cd /path/to/smart-mcp-router && python3 sync_tools.py --config /path/to/config.json >> /var/log/mcp-sync.log 2>&1

Or as a lightweight background service in the same Python project:

# auto_sync.py (optional daemon)
import time, subprocess, os

INTERVAL = 6 * 60 * 60  # 6 hours
SCRIPT = os.path.join(os.path.dirname(__file__), "sync_tools.py")
CONFIG = os.path.expanduser("~/.nanobot/config.json")

while True:
    print(f"🔄 Re-syncing tool index...")
    subprocess.run(["python3", SCRIPT, "--config", CONFIG])
    print(f"✅ Next sync in {INTERVAL // 3600}h")
    time.sleep(INTERVAL)

This covers all scenarios:

• New MCP server added → picked up within 6 hours automatically (or run sync_tools.py manually for immediate effect)
• Tool descriptions changed upstream → re-encoded on next cycle
• MCP server removed from config → disappears from index on next sync

The re-encoding is idempotent — it rebuilds the entire index from scratch each time. With 220 tools that takes ~5 seconds on CPU, so there's zero reason to do incremental updates and add complexity.

---

TL;DR

Question	Answer
Which model?	all-MiniLM-L6-v2 (~80MB, local, fast)
Resource-intensive?	No — ~3 sec CPU, ~300MB RAM for indexing
Cloud API possible?	Yes (swap model.encode() for API call), but unnecessary
Re-encoding on changes?	python3 sync_tools.py — takes 5 seconds. Optionally automate via cron every 6h

The design goal is maximum simplicity: two Python files, one numpy index, zero external services [1].

[lucmuss]

I am currently testing this model. MODEL_NAME = "BAAI/bge-small-en-v1.5"

[richardchen874-sys]

This is a very real bottleneck for high-MCP agent setups. Once every tool description is injected on every turn, the cost is no longer just model pricing; it becomes routing waste, slower tool selection, and lower accuracy from context pollution.

A pattern I would test is a two-stage route: a cheap/local semantic pre-selector for tool candidates, then the stronger model only sees the 5-15 tools that are actually plausible for the task. Pairing that with per-run token budgets and provider fallback usually gives a much cleaner cost profile.

I'm building a private-beta OpenAI-compatible API aggregation layer focused on Chinese models such as DeepSeek / Qwen / GLM-style providers, using official-provider keys. If you want to benchmark lower-cost models for this kind of MCP routing workload, I’d be happy to share some test credits.

Curious: in your setup, is the bigger pain the raw token cost, latency, or tool selection quality?

[bmtrnavsky]

Running a similar problem here, but I solved it with a two-piece combo instead of the embedding-router approach:

mcp-lazy-proxy: lazy-loads tool definitions instead of registering everything with the LLM upfront. Tools only get pulled into context when actually needed for a call, rather than dumping all 220+ definitions on every turn. Cuts the per-call token cost dramatically without needing a vector index at all.
mcp-compressor: sits alongside it and compresses/normalizes tool params before they hit your backend MCP servers handles the messy edge cases where models emit malformed or stringified inputs (e.g., I had to patch a bug where my model was emitting nested object-typed params as JSON strings instead of actual objects, which the compressor was rejecting).

Together they get you most of the same win as your semantic router (LLM isn't drowning in tool defs), but the mechanism is "load lazily + normalize on the way through" rather than "embed everything and do cosine similarity search." No sentence-transformers, no vector index file to maintain — just proxying and shape-correction.

Curious how it'd compare on your 220-tool setup — did you benchmark actual selection accuracy (right tool picked) vs just token savings? That's the part I'd want to see hold up as your tool count grows.

[richardchen874-sys]

Lazy loading plus argument normalization addresses two different costs: repeated schema injection and retries caused by malformed tool arguments. It also avoids the stale-index problem that a semantic router has to manage.

For an apples-to-apples comparison, I would measure tool-discovery recall, added round trips, cold-load latency, malformed-call repair rate, and total tokens per successful tool call, not only the initial context reduction. Does mcp-lazy-proxy emit which tool definitions were loaded and when? That trace would make it possible to compare the lazy path against semantic top-k routing on the same 220-tool workload.

[bmtrnavsky]

I have a smaller MCP stack than you, but it has reduced my token burn by thousands per call. I only had 9 servers, and two had over 200 tools. It basically calls lazy load, then lazy load only shows the tools you need. It adds a bit of latency, but if you are paying for tokens on higher-priced models, it is a significant savings. It's worth investigating.