Repository navigation
mcp-slim: Profiling MCP tool-calling token bloat (-50% to -98%) and redacting leaked credentials in agent pipelines #3636
SalyyS1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi DeepEval community,
When evaluating MCP-enabled LLM agents with DeepEval (
ToolCorrectnessMetric,TaskCompletionMetric), we noticed two recurring issues that hurt both agent accuracy and evaluation cost:github,playwright,notion) inject 15,000 to 45,000+ tokens oftools/listschemas on every turn, causing tool selection degradation as distracting tool count grows.$refpointers and stringified arguments ("limit": "10") cause otherwise-correct tool selections on open-weight models to fail schema validation.We built mcp-slim, a zero-dependency Python MCP stdio proxy and CLI token profiler:
Key capabilities
mcp-slim audit -- <server-cmd>): Profiles exact per-tool schema token costs and estimates savings before running an evaluation suite.--dynamic-toolsets,--toon): Collapses 50+ tools intosearch_tools+call_tooland compacts JSON array outputs.--flatten-schemas,--coerce-args): Inlines$ref/$defspointers and coerces stringified arguments.SecretGuard(--redact-secrets): Scrubs API keys and prompt-injection tags in real time.Repository: https://github.com/SalyyS1/mcp-slim
All reactions