Repository navigation
Context Window & Token Overhead Benchmark: REST Tool Explosion vs. Schema-First GraphQL Approach #3891
letoribo
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi team,
While integrating
github-mcp-server(v2.2.0) into real-world agentic workflows (Claude Desktop), I ran an empirical transport-level benchmark comparing the official REST-based MCP server against a schema-first GraphQL MCP approach (@letoribo/mcp-graphql-enhanced).Using a custom
stdioJSON-RPC interceptor paired with@anthropic-ai/tokenizer, I measured exact token consumption across initialization, static schema injection, and multi-turn live queries.The results highlight a critical bottleneck in static REST tool definitions and show why the efficiency gap compounds dramatically as conversations grow longer.
For transparency, the complete Claude Desktop session used for validation is available here:
Shared chat:
https://claude.ai/share/ffb7b29e-3f65-435b-a19e-6b1d25681817
1. Static Overhead (
tools/list& Initialization)On connection startup, the official server injects 46 individual tool definitions into the model's context window.
tools/listPayload Sizetools/listToken Costinit+ lists)2. Live Multi-Turn Execution Benchmark
In a single Claude Desktop session, I executed two consecutive queries comparing repository metadata and recent issues for:
letoribo/mcp-graphql-enhancedgithub/github-mcp-server(fetching repository metadata plus the 10 most recently updated issues).
github/github-mcp-server+ 10 IssuesWhy the Efficiency Gap Grows Over Long Conversations
1. Zero Re-Introspection
After the initial schema synchronization is cached, every subsequent GraphQL request sends only a compact query document and receives only the exact fields requested.
2. Compounding Context Tax
In multi-turn LLM sessions, the static tool catalog remains part of the model's working context.
With the official server, ~60K tokens of tool metadata must be carried alongside normal reasoning. With a two-tool GraphQL interface, that recurring overhead is effectively eliminated.
The longer the conversation becomes, the wider the efficiency advantage grows in favor of the schema-first approach.
3. Fewer Round-Trips
Fetching repository metadata together with recent issues requires multiple REST-oriented tool invocations.
GraphQL resolves the same information through a single query document and a single response.
3. Data Accuracy & REST Legacy Artifacts
Side-by-side comparison on
github/github-mcp-serveralso surfaced a number of well-known REST v3 artifacts that can mislead LLM agents.Inflated Watcher Counts
search_repositoriesreturned:which mirrors
stargazers_count.GraphQL's:
returned:
which reflects actual repository watchers/subscribers.
Issues vs. Pull Requests Conflation
REST returned:
combining open issues and open pull requests.
GraphQL separately exposed:
making the distinction explicit.
Inconsistent Field Presence
REST omitted comment counters when the value was zero and returned substantial URL-template and permission boilerplate.
GraphQL returned deterministic field values and only the requested fields.
Relation to Issue #2437
I noticed there is already an open feature request:
The benchmark suggests that a schema-first or hybrid GraphQL execution model offers benefits beyond developer convenience:
Takeaway
The benchmark suggests that the primary scalability bottleneck is not query execution but tool surface area.
A schema-first GraphQL model compresses capability discovery into a small set of generic primitives while preserving access to the full GitHub capability surface through introspection and typed queries.
I'm happy to share raw
stdiotraces, benchmark scripts, and measurement methodology if the data would be useful for further discussion.All reactions