[mcp-analysis] MCP Structural Analysis - 2026-09-30

[github-actions[bot]]

Analyzed 10 representative GitHub MCP tools across all available toolsets on 2026-09-30. Average usefulness held steady at 3.0/5; the standout finding is that get_file_contents swung from rating 4 (small file) to rating 1 (README.md, ~12.6K tokens, no truncation/pagination support) purely based on which file was requested — a reminder that response quality for this tool is caller-controlled, not tool-controlled. search_code again failed to GitHub API rate limiting (429), continuing a recurring reliability gap. The persistent integrity-policy filtering on list_issues (21/21 days) continues to silently hide the one open issue in every measured run.

Metric	Value
Tools Analyzed	10
Total Tokens (Today)	18,765
Average Usefulness Rating	3.0/5
Best Rated Tool	github_context: 5/5
Worst Rated Tool	search_code: 1/5 (tied with get_file_contents: 1/5)

Full Structural Analysis Report

Usefulness Ratings for Agentic Work

Tool	Toolset	Rating	Assessment
github_context	workflow_context	⭐⭐⭐⭐⭐	Excellent — zero-cost injected identity block
list_workflows (mcpscripts wrapper)	actions	⭐⭐⭐⭐⭐	Excellent — clean JSON, explicit pagination, no bloat
list_pull_requests	pull_requests	⭐⭐⭐⭐	Good — flat, minimal, directly actionable
get_label	labels	⭐⭐⭐	Adequate — tiny payload but icon _meta overhead dominates
list_code_scanning_alerts	code_security	⭐⭐⭐	Adequate — useful but repeated rule.help boilerplate bloats payload
list_discussions	discussions	⭐⭐⭐	Adequate — good pagination metadata, icon overhead persists
search_users	users	⭐⭐⭐	Adequate — minimal identity fields only
list_issues	issues	⭐⭐	Limited — integrity-filtered issue makes array falsely appear empty
search_code	search	⭐	Poor — failed with 429 rate limit, zero usable data
get_file_contents	repos	⭐	Poor (this run) — README.md returned as a single 50K-char line, spilled to a side file near the 25K-token ceiling

Schema Analysis

Tool	Type	Depth	Key Fields	Notes
github_context	markdown	1	actor, repository, workspace, workflow-run-id	Prompt-injected, no MCP call
list_workflows	object	2	total_count, workflows[].id/name/path/state	Best-in-class wrapper schema
list_pull_requests	array	1	number, title, state, html_url	Flat, no nested overhead
get_label	object	1	name, color, description, id	Content trivial vs. fixed icon overhead
list_code_scanning_alerts	array	3	number, rule.id, rule.help, most_recent_instance.location	Deep nesting, verbatim-repeated help text per rule
list_discussions	object	3	discussions[].category.name, pageInfo, totalCount	Nested category/user relationships
search_users	object	2	items[].login/id/profile_url, total_count	No name/bio, needs follow-up call
list_issues	array	2	number, title, state, labels	Empty due to integrity filter, not truly empty
search_code	error	1	error_message	429 rate limit envelope
get_file_contents	error (oversized)	1	content_text	No size guard; single unbroken line

Response Size Analysis (30-day average)

Toolset	Avg Tokens	Tools Tested
code_security	7,761	18
repos	3,357	18
labels	744	18
discussions	709	18
search	637	18
users	636	18
actions	169	18
issues	94	18
pull_requests	90	18
workflow_context	85	18

Tool-by-Tool Analysis (Today)

Tool	Toolset	Tokens	Schema	Rating	Notes
github_context	workflow_context	90	markdown, depth 1	5	Zero-cost identity, consistent 21/21 days
list_workflows	actions	172	object, depth 2	5	total_count now 378 (up from 377)
list_pull_requests	pull_requests	45	array, depth 1	4	Single open PR #64473 (arXiv researcher ledger update)
get_label	labels	614	object, depth 1	3	bug label, 4 real fields under icon overhead
list_code_scanning_alerts	code_security	3,150	array, depth 3	3	3x go/allocation-size-overflow (#943-945), same family since 09-26
list_discussions	discussions	720	object, depth 3	3	totalCount 11,052 (up from 11,026)
search_users	users	644	object, depth 2	3	pelikhan, minimal identity fields
list_issues	issues	70	array, depth 2	2	Empty array; #64474 integrity-filtered
search_code	search	660	error, depth 1	1	429 rate limit
get_file_contents	repos	12,600	error (oversized), depth 1	1	README.md near 25K-token ceiling, no truncation support

30-Day Trend Summary

Metric	Value
Data Points	180
Average Daily Tokens	14,281
Average Rating Trend	Stable — code_security and repos remain the two heaviest toolsets by a wide margin; list_workflows and list_pull_requests remain consistently lean and high-rated

Recommendations

• High-value tools (rating 4-5): github_context (free), list_workflows via mcpscripts wrapper, list_pull_requests with fields filtering — prefer these patterns whenever available.
• Tools needing improvement: list_issues (integrity-filtered results silently look like "no issues" — agents should check for filtered-item warnings before concluding an empty result set is real); search_code (rate-limit fragility warrants retry/backoff logic); list_code_scanning_alerts (server-side dedup of repeated rule.help text across same-rule alerts would meaningfully cut payload size).
• Context-efficient tools (low tokens, high rating): github_context, list_workflows, list_pull_requests — all under 200 tokens with ratings ≥4.
• Context-heavy tools (high tokens): code_security (7,761 avg) and repos (3,357 avg, but caller-dependent — a targeted small file like CODEOWNERS runs ~700 tokens vs. README.md's 12,600+). Always check file size via a directory listing before calling get_file_contents on an unknown file.

Visualizations

Response Size by Toolset

Usefulness Ratings

Daily Token Trend

Size vs Usefulness

Generated by 🔍 GitHub MCP Structural Analysis · claude · agent · 101.8 AIC · ⌖ 9.22 AIC · ⊞ 7.7K · ◷

• expires on Oct 1, 2026, 4:02 AM UTC-08:00