mcpgrade
mcpgrade is a command-line tool that grades Model Context Protocol (MCP) servers on how usable they are for an AI agent, not just whether they conform to the protocol specification. It is described by its author as "Lighthouse for MCP servers", echoing the web-performance auditing tool Lighthouse, but scoring agent usability instead of page speed.
Most existing MCP quality tools check whether a server is well-formed: does it parse across clients, does it implement the spec correctly. mcpgrade instead asks whether a language model can actually use the server's tools correctly. It grades a server across five weighted categories: descriptions (30%, missing or vague documentation), schema design (30%, type definitions and structure), naming (15%, confusing or generic tool names), token cost (15%, schema efficiency), and consistency (10%, uniform patterns across tools). Each finding comes with a concrete fix suggestion, and the tool outputs a letter grade (A-F) alongside severity-tagged findings. It runs via npx against a live server URL, a stdio process, or a captured tool snapshot, with --json and --fail-on flags for CI use.
The scoring methodology is documented separately in the project's evaluation calibration notes. Rather than static analysis alone, mcpgrade runs live model tests against Claude Haiku, measuring three capabilities: tool selection from a catalog, argument validity, and refusal accuracy on out-of-scope requests. An early version of the methodology produced misleadingly low scores (38-54%) because synthetic test tasks omitted parameter values that a real agent would have received from prior tool calls in a multi-step flow. The team fixed this by embedding concrete parameter values into every synthetic task so single-step selection is evaluated fairly. After the fix, a validation round reached 100% selection accuracy on a clean server, and a further round showed the eval could still discriminate meaningfully: servers with naming collisions and ambiguous tool catalogs produced predictable confusion, consistent with what static linting rules also catch. The live-model approach is cheap: three servers cost about $0.12 to test on Haiku, with a projected $1.50-$2.00 for a 36-server sweep.
The project's comparison document positions mcpgrade against four other MCP quality tools: mcp-lint (schema parsing across clients), mcp-compliance (spec conformance), MCP Inspector (exploratory debugging UI), and mcp-observatory (overall server health and security). Its stated distinction from mcp-lint is that a server can pass one and fail the other in either direction: mcp-lint asks whether every client can parse the server, mcpgrade asks whether a model can use it. The document's example is a deliberately broken tool that scored B/80 on mcp-lint but F/37 on mcpgrade, due to generic naming, missing schemas, and confusing duplicate descriptions likely to cause model errors in production. The authors recommend running both before deployment. Results across servers are published on a public leaderboard.
The repository was created on 2026-07-18 and had 19 stars and 7 open issues as of this review. The open issues point to a methodology still settling: a proposal to replace binary selection accuracy with a four-outcome taxonomy, a plan for held-out task authoring so synthetic tasks cannot flatter the schemas that generated them, a request to pin the full environment fingerprint in every eval result, and an unresolved question about correlating scores with production north-star metrics. This puts mcpgrade at the early end of Assess: the usability-grading idea is distinct and useful alongside spec linters, but the eval design is not yet settled and there is no first-person production use to draw on.
