Fixed
tools_description_present_in_allnow catches absent descriptions.
The old predicate compared against the empty string only, so a tool with
no description at all (or a whitespace-only one) passed the HIGH rule
that exists to catch exactly that — the most common real-world defect
earned the points. The rule now uses the same predicate as its
prompt/resource siblings (absent, empty, and whitespace all count as
missing). Scores move for servers whose tools omit descriptions; both
regression tests were proven to fail against the old implementation.
Changed
-
MCP Python SDK bumped to
mcp==2.1.1(from 2.0.0). Two client-side
SDK fixes ride along and can move scores on affected live servers by
turning listing failures into real audits: pre-2026
sessions now drop later-revision cache-hint fields instead of failing
list_tools(), and boolean sub-schemas in toolpropertiesnow parse.
Servers whose listings previously died inside the SDK go from
insufficient-data skips to actually being judged. -
The "Tools Quality" category is now "Primitives" in the README and
docs — the MCP specification's own collective noun for tools, prompts,
resources, and resource templates, which had outgrown the old name (tools
are 18 of the category's 50 rules). Engine reports are unaffected: the
category taxonomy lives in mcpscore.dev's report surface, andrule_ids,
group_names, and scores are unchanged.
Added
-
Smoke mode:
--smokeinvokes your own server's tools after the audit.
The audit itself still never callstools/call; smoke mode is the separate,
opt-in surface for developers testing a server they operate (typically in
CI). It verifies what listing alone cannot: that a declaredoutputSchema
is honored by realstructuredContent(per MCP 2025-11-25 Tools §Output
Schema, a MUST — an explicit empty{}schema still requires the field to
be present), that schema-invalid arguments are rejected with anisError
result or a protocol error, and that unknown tool names get a JSON-RPC
error rather than a result. For both rejection checks, any error code
counts as a rejection except a hang, a closed connection, or-32603
internal error — those are crashes, not rejections.
The probed unknown name is derived deterministically to be
provably absent from the server's own catalog; when the tool listing is
missing or incomplete, that check skips rather than risk invoking a real
tool. By default only tools annotated
readOnlyHint: trueare called;--call-allis the explicit consent for
the rest — so unannotated tools visibly cost smoke coverage. Argument
synthesis is deterministic (defaults / const / examples / enum-first / type
zero-values — never an LLM) and tolerates broken schemas: malformed
non-stringrequiredentries are ignored rather than aborting the smoke
run. Verdicts are pass/fail/skip, and a tool whose
upstream is down reports as skip: someone else's outage should not fail
the build. Results land in a newsmokesection of the--jsonreport and
never touchscore/max_score; any smoke failure exits with the new code
4 (a failed--fail-undergate still takes precedence with 3). The
web service deliberately has no smoke mode — CLI (and the GitHub Action,
in a coming release) only. -
Modern-only stdio servers are now fully auditable. When a local server
rejects the legacyinitializehandshake but answers MCP 2026-07-28
stateless requests, mcpscore retains its resolved launch command, probes it
over fresh stdio processes, reconstructs the catalog and server metadata,
and emits a normal full audit withera: moderninstead of exiting 2. A
server that neither handshakes nor demonstrates modern support still exits
2, preserving the CLI contract. This works for direct.py/.jstargets
and arbitrary-language--stdiocommands. -
Three catalog connection-independence rules. Modern servers are checked
that the complete tools, resources, and prompts identity sets do not vary
across two independent connections carrying the same authorization context.
Comparisons ignore ordering, traverse pagination completely, bound reported
differences, and skip rather than fail when either observation is incomplete.
Each requirement is HIGH and scoped to MCP 2026-07-28, where catalog sets
MUST NOT vary per connection.