Skip to content

v1.11.0

Latest

Choose a tag to compare

@a-akimov a-akimov released this 01 Sep 18:52
16dcd4f

Fixed

  • tools_description_present_in_all now catches absent descriptions.
    The old predicate compared against the empty string only, so a tool with
    no description at all (or a whitespace-only one) passed the HIGH rule
    that exists to catch exactly that — the most common real-world defect
    earned the points. The rule now uses the same predicate as its
    prompt/resource siblings (absent, empty, and whitespace all count as
    missing). Scores move for servers whose tools omit descriptions; both
    regression tests were proven to fail against the old implementation.

Changed

  • MCP Python SDK bumped to mcp==2.1.1 (from 2.0.0). Two client-side
    SDK fixes ride along and can move scores on affected live servers by
    turning listing failures into real audits: pre-2026
    sessions now drop later-revision cache-hint fields instead of failing
    list_tools(), and boolean sub-schemas in tool properties now parse.
    Servers whose listings previously died inside the SDK go from
    insufficient-data skips to actually being judged.

  • The "Tools Quality" category is now "Primitives" in the README and
    docs — the MCP specification's own collective noun for tools, prompts,
    resources, and resource templates, which had outgrown the old name (tools
    are 18 of the category's 50 rules). Engine reports are unaffected: the
    category taxonomy lives in mcpscore.dev's report surface, and rule_ids,
    group_names, and scores are unchanged.

Added

  • Smoke mode: --smoke invokes your own server's tools after the audit.
    The audit itself still never calls tools/call; smoke mode is the separate,
    opt-in surface for developers testing a server they operate (typically in
    CI). It verifies what listing alone cannot: that a declared outputSchema
    is honored by real structuredContent (per MCP 2025-11-25 Tools §Output
    Schema, a MUST — an explicit empty {} schema still requires the field to
    be present), that schema-invalid arguments are rejected with an isError
    result or a protocol error, and that unknown tool names get a JSON-RPC
    error rather than a result. For both rejection checks, any error code
    counts as a rejection except a hang, a closed connection, or -32603
    internal error — those are crashes, not rejections.
    The probed unknown name is derived deterministically to be
    provably absent from the server's own catalog; when the tool listing is
    missing or incomplete, that check skips rather than risk invoking a real
    tool. By default only tools annotated
    readOnlyHint: true are called; --call-all is the explicit consent for
    the rest — so unannotated tools visibly cost smoke coverage. Argument
    synthesis is deterministic (defaults / const / examples / enum-first / type
    zero-values — never an LLM) and tolerates broken schemas: malformed
    non-string required entries are ignored rather than aborting the smoke
    run. Verdicts are pass/fail/skip, and a tool whose
    upstream is down reports as skip: someone else's outage should not fail
    the build. Results land in a new smoke section of the --json report and
    never touch score/max_score; any smoke failure exits with the new code
    4 (a failed --fail-under gate still takes precedence with 3). The
    web service deliberately has no smoke mode — CLI (and the GitHub Action,
    in a coming release) only.

  • Modern-only stdio servers are now fully auditable. When a local server
    rejects the legacy initialize handshake but answers MCP 2026-07-28
    stateless requests, mcpscore retains its resolved launch command, probes it
    over fresh stdio processes, reconstructs the catalog and server metadata,
    and emits a normal full audit with era: modern instead of exiting 2. A
    server that neither handshakes nor demonstrates modern support still exits
    2, preserving the CLI contract. This works for direct .py/.js targets
    and arbitrary-language --stdio commands.

  • Three catalog connection-independence rules. Modern servers are checked
    that the complete tools, resources, and prompts identity sets do not vary
    across two independent connections carrying the same authorization context.
    Comparisons ignore ordering, traverse pagination completely, bound reported
    differences, and skip rather than fail when either observation is incomplete.
    Each requirement is HIGH and scoped to MCP 2026-07-28, where catalog sets
    MUST NOT vary per connection.