Skip to content

Releases: SMWundefined/RateMyAgent

v0.1.6 — mutating-tool gate, readable latency, honest verdict

Choose a tag to compare

@SMWundefined SMWundefined released this 06 Sep 23:15

Behavioural change. passed now requires score >= pass_score AND no failed check. A scan that reported PASS at 0.1.5 can report FAIL at 0.1.6 with an unchanged score. Weights, thresholds and the 0-100 scale did not move.

Mutating-tool gate. Auto-selection previously picked the first tool in the discovery list and invoked it for real. Against a knowledge-graph server that meant create_entities, called under fault injection, with only a warning. Auto-selection now requires readOnlyHint=True or a read-only name pattern; UNKNOWN refuses alongside MUTATING. Explicit --tool on a mutating tool requires --allow-mutating. destructiveHint is recorded but not gated on.

Readable sub-second latency. p95 0.00s was a real 0.73ms measurement formatted into something indistinguishable from a missing value, including one comparison that rendered as p95 rose to 0.00s from 0.00s. Durations under 10ms now render in milliseconds.

Provenance in artifacts. Reports and AGENTS.md state blocks now record Probe tool and Probe arguments, so a saved scan says what produced its numbers.

Known limitations: --uri supports stdio:// and sse:// only; Streamable HTTP is not yet supported. Argument synthesis can produce semantically empty calls against tools with required array or object fields. Caller-strategy behaviour metrics are not meaningful against bare server targets.

665 tests.

v0.1.5 — documentation fixes

Choose a tag to compare

@SMWundefined SMWundefined released this 06 Sep 19:25

Documentation release. No behavior changes. Fixes broken repro script under mcp 2.x, removes stale retraction prose, ships examples in sdist.

v0.1.4 — crash detection and mcp SDK 2.x compatibility

Choose a tag to compare

@SMWundefined SMWundefined released this 06 Sep 18:51

Retracted finding: Earlier versions reported a 33-50% contract crash rate against mcp-server-git and @modelcontextprotocol/server-filesystem, filed upstream as modelcontextprotocol/servers#4754. That finding was wrong and has been retracted — neither server crashes. It was a bug in this scanner's error classifier, fixed below.

Fix 1 — crash detection no longer reads error message text. Response.delivered is now the crash signal: false only when a response was built from a raised exception. Rejections we cannot classify are reported as unclassified rather than scored as crashes.

Fix 2 — mcp SDK 2.x compatibility. The SDK renamed isError, inputSchema and serverInfo in 2.0, and every read used getattr with a default — so a 2.x install reported every tool error as a success. Both majors now work, with a regression test per major. Dependency capped at mcp>=1.0.0,<3.

610 tests. All nine previously scanned servers re-run side by side. Three scores changed — the two wrongly accused servers.

v0.1.3 — terminal colors, CI, PyPI metadata

Choose a tag to compare

@SMWundefined SMWundefined released this 04 Sep 23:12

Terminal scorecard now has color-coded pass/fail, byline, and severity markers. Added GitHub Actions CI on Python 3.10-3.13. Fixed PyPI project URLs.

v0.1.2 — MCP compatibility fixes

Choose a tag to compare

@SMWundefined SMWundefined released this 04 Sep 22:23

Fixes two adapter bugs found scanning real MCP servers:

  • Error payloads returned inside successful responses (FastMCP-style servers) are now counted as failures. Previously a scan where every call was rejected reported a 0% error rate.
  • The initialize handshake is now best-effort, so servers on MCP's stateless core (spec 2026-07-28) can be scanned.