mispick 0.1.0

Find out which of your MCP tools the model mixes up — measured, not guessed.

uvx mispick run --cmd "python -m my_server"

Point mispick at any MCP server. It generates realistic user requests, asks a model to pick a tool for each, and shows a confusion matrix: which tools get chosen for which intent.

It never calls a tool — only initialize and tools/list — so it is safe to point at a server that deletes things. That is enforced in code and by three tests.

What is in this release

• Load tools from a stdio server, Streamable HTTP, a captured snapshot, or every server in a claude_desktop_config.json / .mcp.json at once.
• Measure selection accuracy, per-tool precision and recall, a confusion matrix with (none) and (phantom) rows, argument validity against inputSchema, stability across K runs, over-triggering, and a tool-list token estimate. Every rate carries a Wilson 95% interval — with K=3 bare percentages would overstate what is known.
• Cross-server collisions. Which bare tool names clash between servers, and how often a request for one server's tool is answered by another server's. The MCP spec names this problem and warns serverInfo.name is not unique, so mispick keys on your config label.
• Proven fixes. mispick fix rewrites both descriptions in a confused pair, re-runs the same queries against the rewrite, and accepts it only when an exact McNemar test agrees. Rejected rewrites are shown too. Hasan et al. found LLM-rewritten descriptions regress in 16.67% of cases (arXiv 2602.14878) — an unmeasured rewrite is a one-in-six chance of making things worse.
• Reports: terminal, JSON with a versioned schema, Markdown for PR comments, a self-contained HTML page that opens offline, and an SVG badge. mispick report re-renders a saved run; mispick compare diffs two.
• GitHub Action with a sticky PR comment, base-branch comparison and --fail-under. With no model key it skips with an explanation rather than failing.
• Free by default: a local model through Ollama, no account needed. OpenAI and Anthropic are optional.

Known limitations

Read these before quoting a number.

• Results depend on the model. A score is one model's reading of one tool surface on one day. Every report states the model, N, K and the date.
• mispick fix often accepts nothing, especially on a small local model. In our runs against qwen3.5:4b, two of three proposed rewrites measurably made selection worse. That is the feature working; fix mode is more productive with a larger model.
• The accept path has not yet fired on a live model. It is unit-tested, but no real model has earned an ACCEPTED verdict here yet.
• The default model was chosen from research, not measurement. qwen3.5:4b works; the comparison against qwen3.5:9b, granite4.1:3b and qwen3:8b has not been run.
• It measures one selection turn — not multi-step agent behaviour, tool output quality, or recovery.

Full detail in the README, and the competitive landscape in docs/research.md.