feat(agent): connect the MCP spec docs server for protocol questions - #3631
Conversation
The in-app agent could search MCPJam's own docs and the web, but nothing gave it the MCP protocol spec — so protocol questions were answered from training data, which degrades across exactly the version range this debugger targets (2025-11-25, 2026-07-28/draft). modelcontextprotocol.io runs the same Mintlify docs-server product as docs.mcpjam.com: unauthenticated streamable HTTP, with search plus a read-only virtual filesystem over the .mdx sources. `ls /specification` returns 2024-11-05 … 2026-07-28 and draft, so retrieval is ADDRESSED — the model must name a version to read one, and cannot silently answer a 2026-07-28 question out of the 2025-03-26 text. That property is why this beats both a prompt dump and embedding-based retrieval, where four near-identical spec versions are precisely the worst case. Both docs servers also ship a `submit_feedback` WRITE tool, declined here via a new `excludeMcpToolNames` prepare option. Two independent reasons: it posts model-authored free text to a docs team unattended and invisibly (the class of action this surface dropped the platform worker to avoid), and `getToolsForAiSdk` flattens servers last-in-wins on colliding names — so advertising it would have silently routed MCPJam docs feedback to the MCP project. The preflight already degrades per server, which now matters more: one of the two knowledge servers is a third party we don't operate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dosubot
Bot
added
size:L
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_47601eab-5214-45d3-aa22-c70fa07ca3fe) |
📄 Knowledge reviewDosu skipped reviewing this PR because your organization has used its
|
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThe agent now uses separate MCPJam documentation and MCP specification servers with shared unauthenticated configuration. Preflight selects responsive servers and supports degraded operation. MCP protocol prompts require version-specific specification guidance. The Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
mcpjam-inspector/server/routes/web/__tests__/mcpjam-agent.ui-only.test.tsOops! Something went wrong! :( ESLint: 8.57.1 Error: ESLint configuration in --config is invalid:
mcpjam-inspector/server/routes/web/mcpjam-agent.tsOops! Something went wrong! :( ESLint: 8.57.1 Error: ESLint configuration in --config is invalid:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@mcpjam-inspector/server/routes/web/mcpjam-agent.ts`:
- Around line 160-161: The specification and knowledge-server guidance in the
agent prompt must remain active when platform tools are disabled. Separate that
guidance from the UI-only action guidance around AGENT_IDENTITY_PROMPT, include
the specification-routing text in both platform and rollback modes, and add a
rollback-mode test verifying that protocol questions require establishing the
specification version first.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 658376d0-2c12-44fd-8094-1ec69991034b
📒 Files selected for processing (7)
mcpjam-inspector/server/routes/web/__tests__/mcpjam-agent-widget-content.test.tsmcpjam-inspector/server/routes/web/__tests__/mcpjam-agent.ui-only.test.tsmcpjam-inspector/server/routes/web/__tests__/mcpjam-agent.uitools.test.tsmcpjam-inspector/server/routes/web/mcpjam-agent.tsmcpjam-inspector/server/utils/__tests__/chat-v2-orchestration.test.tsmcpjam-inspector/server/utils/chat-v2-orchestration.tsmcpjam-inspector/server/utils/web-chat-turn.ts
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
… prompt Under MCPJAM_AGENT_PLATFORM_TOOLS=1 the spec server was connected but the version-first routing instruction was dropped, because it lived inside AGENT_IDENTITY_PROMPT and the kill-switch removes that whole section. The result was the one combination with the feature's cost and none of its value: the authoritative protocol source attached, and no instruction to read it or to pin a version. The stated principle — the switch governs how the agent ACTS, not what it may READ — was applied to the server but not to its guidance. Extract SPEC_DOCS_PROMPT and gate it on the spec server surviving preflight, the same rule ambientContextPrompt follows, so the two can no longer disagree. Reported by cubic on #3631. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub Actions
Error
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_9b9f959f-e00a-4b12-a2b8-1cd9a1b74691) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1d9d1a61f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ? [DOCS_SERVER_ID, PLATFORM_SERVER_ID] | ||
| : [DOCS_SERVER_ID]; | ||
| ? [DOCS_SERVER_ID, SPEC_SERVER_ID, PLATFORM_SERVER_ID] | ||
| : [DOCS_SERVER_ID, SPEC_SERVER_ID]; |
There was a problem hiding this comment.
When the third-party spec server accepts a connection but stalls on tools/list, every agent request—including UI or MCPJam product questions that do not need protocol docs—waits for this preflight because Promise.allSettled blocks on all candidates. The shared config permits that request to run for up to 30 seconds, so the advertised degradation to MCPJam docs and web_search only happens after a long user-visible pause; use a short independent deadline or defer the spec connection until it is needed.
Useful? React with 👍 / 👎.
1 participant
The gap
The in-app agent could search MCPJam's own docs (
docs.mcpjam.com/mcp) and the web, but nothing gave it the MCP protocol spec. So "what doestoolCalledWithmatch?" was answered from docs, and "doestools/listsupport cursor in 2025-11-25?" was answered from training data — which degrades across exactly the version range this debugger targets.Why a second docs server (and not a prompt dump or RAG)
modelcontextprotocol.io/mcpruns the same Mintlify docs-server product as ours: unauthenticated streamable HTTP, stateless, so it drops into the existing config builder unchanged. It exposes search plus a read-only virtual filesystem over the.mdxsources, and:That makes retrieval addressed, not similarity-based: the model has to name a version to read one, so it cannot silently answer a 2026-07-28 question out of the 2025-03-26 text. Four near-identical spec versions are the worst possible case for embedding retrieval and the best case for path addressing — which is also why this beats a pinned in-repo spec skill (build-static, needs a deploy to correct) and a prompt dump (paid every turn by every user, including the ones asking where the Connect button is).
Verified live through our own SDK, with our client capabilities:
The
submit_feedbackproblemThat third tool is a write (
readOnlyHint: false, free-textfeedbackstring), and both Mintlify docs servers ship it under the same unqualified name. Two independent problems, either sufficient on its own:getToolsForAiSdkflattens selected servers last-in-wins on name collisions, so advertising it would have silently routed MCPJam docs feedback to the MCP project — with nothing at the call site to reveal it.Fixed with a new
excludeMcpToolNamesoption onprepareChatV2/streamWebChatTurn, applied right after the SEP-1865 visibility filter. It is the mirror image ofrespectToolVisibility: that honors a policy the server declares, this is the host declining a tool the server is happy to offer. No default — a surface that wants nothing filtered omits it. Deleting by name resolves the collision too: neither copy survives.Note this write tool was already reachable via our own docs server before this PR; the collision is what's new, and the fix covers both.
Other decisions
Promise.allSettleddegrade-per-server path. This matters more now — one of the two knowledge servers is a third party we don't operate, andgetToolsForAiSdkfails the whole turn if any selected server errors at connect/list time. Covered by a new test.accessToken, asserted in a test. Forwarding the caller's AuthKit bearer to a third party for a public docs search would be a leak.MCPJAM_AGENT_PLATFORM_TOOLS=1restores the old action contract. It does not gate knowledge sources — the switch governs how the agent acts, not what it may read — so both docs servers connect in either mode, rather than giving the config two shapes. Docstring updated to say so, with a test.MCPJAM_SPEC_MCP_URLoverrides the URL, matchingMCPJAM_DOCS_MCP_URL.Testing
vitest run --project server— 303 files, 4068 tests, all passing (both shared files are widely consumed, so the full project was run, not just the touched suites).submit_feedbackdeclined, knowledge servers present under the kill-switch, version-first prompt rules, and threeexcludeMcpToolNamesunit tests inchat-v2-orchestration.test.ts(declines, covers collisions, no-op when omitted).logger.warnwarning, untouched).Follow-up (not in this PR)
The planned Slack app hits the same gap — its
/api/v1/.../agentendpoint should connect these same two knowledge servers, and can reuseexcludeMcpToolNamesverbatim.🤖 Generated with Claude Code
Note
Medium Risk
Changes agent MCP wiring, system prompts, and shared chat orchestration; behavior is well-tested but affects every agent turn and tool advertisement when both docs servers are selected.
Overview
Adds a second read-only knowledge MCP server (
mcp-spec→https://modelcontextprotocol.io/mcp) alongside MCPJam docs so the in-app agent can answer protocol questions from versioned spec sources instead of training data. Preflight now considers both servers and degrades per server (a spec outage drops only that server and its prompt section, not the whole turn).Introduces
excludeMcpToolNamesonprepareChatV2/streamWebChatTurnso hosts can strip MCP tools by name after visibility filtering; the agent uses it to declinesubmit_feedbackfrom both Mintlify docs servers (write tool + duplicate name collision when servers are flattened).New
SPEC_DOCS_PROMPTsteers protocol Q&A to the spec server with version pinning; it stays enabled underMCPJAM_AGENT_PLATFORM_TOOLS=1(read path) but is omitted when spec preflight fails. Docs config is refactored through a shared unauthenticatedbuildDocsServerConfig; optionalMCPJAM_SPEC_MCP_URLoverride matches the docs URL env pattern.Reviewed by Cursor Bugbot for commit d1d9d1a. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Connect the MCP spec docs server so the agent can answer protocol questions by version instead of from memory. Add a host-side filter to drop
submit_feedback, preflight each server to degrade instead of fail, and emit spec guidance with the server so it survives the kill-switch.New Features
https://modelcontextprotocol.io/mcpasmcp-spec(unauthenticated)./specification/<version>.excludeMcpToolNamesto dropsubmit_feedbackacross servers; applied inprepareChatV2andstreamWebChatTurn.web_searchif a docs server is down.MCPJAM_AGENT_PLATFORM_TOOLS=1restores actions only; docs servers stay connected.MCPJAM_SPEC_MCP_URLadded to override the spec URL.Bug Fixes
mcp-specpreflight and emitted independently of the identity prompt, so it remains active underMCPJAM_AGENT_PLATFORM_TOOLS=1.Written for commit d1d9d1a. Summary will update on new commits.