Repository navigation
Design Proposal: Dynamic Runtime Refresh of MCP Tools and Prompts (#458) #1390
Rishankgupta08
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Context
This discussion proposes an approach for implementing the remaining runtime-refresh functionality in #458.
PR #1373 moved MCP tool and prompt discovery from plan construction to operator startup for Java and Python. The remaining gap is that the discovered catalog stays static while the job is running.
Relevant existing components include:
ActionExecutionOperator.open()— initializes runtime resource discovery.JavaMCPResourceDiscoveryandPythonMCPResourceDiscovery— discover tools and prompts.ResourceCache— stores and resolves runtime resources.BaseChatModelSetup— resolves configured prompts and tools for model requests.ToolCallAction— resolves and executes tools returned by the model.One important consideration:
BaseChatModelSetupcurrently resolves its tool list duringopen()and reuses it for subsequent chat calls. Therefore, updating the resource cache alone would not make newly discovered tools available in later model requests.1. Refresh strategy comparison
I see two primary approaches.
Option A: Notification-driven refresh
Use MCP list-change notifications to detect changes in the server's advertised tool and prompt catalogs.
Relevant protocol notifications:
notifications/tools/list_changednotifications/prompts/list_changedThe server advertises
listChangedsupport through its capabilities. The client can listen for supported changes and request the updated catalog.References:
MCP Tools specification
MCP Prompts specification
Flow:
Establish the MCP connection and perform initial discovery.
Register the appropriate change-notification handlers or subscriptions.
When a notification arrives, mark the affected catalog as stale.
Schedule a refresh outside the Flink mailbox thread.
Fetch the latest tools and prompts.
Validate the complete candidate catalog.
Publish the updated catalog atomically.
Advantages:
Low refresh latency.
No repeated list requests when nothing changes.
Natural fit for MCP servers supporting list-change notifications.
Trade-offs:
Depends on server and client SDK notification support.
Notifications can be missed during disconnection or reconnection.
Multiple notifications can arrive in quick succession and need coalescing.
Java and Python SDK APIs may differ.
Option B: On-demand refresh with optional TTL
Maintain a timestamp for the last successful catalog refresh.
Before a model request uses the catalog, check whether the configured TTL has expired. If expired, refresh the catalog before building the next request.
A single-flight mechanism should prevent multiple concurrent requests from triggering duplicate refresh operations.
Advantages:
Does not require persistent notification listeners.
Can support servers that do not advertise list-change notifications.
Easier to reason about and test across Java and Python.
Provides a recovery mechanism after disconnection.
Trade-offs:
Changes may not become visible until the next request.
TTL introduces a bounded period of stale catalog data.
Frequent refreshes can increase MCP server traffic and request latency.
2. Proposed direction: Hybrid refresh mechanism
My initial recommendation is a common refresh abstraction supporting both strategies.
Notifications should trigger an immediate refresh when supported.
TTL-based refresh should provide reconciliation and fallback.
Servers without notification support should still work through on-demand refresh.
Multiple refresh triggers should be coalesced into one operation.
The refresh policy should be configurable, rather than making notification support mandatory.
The exact default TTL and whether notifications should be enabled in the first implementation can be agreed upon before implementation.
3. Runtime catalog and consistency model
I propose introducing an immutable catalog snapshot for each runtime resource scope.
A snapshot would contain:
Tool definitions and their schemas.
Prompt definitions and metadata.
The owning MCP server for each discovered resource.
A catalog generation identifier.
The refresh process should build a candidate snapshot separately from the active snapshot.
Before publishing it, validate:
Duplicate tool names across MCP servers.
Duplicate prompt names across MCP servers.
Conflicts with statically declared resources.
Reserved tool names beginning with
_subagent_.Changes to existing tool schemas and descriptions.
Tools and prompts should retain separate namespaces, consistent with the existing resource model.
If validation or discovery fails, reject the candidate and retain the last known-good snapshot. A partially updated catalog must never become visible.
Once validated, publish the new snapshot atomically.
Additions, removals and definition changes
Addition: Newly discovered tools become available to subsequent model requests, subject to the Agent's existing tool-selection and configuration rules.
Removal: Removed tools are excluded from subsequent model requests.
Definition change: Updated descriptions and input schemas are used in subsequent requests.
Prompt changes: The updated prompt catalog should be used for subsequent prompt resolution. A newly discovered prompt should not be automatically selected unless the Agent's configuration or prompt-selection policy requests it.
An important design question is whether all newly discovered MCP tools should automatically become callable by an Agent, or whether existing explicit tool allow-lists should remain authoritative.
4. In-flight requests and resource lifecycle
A refresh must not invalidate a tool call that was already generated from an earlier catalog.
For example:
Model request A uses catalog generation 1.
The MCP server removes a tool.
The runtime publishes catalog generation 2.
Model request B uses generation 2.
An already-issued tool call from request A must still be handled consistently with generation 1.
I propose retaining old catalog generations while requests or tool executions still reference them.
A request-scoped snapshot reference or generation identifier could be carried through the model/tool-call execution flow.
When no active execution references an old generation, its retired MCP resources can be closed safely.
This also avoids closing a server connection while an in-flight tool call still depends on it.
5. Failure handling and reconnection
Scenario | Proposed behaviour -- | -- Notification received | Mark catalog stale and schedule refresh Multiple notifications | Coalesce into one refresh MCP server temporarily unavailable | Retain last known-good catalog Catalog validation fails | Reject candidate snapshot Duplicate names | Reject conflicting candidate with a clear error Server reconnects | Trigger a full refresh Tool removed during execution | Allow existing execution to finish using its retained generation Repeated refreshes | Retire old resources only after references are released Operator shutdown | Close current and retired MCP resources safelyRefresh failures should be logged and observable through appropriate metrics. Retries should use bounded backoff and should not block the Flink mailbox thread.
6. Proposed Java/Python scope
Both Java and Python runtime paths should follow the same behavioral contract.
However, the notification and connection-management implementations should remain language-specific adapters because their MCP SDK APIs and execution models differ.
Java path:
Integrate notification handling with the Java MCP client.
Coordinate refresh through the runtime resource cache.
Ensure publication is safe with respect to concurrent resource lookups and model requests.
Python path:
Integrate the corresponding notification mechanism where the Python MCP client exposes it.
Preserve the single-threaded mailbox execution contract of the Python resource cache.
Avoid concurrent mutation of the active catalog during tool execution.
Before committing to notification-driven parity, I would verify the exact MCP SDK versions currently pinned by this repository and whether they expose the required client-side notification callbacks. If an SDK does not expose them, TTL-based refresh can provide a compatible fallback.
The goal should be identical runtime behaviour for Java and Python, even if their internal refresh implementations differ.
7. Proposed sequence diagram
All reactions