Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ The Provar DX CLI ships with a built-in **Model Context Protocol (MCP) server**
- [AI loop pattern](#ai-loop-pattern)
- [Quality scores explained](#quality-scores-explained)
- [API compatibility — `xml` vs `xml_content`](#api-compatibility--xml-vs-xml_content)
- [Performance Tuning](#performance-tuning)

---

Expand Down Expand Up @@ -2102,3 +2103,80 @@ provar_nitrox_patch → apply targeted edits to an existing .po.json (RFC 7
```

> **Note:** `provar_automation_*` and `provar_qualityhub_*` tools invoke `sf` CLI subprocesses. The Salesforce CLI must be installed and in `PATH`, or pass `sf_path` pointing to the executable directly (e.g. `~/.nvm/versions/node/v22.0.0/bin/sf`). A missing `sf` binary returns the error code `SF_NOT_FOUND` with an installation hint.

---

## Performance Tuning

These environment variables let you control agentic-loop safety and observability without modifying tool code.

### Agentic loop guard (`PROVAR_MCP_MAX_TOOL_DEPTH`)

Limits the number of Provar tool calls an AI agent may make within a single MCP session before the server starts returning errors instead of results.

```
PROVAR_MCP_MAX_TOOL_DEPTH=30 # allow at most 30 tool calls per session (default: 50)
```

Once the limit is reached, every further call returns:

```json
{
"error": "TOOL_BUDGET_EXCEEDED",
"callsMade": 30,
"limit": 30,
"suggestion": "Summarize progress and return control to the user."
}
```

| Property | Value |
| --------- | -------------------------------------------------------------------------- |
| Default | `50` |
| Scope | Per MCP session (`sessionId` from the MCP SDK) |
| Exemption | `provardx_ping` is never counted or blocked |
| Memory | Sessions are tracked in-process; restarting the server resets all counters |

The guard is designed to prevent runaway agentic loops from making hundreds of tool calls without human review. Set it lower (e.g. `10`) for tightly supervised workflows; raise it or omit it for long-running automation pipelines where you trust the agent.

### Per-call token attribution (`PROVAR_MCP_EMIT_TOKEN_META`)

Appends a `_meta` object to `structuredContent` on every tool response, giving observability tooling a lightweight token-cost signal per call.

```
PROVAR_MCP_EMIT_TOKEN_META=true
```

When enabled, `structuredContent` gains a `_meta` key:

```json
{
"result": "...",
"_meta": {
"tool": "provar_project_inspect",
"detailLevel": "standard",
"estimatedTokens": 412
}
}
```

On `TOOL_BUDGET_EXCEEDED` errors the meta also includes the session cumulative total:

```json
{
"_meta": {
"tool": "provar_project_inspect",
"detailLevel": "standard",
"estimatedTokens": 38,
"sessionTotalEstimatedTokens": 8204
}
}
```

| Field | Description |
| ----------------------------- | -------------------------------------------------------------------------------------------- |
| `tool` | Name of the tool that produced this response |
| `detailLevel` | Value of the `detail` argument passed by the caller (`"summary"`, `"standard"`, or `"full"`) |
| `estimatedTokens` | `ceil(len(JSON.stringify(response)) / 4)` — a rough character-to-token estimate |
| `sessionTotalEstimatedTokens` | Cumulative estimate for the session; only present on budget-exceeded errors |

> **Implementation note:** `_meta` is intentionally placed only in `structuredContent`, never in `content[0].text`. LLM clients read `content[0].text`; including observability data there would waste tokens on every response.
21 changes: 21 additions & 0 deletions src/mcp/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ import { registerAllNitroXTools } from './tools/nitroXTools.js';
import { registerAllTestCaseStepTools } from './tools/testCaseStepTools.js';
import { registerAllConnectionTools } from './tools/connectionTools.js';
import { registerAllPrompts } from './prompts/index.js';
import {
createDepthGuardState,
wrapWithDepthGuard,
type AnyToolCallback,
type DepthGuardState,
} from './utils/tokenMeta.js';
import { desc } from './tools/descHelper.js';

// ── Tool group registry ───────────────────────────────────────────────────────
Expand Down Expand Up @@ -129,6 +135,12 @@ export function createProvarMcpServer(config: ServerConfig): McpServer {
}
);

// ── Depth-guard middleware (PDX-474) ─────────────────────────────────────────
const rawLimit = parseInt(process.env['PROVAR_MCP_MAX_TOOL_DEPTH'] ?? '50', 10);
const depthLimit = Number.isNaN(rawLimit) || rawLimit <= 0 ? 50 : rawLimit;
const depthState = createDepthGuardState();
patchWithMiddleware(server, depthState, depthLimit);

// ── Provar tools ─────────────────────────────────────────────────────────────
const activeGroups = parseActiveGroups();
for (const [group, registrars] of Object.entries(TOOL_GROUPS)) {
Expand Down Expand Up @@ -254,6 +266,15 @@ export function createProvarMcpServer(config: ServerConfig): McpServer {
return server;
}

function patchWithMiddleware(server: McpServer, state: DepthGuardState, limit: number): void {
const orig = server.registerTool.bind(server);
type RegisterToolFn = (n: string, c: unknown, h: AnyToolCallback) => unknown;
// Cast through unknown to patch the overloaded method without triggering no-unsafe-any.
const patchable = server as unknown as { registerTool: RegisterToolFn };
patchable.registerTool = (name: string, config: unknown, handler: AnyToolCallback): unknown =>
(orig as unknown as RegisterToolFn)(name, config, wrapWithDepthGuard(name, handler, state, limit));
}

/**
* Resolve the docs directory for bundled MCP Markdown resources.
* In compiled output (lib/mcp/) the sibling docs/ dir exists; in dev/ts-node
Expand Down
140 changes: 140 additions & 0 deletions src/mcp/utils/tokenMeta.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
/*
* Copyright (c) 2024 Provar Limited.
* All rights reserved.
* Licensed under the BSD 3-Clause license.
* For full license text, see LICENSE.md file in the repo root or https://opensource.org/licenses/BSD-3-Clause
*/

// --------------------------------------------------------------------------- //
// Minimal structural types — avoids importing SDK internal paths.
// --------------------------------------------------------------------------- //

type ContentItem = { type: 'text'; text: string };

export interface ToolResult {
content: ContentItem[];
structuredContent?: Record<string, unknown>;
isError?: boolean;
}

interface ToolExtra {
sessionId?: string;
}

export type AnyToolCallback = (args: Record<string, unknown>, extra: ToolExtra) => ToolResult | Promise<ToolResult>;

// --------------------------------------------------------------------------- //
// PDX-474 — Depth Guard (PROVAR_MCP_MAX_TOOL_DEPTH)
// --------------------------------------------------------------------------- //

interface SessionEntry {
calls: number;
totalEstimatedTokens: number;
}

export type DepthGuardState = Map<string, SessionEntry>;

const MAX_SESSIONS = 1000;

export function createDepthGuardState(): DepthGuardState {
return new Map();
}

function getOrCreateEntry(state: DepthGuardState, sessionId: string): SessionEntry {
if (!state.has(sessionId)) {
if (state.size >= MAX_SESSIONS) {
const oldest: string | undefined = state.keys().next().value as string | undefined;
if (oldest !== undefined) state.delete(oldest);
}
state.set(sessionId, { calls: 0, totalEstimatedTokens: 0 });
}
// Non-null guaranteed by the set above or pre-existing entry.
return state.get(sessionId) as SessionEntry;
}

/**
* Wraps a tool handler to enforce a per-session call budget.
* Once `limit` calls have been made for a session, every further call returns
* TOOL_BUDGET_EXCEEDED without invoking the underlying handler.
* Callers without a sessionId (stdio transports — Claude Desktop, Cursor, etc.)
* share a single 'anon' bucket so the budget actually limits runaway tool use;
* giving each anon call a fresh UUID would defeat the purpose of the guard.
* `provardx_ping` is excluded from wrapping at the call site in server.ts.
*/
export function wrapWithDepthGuard(
toolName: string,
handler: AnyToolCallback,
state: DepthGuardState,
limit: number
): AnyToolCallback {
return async (args, extra) => {
const sessionId = extra.sessionId ?? 'anon';
const entry = getOrCreateEntry(state, sessionId);

if (entry.calls >= limit) {
const payload = {
error: 'TOOL_BUDGET_EXCEEDED',
callsMade: entry.calls,
limit,
suggestion: 'Summarize progress and return control to the user.',
};
const response: ToolResult = {
isError: true,
content: [{ type: 'text' as const, text: JSON.stringify(payload) }],
structuredContent: payload,
};
return attachMeta(response, toolName, 'standard', entry.totalEstimatedTokens);
}

entry.calls++;
const result = await handler(args, extra);

if (process.env['PROVAR_MCP_EMIT_TOKEN_META'] === 'true') {
entry.totalEstimatedTokens += estimateTokens(result);
}

const detailLevel = typeof args['detail'] === 'string' ? args['detail'] : 'standard';
return attachMeta(result, toolName, detailLevel);
Comment on lines +89 to +97
};
}

// --------------------------------------------------------------------------- //
// PDX-475 — Token meta attachment (PROVAR_MCP_EMIT_TOKEN_META)
// --------------------------------------------------------------------------- //

export function estimateTokens(payload: unknown): number {
return Math.ceil(JSON.stringify(payload).length / 4);
}

/**
* Appends a `_meta` key to `structuredContent` when PROVAR_MCP_EMIT_TOKEN_META=true.
* The `content[0].text` string is intentionally left unchanged — LLMs read that
* field, so including meta there would waste tokens on observability data.
*
* @param sessionTotalTokens - Cumulative estimated tokens for the session,
* included only on TOOL_BUDGET_EXCEEDED errors.
*/
export function attachMeta(
response: ToolResult,
toolName: string,
detailLevel: string,
sessionTotalTokens?: number
): ToolResult {
if (process.env['PROVAR_MCP_EMIT_TOKEN_META'] !== 'true') return response;

const meta: Record<string, unknown> = {
tool: toolName,
detailLevel,
estimatedTokens: estimateTokens(response),
};

if (sessionTotalTokens !== undefined) {
meta['sessionTotalEstimatedTokens'] = sessionTotalTokens;
}

const existing = response.structuredContent ?? {};
return {
...response,
structuredContent: { ...existing, _meta: meta },
};
}
Loading
Loading