Develop by mrdailey99 · Pull Request #127 · ProvarTesting/provardx-cli

mrdailey99 · 2026-04-22T20:53:29Z

No description provided.

…mpts Req: Phase 2 corpus feature — canonical XML step reference bundled into MCP server and served as provar://docs/step-reference resource Fix: Add PROVAR_TEST_STEP_REFERENCE.md with all step API IDs and three GitHub Copilot agent prompts for migration workflows Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…st generation Req: Phase 2 corpus feature — orchestrated MCP prompts that ground AI agents in corpus examples before generating or fixing Provar XML test cases Fix: Register 7 prompts via MCP protocol: provar.migrate.crt/selenium/playwright and provar.loop.generate/fix/review/coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…uth test lint Req: CLAUDE.md requires unit tests for new MCP functionality covering registration, argument interpolation, and optional arg handling Fix: Add migrationPrompts.test.ts (16 tests) and loopPrompts.test.ts (24 tests); add eslint-disable camelcase to auth status test for expires_at fields Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…erver Req: Phase 2 corpus feature — expose prompts via MCP protocol and serve canonical step reference as a resource at provar://docs/step-reference Fix: Register prompts, add MCP resource with readFileSync, update compile script to bundle docs, sync server.json and bump to beta.8 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ference Req: Phase 2 migration corpus feature — document 7 new MCP prompts (provar.migrate.crt/selenium/playwright, provar.loop.generate/fix/review/coverage) and the provar://docs/step-reference MCP resource Fix: Add MCP Prompts section and MCP Resources section to docs/mcp.md, update README tool count blurb, add Scenario 11 migration evaluation scenario to mcp-pilot-guide.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… eslint narrowing Req: Fix three Copilot PR review findings: broken ToC anchor fragments in docs/mcp.md, missing targetOrg argument in provar.loop.coverage docs table, and overly broad eslint-disable camelcase in auth status test. Fix: Corrected anchor hrefs (#provarmigrateselenium, #provarmigrateplaywright, #provarloopcoverage), added targetOrg row to arguments table, narrowed eslint disable to two eslint-disable-next-line comments on the specific expires_at lines.

…eturns no examples Req: When provar.qualityhub.examples.retrieve degrades gracefully (API unavailable, unauthenticated, or rate-limited), AI agents had no alternative grounding source. This left direct tool invocations and Copilot prompt users without format guidance. Fix: Added CORPUS_FALLBACK_HINT to all four warning constants in the retrieve tool so the hint surfaces regardless of how the tool is invoked. Updated migrationOrchestration in migrationPrompts.ts, all corpus-retrieval steps in loopPrompts.ts, and all three .github/prompts/*.prompt.md files to instruct agents to read provar://docs/step-reference when count:0 + warning is returned.

… MCP registry listing Req: MCP registry listing benefits from display name, docs URL, and branded icon for client UI rendering. These are optional schema fields but expected by the registry publisher and MCP clients. Fix: Added title, websiteUrl pointing to docs/mcp.md, and icons referencing the Provar automation icon at assets/icon.png via raw.githubusercontent.com HTTPS URL.

…orpus feat(mcp): add 7 MCP prompts for test migration and AI loop workflows

Copilot

Pull request overview

Adds first-class MCP prompts for common Provar AI workflows (migration + iterative loop), plus a packaged step reference resource to support offline/fallback guidance when Quality Hub corpus retrieval is unavailable.

Changes:

Register 7 MCP prompts (provar.migrate.*, provar.loop.*) and wire them into the MCP server.
Add provar://docs/step-reference MCP resource and update corpus-retrieval warnings to point to it as a fallback.
Add/extend smoke + unit tests and update docs/versioning/assets to publish the new capabilities.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`src/mcp/prompts/migrationPrompts.ts`	Adds migration prompts for CRT/Selenium/Playwright with a shared orchestration workflow.
`src/mcp/prompts/loopPrompts.ts`	Adds loop prompts for generate/fix/review/coverage workflows.
`src/mcp/prompts/index.ts`	Central registry for all prompt registrations.
`src/mcp/server.ts`	Registers prompts and adds the `provar://docs/step-reference` resource backed by packaged markdown.
`src/mcp/tools/qualityHubApiTools.ts`	Extends corpus warning messages with a step-reference fallback hint.
`docs/PROVAR_TEST_STEP_REFERENCE.md`	New canonical step reference document (packaged + exposed via MCP resource).
`docs/mcp.md`	Documents the new prompts and the new MCP resource.
`docs/mcp-pilot-guide.md`	Adds a migration prompt usage scenario to the pilot guide.
`test/unit/mcp/migrationPrompts.test.ts`	Unit tests for migration prompt registration and message content.
`test/unit/mcp/loopPrompts.test.ts`	Unit tests for loop prompt registration and message content.
`test/unit/commands/provar/auth/status.test.ts`	Adds targeted eslint disables for `expires_at` camelcase usage in tests.
`scripts/mcp-smoke.cjs`	Extends smoke test coverage to `prompts/list` and all new prompts.
`package.json`	Version bump + build step copies step-reference doc into `lib/mcp/docs/`.
`server.json`	Adds MCP server manifest metadata for distribution.
`.github/prompts/*.prompt.md`	Adds GitHub prompt templates for CRT/Selenium/Playwright migration.
`README.md`	Updates MCP docs blurb to mention “7 MCP prompts”.
`assets/icon.png`	Adds an icon for the MCP server manifest.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-22T20:58:51Z

+  const docsDir = join(dirname(fileURLToPath(import.meta.url)), 'docs');
+  server.resource(
+    'provar-step-reference',
+    'provar://docs/step-reference',
+    {
+      description:
+        'Canonical reference for all Provar XML test step API IDs, argument formats, validation rules, and corpus-verified examples. Use this to understand correct step structure when generating or reviewing test cases.',
+      mimeType: 'text/markdown',
+    },
+    () => {
+      try {
+        const content = readFileSync(join(docsDir, 'PROVAR_TEST_STEP_REFERENCE.md'), 'utf-8');
+        return {
+          contents: [
+            {
+              uri: 'provar://docs/step-reference',
+              mimeType: 'text/markdown',
+              text: content,
+            },


The resource handler reads the markdown from disk on every provar://docs/step-reference request. This adds synchronous I/O to the request path and can become a bottleneck if clients fetch the resource repeatedly. Consider reading the file once (lazy cache) and reusing the cached string, falling back only if the initial load fails.

Copilot · 2026-04-22T20:58:52Z

+Follow these steps in order:
+
+1. **Read the failing test case** — read the XML file at the path above using your file reading tools.
+
+2. **Parse the failure** — from the RCA output, identify:
+   - Which step is failing (step name, testItemId, or title)
+   - The failure category (e.g. element not found, assertion mismatch, connection error, XML structure error)
+   - The specific error message
+
+3. **Get corpus examples** — call \`provar.qualityhub.examples.retrieve\` with keywords describing the
+   failing step's scenario (e.g. "close opportunity UiDoAction" or "assert field value UiAssert").
+   Use the returned examples to verify the correct structure for the failing step type.
+   If the response has \`"count": 0\` with a \`"warning"\` field, fall back: read the
+   \`provar://docs/step-reference\` MCP resource for the correct attribute schema for the failing
+   step type, then continue.
+
+4. **Diagnose the root cause** — compare the failing step's XML against the corpus examples. Common issues:
+   - Wrong interaction URI (action vs set vs file)
+   - Missing required UiAssert arguments (columnAssertions, pageAssertions, resultScope, captureAfter, beforeWait, autoRetry)
+   - UiDoAction (set) missing the value argument
+   - UiWithScreen navigate="Dont" on the first screen
+   - navigate="Always" on Edit/View without sfUiTargetObjectId
+   - Incorrect locator URI format
+   - connectionId using valueClass="string" instead of valueClass="id"
+
+5. **Apply the fix** — rewrite only the failing step(s). Preserve all other steps unchanged. Write the
+   updated XML back to the same file path.
+
+6. **Validate** — call \`provar.testcase.validate\` on the updated file. If new errors appear, fix them
+   and re-validate until the file passes clean.
+
+7. **Report** — summarise:
+   - The root cause of the failure
+   - Exactly what was changed (step name, argument changed, before/after values)
+   - Any remaining warnings from validation
+
+Begin with step 1: read the file at: ${testcasePath}`,


This prompt workflow instructs the agent to “read the XML file … using your file reading tools” and later “write the updated XML back to the same file path”, but this MCP server does not expose any general-purpose file read/write tools for .testcase files (only specialized tools like provar.testcase.validate which don’t return editable XML). As written, provar.loop.fix is likely not executable in clients that only have MCP access (e.g. Claude Desktop). Either add explicit MCP tools for reading/writing .testcase content (guarded by allowedPaths) or update the prompt to use an existing tool-based flow that can actually perform the required file operations.

Copilot · 2026-04-22T20:58:52Z

+1. **Scan local test files** — list all .testcase files under ${projectPath}/tests/. For each file that
+   references "${objectName}" (search for the string in the file content), read it and extract:
+   - The test scenario (from the file name or meta comment)
+   - Which operations are covered: API Create, API Read, API Update, API Delete, UI New, UI View,
+     UI Edit, UI Delete, UI assertions on fields
+   - Which fields are set or asserted
+


The provar.loop.coverage workflow asks the agent to list and read all .testcase files under ${projectPath}/tests/, but the MCP server doesn’t provide a tool to enumerate and read arbitrary test case files. In MCP-only clients this step can’t be completed as described. Consider either (a) adding a tool to list/read .testcase files under an allowed project root, or (b) revising the prompt to rely on existing tools like provar.project.inspect for discovery and whatever supported mechanism you intend for reading file contents.

Copilot · 2026-04-22T20:58:52Z


+## MCP Prompts
+
+The Provar MCP server registers **7 MCP prompts** that pre-wire the tool chain into turnkey workflows. AI clients that support MCP prompts (Claude Desktop, Claude Code) can invoke them directly by name instead of manually orchestrating the underlying tool sequence.


This section describes the prompts as “turnkey workflows” usable by Claude Desktop/Claude Code, but several prompts (notably provar.loop.fix and provar.loop.coverage) require listing/reading/writing local .testcase files and the MCP server doesn’t currently expose file system tools to do that. Consider clarifying the client requirements (e.g., needs a client with its own file tools like Claude Code) or updating the server/prompts so the workflows are actually achievable via MCP alone.

Suggested change

The Provar MCP server registers **7 MCP prompts** that pre-wire the tool chain into turnkey workflows. AI clients that support MCP prompts (Claude Desktop, Claude Code) can invoke them directly by name instead of manually orchestrating the underlying tool sequence.

The Provar MCP server registers **7 MCP prompts** that pre-wire the tool chain into guided workflows. AI clients that support MCP prompts can invoke them directly by name instead of manually orchestrating the underlying tool sequence. **Important:** prompts that need to list, read, or write local project files (for example, `.testcase` files used by loop/fix and coverage workflows) also require a client with its own workspace/file tools, such as Claude Code or another MCP-compatible client with local file access configured; MCP prompt support alone is not sufficient for those workflows.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

… docs Req: Real-world DB test session exposed 3 root causes: wrong funcCall/variable-path patterns, connectionId using valueClass=string instead of id, and mismatched resultName/dbConnectionName coupling. Fix: Added provar.loop.db prompt enforcing DB-specific rules; expanded PROVAR_TEST_STEP_REFERENCE.md Database section with critical constraints and correct counter-examples for funcCall and structured paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Req: PR #127 Copilot review flagged two issues: fallback error message uses dev-only yarn compile guidance, and MCP Prompts docs did not clarify client file-access requirements for loop.fix and loop.coverage workflows. Fix: Updated step-reference fallback text to be meaningful for installed users (reinstall/rebuild), and updated MCP Prompts docs intro to note that file-dependent prompts require a client with local file tools such as Claude Code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Req: Two valid Copilot review findings on PR #128 — ambiguous parameter naming and wrong valueClass casing in XML emitter. Fix: Rename provar.loop.db param dbConnectionName->connectionName to avoid confusion with SqlQuery.dbConnectionName; fix buildArgumentsXml to emit lowercase valueClass="string" instead of "String". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(mcp): add provar.loop.db prompt and fix database step reference

mrdailey99 and others added 9 commits April 22, 2026 15:03

Merge pull request #126 from ProvarTesting/feature/phase2-migration-c…

4bb0fac

…orpus feat(mcp): add 7 MCP prompts for test migration and AI loop workflows

Copilot AI review requested due to automatic review settings April 22, 2026 20:53

Copilot started reviewing on behalf of mrdailey99 April 22, 2026 20:53 View session

Copilot AI reviewed Apr 22, 2026

View reviewed changes

mrdailey99 and others added 5 commits April 22, 2026 16:21

Update src/mcp/server.ts

485e2e4

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge pull request #128 from ProvarTesting/feature/phase2-db-feedback

c3946cd

feat(mcp): add provar.loop.db prompt and fix database step reference

mrdailey99 merged commit 7ba1616 into main Apr 23, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Develop#127

Develop#127
mrdailey99 merged 14 commits intomainfrom
develop

mrdailey99 commented Apr 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 22, 2026

Uh oh!

Uh oh!

Copilot AI Apr 22, 2026

Uh oh!

Copilot AI Apr 22, 2026

Uh oh!

Copilot AI Apr 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		## MCP Prompts

		The Provar MCP server registers 7 MCP prompts that pre-wire the tool chain into turnkey workflows. AI clients that support MCP prompts (Claude Desktop, Claude Code) can invoke them directly by name instead of manually orchestrating the underlying tool sequence.

Conversation

mrdailey99 commented Apr 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants