docs(k8s-proxy-llm-workflow): point Routine A at legacy /tr MCP tools#863
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the v4.0.0 K8s Proxy “Developer + LLM Workflow” playbook so Routine A fetches failing keploy cloud replay --branch-name results from the legacy /tr test-run-report surface (via listTestReports + getTestReportFull) and uses the legacy edit path (updateTestCase) instead of the api-testing test_suite_run surface.
Changes:
- Switch Routine A report discovery/fetch from
listTestRuns+get_session_reporttolistTestReports+getTestReportFull. - Update diagnosis guidance to read legacy
/trreport fields (test_sets[].test_cases[].oss_report.*) and update write actions toupdateTestCase/update_mock. - Fix the run-report dashboard URL to point at
/tr/<test_run_id>?appId=<app_id>.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
`keploy cloud replay --branch-name` writes its uploaded report into
the legacy /tr collection (`testrunReports` in Mongo). The playbook
was directing the agent at the api-testing test-suite-run surface
(`listTestRuns({kind: "test_suite_run"})` + `get_session_report`),
which queries a different collection — so Phase A1 would return
nothing for the dev's branch and Phase A2 (even when given a
test_run_id) would return the sandbox-session shape (`failed_steps[]`
/ `mock_mismatch_dominant`) instead of legacy /tr (per-test-case
`oss_report.result.*` / `oss_report.mock_mismatches`). The agent's
autonomous diagnosis path would silently fall back to asking the dev.
Switch the playbook to the legacy /tr tools that match the underlying
data model:
- Phase A1: `listTestReports({appId, branch_id, status: "FAILED", limit: 5})`,
with `source: "ci"` for the CI form.
- Phase A2: `getTestReportFull({appId, reportId})` — returns rollup +
every test set + per-test-case diff + `mock_mismatches` in one
round-trip; `mock_mismatches_only=true` flag for mock-drift focus.
- Phase A3 / A4 / A5: read fields as `oss_report.result.body_result[]`
/ `oss_report.mock_mismatches.{expected_mocks,actual_mocks}`; write
via `updateTestCase` (legacy /tr's edit tool) instead of
`update_test_suite` (api-testing's suite-edit tool).
- "Run report" URL goes from `/tr?appId=<app_id>&branch=<branch>`
(the list page with a non-existent `branch` query param) to
`/tr/<test_run_id>?appId=<app_id>` (the actual detail page).
Updates the hard rules, the three Phase tables in Step 3, and the
Scenario 2 / 3 narrative examples for consistency.
Signed-off-by: Charan Kamarapu <kamarapucharan@gmail.com>
charankamarapu
force-pushed
the
docs/fix-llm-workflow-mcp-tool-names
branch
from
3e62409 to
31864b5
Compare
…check Vale's spelling rule rejects `rollup` (not in the accept vocab). The hyphenated form passes the same rule with the same meaning. Single substitution across the page (4 occurrences). Signed-off-by: Charan Kamarapu <kamarapucharan@gmail.com>
Address Copilot review feedback: - Phase A2: clarify that `getTestReportFull`'s **path** params (appId / reportId) are camelCase per the OpenAPI spec, while its **query** params (include_oss_report / mock_mismatches_only / max_test_cases_per_set) stay snake_case as declared. The earlier wording implied the query params were also camelCase, which would make readers/agents copy invalid object literals. - Phase A3 heading + the Step 3 phase table now read "test case" instead of "step" — the report surface is per test_sets[].test_cases[], and the A5 report template was already "Test set | Test case", so the heading/table phrasing now matches. Signed-off-by: Charan Kamarapu <kamarapucharan@gmail.com>
ayush3160
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
keploy cloud replay --branch-nameuploads its report into the legacy/trcollection (testrunReportsin Mongo, served by thelistTestReports/getTestReport/getTestReportFullMCP tools). The page #862 just merged was pointing Routine A at the api-testing test-suite-run surface instead (listTestRuns({kind: \"test_suite_run\"})+get_session_report), which queries a different collection and returns a different report shape.Concretely, an agent running the playbook as written would:
listTestRuns({kind: \"test_suite_run\"})returns nothing for the dev's branch (the run lives intestrunReports, nottestSuiteRun). Agent gives up and asks the dev for the report id, breaking the "autonomous" promise.test_run_idin hand,get_session_reportis the sandbox-session report; it returns keys likefailed_steps[]/mock_mismatch_dominant. Legacy/trreports don't have those — they havetest_sets[].test_cases[].oss_report.result.{body_result,headers_result,status_code}andoss_report.mock_mismatches. The playbook's reads returnundefinedand the diagnosis can't run.This PR points Routine A at the correct tool surface (which now exists in api-server PR #1679 —
getTestReportFullwas added explicitly for the AI/MCP analysis case):/tredit tool) instead of `update_test_suite` (which is api-testing's tool).Hard rules, the three Phase tables in Step 3, and the Scenario 2 / 3 narrative examples updated for consistency.
Test plan
🤖 Generated with Claude Code