Skip to content

test(mcp): regression harness for tools/list, tool results, CLI parity and staging stdio - #65

Merged
jernejk merged 3 commits into
mainfrom
test/mcp-regression-harness
Sep 12, 2026
Merged

test(mcp): regression harness for tools/list, tool results, CLI parity and staging stdio#65
jernejk merged 3 commits into
mainfrom
test/mcp-regression-harness

Conversation

@jernejk

@jernejk jernejk commented Sep 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

The MCP regression harness that has to land before #38, #46 and #57 change the code 47 MCP tools
sit on. Test-only apart from one production change needed for config isolation.

Closes #63

What changed

Production (one change). ConfigPaths honours TIMEPRO_CLI_CONFIG_DIR, expanding ~ and
relative paths. It is read once, so it relocates the tenants directory, repo mappings and the local
command log too. McpHostCommand constructs new ConfigService() directly, so without this a stdio
test would run against the developer's real tenants and feature flags. Documented in AGENTS.md and
covered by ConfigPathsTests.

1. Golden tools/list from the real stdio server. McpStdioClient launches
dotnet SSW.TimePro.Cli.dll mcp as a child process against a throwaway config directory, does
initialize / notifications/initialized / paginated tools/list, and snapshots the whole tool
list — names, descriptions, input schemas, execution, annotations — to
Goldens/Mcp/Discovery/tools-list.{default,accounting}.json (18 and 47 tools). Separate tests
assert the counts, that the set difference is exactly the accounting tools, and that names are
unique. Tool names on the wire are snake_case (accept_suggested_timesheet), so every cross-check
goes through McpToolInventory.WireName.

2. Per-tool result goldens. McpToolCatalog declares one populated case per tool; empty and
apiError variants are generated from each case's declared primary route. 130 goldens in
Goldens/Mcp/Tools/, holding the raw JSON text the tool returned. All fixtures come from one fake
instance, NorthwindApi, whose bodies are serialised from the real DTOs rather than hand-written,
so a fixture that stops binding fails instead of producing a golden full of nulls. An API failure
that escapes a tool as a protocol error rather than an isError payload is snapshotted as such, so
changing that policy is a reviewed change. Six tools also run through real tools/call over stdio
(Goldens/Mcp/Calls/) and five of those are asserted byte-equal to their direct-call golden.

3. CLI/MCP parity runner. McpCliParityTable has a row for every one of the 47 tools. Nine rows
are executable today: they run the real command tree in-process with --json and the MCP tool
against identical, reset WireMock state, and compare complete parsed documents. Differences are
declared per case as JSON paths (JsonDiff) — there is no generic normalisation. Writes also assert
method, path, payload fragments, call count and forbidden calls. Two rows are already at full
parity (ts delete on a real row, leave balances status); the rest carry ExpectParity = false
plus a note and separate goldens, so a later slice flips a flag.

Reflection mapping test. Every [McpServerTool] method must have a table row and its CLI
command path must resolve in the real command tree, with a shrink-only allowlist for the five tools
that have no CLI mirror (GetInvoicesByClient, GetUnpaidInvoicesByClient, ListCategories,
ListBillableTypes, ListAllSkus).

4. Staging stdio smoke. scripts/e2e/test-mcp-smoke.sh wraps scripts/e2e/mcp_smoke.py
(standard library only). It refuses to continue unless the resolved tenant is non-production,
compares discovery to the golden, reads Northwind projects/iterations/rate/timesheets, writes one
uniquely marked short entry through MCP, reads it back, then deletes only that unambiguously
identified row and verifies absence. Cleanup runs in finally and a cleanup it cannot complete
fails the run and prints how to find the leftover row. The target project and iteration come from
TIMEPRO_MCP_SMOKE_PROJECT / TIMEPRO_MCP_SMOKE_ITERATION with no committed default — this is
a public repository — and the script skips when the project is unset. Variables are documented in
scripts/e2e/README.md. run-all.sh picks it up automatically via its test-*.sh glob.

Goldens are read from and written to the source tree; regenerate deliberately with
UPDATE_MCP_GOLDENS=1 dotnet test.

How verified

Test counts before → after:

suite before after
tests/SSW.TimePro.Cli.Tests 540 546
tests/SSW.TimePro.Cli.Integration 105 266 
$ dotnet test tests/SSW.TimePro.Cli.Tests/
Passed!  - Failed: 0, Passed: 546, Total: 546, Duration: 1 s

$ dotnet test tests/SSW.TimePro.Cli.Integration/
Passed!  - Failed: 0, Passed: 266, Total: 266, Duration: 8 s

The 161 new integration tests add about 8 s, dominated by the child-process MCP hosts the stdio
collection starts. The stdio tests also pass under -c Release, confirming CliBinary resolves the
configuration it is actually running in.

Forced failure — perturbing two goldens (a totalTime value and a tool description) fails exactly
the three tests that should notice:

$ dotnet test tests/SSW.TimePro.Cli.Integration/ --filter "FullyQualifiedName~Integration.Mcp"
  McpToolResultGoldenTests.ToolResult_MatchesGolden(caseName: "GetTimesheets/populated") [FAIL]
  McpStdioDiscoveryTests.ToolsList_WithAccountingDisabled_MatchesGolden [FAIL]
  McpStdioToolCallTests.ToolCall_OverStdio_ReturnsTheSameTextAsTheDirectCall(...get_timesheets...) [FAIL]
Failed!  - Failed: 3, Passed: 155, Total: 158

The harness also earned its keep on the rebase onto current main, catching exactly four
behaviour changes already shipped there and nothing else:

#61 stated the tool output was unaffected for the existing filters. That is now verified rather
than assumed: a new GetLeaveEntries.past golden is byte-identical to the UPCOMING one (same
sha256). New cases also cover ALL, whose merge and de-duplication run against a deliberately
distinct PAST row (2 items out, 1 from each filter), and an unknown filter, which returns the
validation message with zero API calls.

Staging run of the smoke script against the release build (trimmed; server logs removed):

$ TIMEPRO_MCP_SMOKE_PROJECT=<a NWIND project that uses iterations> \
  TIMEPRO_MCP_SMOKE_TP="dotnet .../SSW.TimePro.Cli.dll" scripts/e2e/test-mcp-smoke.sh
  Testing: tp mcp (stdio) against staging
  tenant ssw-staging resolves to a non-production host: https://api.staging-sswtimepro.com
  initialize negotiated protocol 2024-11-05
  tools/list returned 18 tools, all 18 default tools present
  the configured NWIND project is present
  selected 1 of 3 iterations on the project
  client rate present for NWIND
  marker 'MCP smoke <token>, safe to delete' is unused on <weekday>
  nothing to clean up: the marker note is not on the day
MCP smoke FAILED: create_timesheet returned isError: {...}

Skip path, with the project variable unset:

  Testing: tp mcp (stdio) against staging
    SKIPPED: TIMEPRO_MCP_SMOKE_PROJECT not set (see scripts/e2e/README.md)

Two failure paths were exercised against staging with locally patched copies of the driver, to prove
the parts a passing run never reaches:

  • Failure after a successful write. Injecting an exception straight after the row is created:
    cleaned up entry <id> and verified its absence, exit 1. The row does not survive.
  • A silent MCP host. Pointing the driver at a stand-in that accepts input and never answers,
    with the timeout lowered to 5 s: timed out after 5s waiting for response 1; terminated the MCP host, exit 1, elapsed 5 s, child process confirmed gone. Before the queue-based reader this
    hung indefinitely.

Finding: the MCP create tool is broken against a real server

The smoke script found it on the first staging run. CreateTimesheet builds a TimesheetRequest
with no SellPrice; CreateCommand resolves one from the client rate first, and its own comment
says the API rejects a timesheet it cannot derive a sell price for. Staging confirms it: with a
category supplied the CLI create succeeds, the MCP tool with the identical arguments still returns
400, and SellPrice is the only field that differs between the two payloads. That is #57.

The smoke script does not route around it: the write goes through MCP on purpose, so this gate
stays red until the shared create orchestration lands. The remaining steps were verified by
temporarily substituting a CLI create locally — read-back, field assertions, delete and
absence-check all pass, and every row written during these runs was deleted and its absence
verified.

Everything above was verified by execution. Verified by inspection only: that the goldens capture
every field a client depends on — they capture every field the tools currently emit, which is the
strongest available statement without a real client in the loop. As the plan notes, behaviour in
Claude Code, Codex CLI, Claude Desktop and VS Code remains unverified.

Risks

  • TIMEPRO_CLI_CONFIG_DIR is read once into a static, so a process that sets it after first use
    will not see the change. That is fine for a CLI process and for child-process tests, but it is
    not a runtime knob.
  • The stdio tests shell out to dotnet and require the CLI project's build output for the same
    configuration and target framework; CliBinary fails with the expected path and the build command
    when it is missing.
  • ts check / CheckWeek derive their window from the machine clock with no seam, so the current
    week's five dates are tokenised. That is the only normalisation in the harness and it is declared
    in WeekTokens.
  • Wiring the smoke into run-all.sh makes the staging E2E suite red while the MCP create gap
    stands. That is intentional, but it is a visible change to that suite's status.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 12, 2026 14:36
Copilot AI reviewed Sep 12, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds isolated config support and a comprehensive MCP regression harness covering stdio discovery, tool results, CLI parity, and staging smoke tests.

Changes:

  • Adds TIMEPRO_CLI_CONFIG_DIR support.
  • Adds WireMock fixtures, MCP goldens, parity tests, and stdio protocol tests.
  • Adds the staging MCP smoke wrapper and documentation.
File summaries
File Description
tests/SSW.TimePro.Cli.Tests/Infrastructure/ConfigPathsTests.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/WeekTokens.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/TestConfigService.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpToolResultGoldenTests.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpToolInventory.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpToolHost.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpToolCase.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpStdioToolCallTests.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpStdioDiscoveryTests.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpStdioClient.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/McpCliParityTable.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/JsonDiff.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/IsolatedConfigDirectory.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/Golden.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/CliRunner.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/CliMcpParityTests.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Mcp/CliBinary.cs Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/UpdateTimesheet.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/UpdateTimesheet.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/UpdateLeave.dryRun.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/UpdateLeave.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/SearchClients.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/SearchClients.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/SearchClients.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/QueryTimesheets.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/QueryTimesheets.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/QueryTimesheets.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListRecurringInvoices.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListRecurringInvoices.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListRecurringInvoices.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListProducts.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListProducts.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListProducts.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListPaidReceipts.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListPaidReceipts.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListPaidReceipts.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListLocations.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListLocations.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListLocations.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListIterations.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListIterations.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListIterations.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListInvoices.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListInvoices.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListInvoices.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCreditNotes.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCreditNotes.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCreditNotes.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListClientRates.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListClientRates.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListClientRates.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCategories.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCategories.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListCategories.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListBillableTypes.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListBillableTypes.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListBillableTypes.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListAllSkus.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListAllSkus.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ListAllSkus.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ImportLeaveBalances.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/ImportLeaveBalances.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnpaidInvoicesByClient.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnpaidInvoicesByClient.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnpaidInvoicesByClient.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnbilledTimesheetsForClient.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnbilledTimesheetsForClient.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetUnbilledTimesheetsForClient.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetTimesheets.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetTimesheets.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetTimesheets.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetSuggestedTimesheets.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetSuggestedTimesheets.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetSuggestedTimesheets.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetRecurringInvoice.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetRecurringInvoice.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetRecurringInvoice.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetReceiptDetail.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetReceiptDetail.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetReceiptDetail.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsSummary.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsSummary.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsSummary.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsForClient.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsForClient.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProjectsForClient.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProductDiscountsForClient.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProductDiscountsForClient.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProductDiscountsForClient.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProduct.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProduct.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetProduct.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetPrepaidStatusPdf.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetPrepaidStatusPdf.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetPrepaidStatus.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetPrepaidStatus.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetPrepaidStatus.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLocationAndMapping.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveEntries.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveEntries.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveEntries.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalanceStatus.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalanceStatus.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalanceStatus.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalance.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalance.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetLeaveBalance.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceTimesheets.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceTimesheets.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceTimesheets.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoicesByClient.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoicesByClient.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoicesByClient.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceReceipts.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceReceipts.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceReceipts.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceLines.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceLines.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoiceLines.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoice.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoice.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetInvoice.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCurrentUser.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCurrentUser.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCurrentUser.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCrmBookings.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCrmBookings.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetCrmBookings.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientsWithOutstandingTime.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientsWithOutstandingTime.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientsWithOutstandingTime.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientRate.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientRate.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientRate.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientOutstanding.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientOutstanding.empty.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/GetClientOutstanding.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/DeleteTimesheet.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/DeleteTimesheet.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CreateTimesheet.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CreateTimesheet.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CreateLeave.dryRun.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CreateLeave.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CheckWeek.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/CheckWeek.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/AcceptSuggestedTimesheet.populated.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Tools/AcceptSuggestedTimesheet.apiError.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/UpdateTimesheet.0.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/UpdateTimesheet.0.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/UpdateLeave.6.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/UpdateLeave.6.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/ImportLeaveBalances.7.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/ImportLeaveBalances.7.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/GetLeaveBalanceStatus.8.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/GetLeaveBalanceStatus.8.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/DeleteTimesheet.3.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/DeleteTimesheet.3.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/DeleteTimesheet.2.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/DeleteTimesheet.2.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/CreateLeave.5.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/CreateLeave.5.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/CheckWeek.4.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/CheckWeek.4.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/AcceptSuggestedTimesheet.1.mcp.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Parity/AcceptSuggestedTimesheet.1.cli.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/update_timesheet.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/list_iterations.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/get_timesheets.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/get_leave_entries.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/get_client_rate.json Updated as part of this pull request.
tests/SSW.TimePro.Cli.Integration/Goldens/Mcp/Calls/delete_timesheet.json Updated as part of this pull request.
src/SSW.TimePro.Cli/Infrastructure/Config/ConfigPaths.cs Updated as part of this pull request.
scripts/e2e/test-mcp-smoke.sh Updated as part of this pull request.
scripts/e2e/mcp_smoke.py Updated as part of this pull request.
AGENTS.md Updated as part of this pull request.
.gitignore Updated as part of this pull request.
Review details

Suppressed comments (3)

scripts/e2e/mcp_smoke.py:22

  • When invoked through run-all.sh, this fallback does not match the runner's resolved tenant: run-all.sh:25 defaults TIMEPRO_E2E_TENANT to ssw, logs in only that config, and then invokes this script. A fresh documented run-all.sh run can therefore fail because ssw-staging was never created or updated; derive the fallback from the runner's tenant or pass the resolved tenant explicitly.
TENANT = os.environ.get("TIMEPRO_MCP_SMOKE_TENANT", "ssw-staging")

tests/SSW.TimePro.Cli.Integration/Mcp/McpCliParityTable.cs:10

  • BodyContains only checks selected substrings, so a write can omit or corrupt any unlisted field and still satisfy the parity assertion. The linked harness requirement calls for comparing the full write payload; if fragment matching is intentional for this interim slice, the test should explicitly document which fields are out of scope rather than presenting this as complete payload protection.
    tests/SSW.TimePro.Cli.Integration/Mcp/McpToolInventory.cs:15
  • The inventory only scans these four concrete types. A new [McpServerToolType] class registered in McpHostCommand but omitted here will be invisible to AllMethods, so the catalog-coverage and CLI-mapping tests still pass without covering it, contrary to the stated every-registered-tool guarantee. Derive the inventory from the registered/attributed types or add an independent registration-versus-inventory assertion.
  • Files reviewed: 179/180 changed files
  • Comments generated: 10
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/e2e/mcp_smoke.py Outdated
Comment on lines +86 to +94
deadline = threading.Event()
timer = threading.Timer(READ_TIMEOUT_SECONDS, deadline.set)
timer.start()
try:
while True:
if deadline.is_set():
raise SmokeFailure(f"timed out waiting for response {request_id}")

line = self._process.stdout.readline()
Comment thread scripts/e2e/test-mcp-smoke.sh
TIMEPRO_MCP_SMOKE_TP=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1].split()))' "$TP_COMMAND")
export TIMEPRO_MCP_SMOKE_TP

python3 "$SCRIPT_DIR/mcp_smoke.py"
Comment thread scripts/e2e/mcp_smoke.py Outdated
READ_TIMEOUT_SECONDS = 90

CLIENT_ID = "NWIND"
PROJECT_ID = os.environ.get("TIMEPRO_MCP_SMOKE_PROJECT", "8W52M2")
Comment thread scripts/e2e/mcp_smoke.py
Comment on lines +208 to +212
missing = sorted(expected - set(names))
if missing:
raise SmokeFailure(f"tools/list is missing golden tools: {missing}")

print(f" tools/list returned {len(names)} tools, all {len(expected)} default tools present")
Comment thread scripts/e2e/mcp_smoke.py Outdated
)

created = matches[0]
created_id = created["timeId"]
Comment thread scripts/e2e/test-mcp-smoke.sh Outdated
#
# TIMEPRO_MCP_SMOKE_TP how to invoke the candidate CLI (default: tp)
# TIMEPRO_MCP_SMOKE_TENANT tenant config to bind (default: ssw-staging)
# TIMEPRO_MCP_SMOKE_PROJECT Northwind project that uses iterations (default: 8W52M2)
Comment thread scripts/e2e/test-mcp-smoke.sh

# Serialise the command as JSON so "dotnet run --project ... --" works as well as a bare "tp".
TIMEPRO_MCP_SMOKE_TP=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1].split()))' "$TP_COMMAND")
export TIMEPRO_MCP_SMOKE_TP
Comment thread tests/SSW.TimePro.Cli.Integration/Mcp/CliMcpParityTests.cs
Comment on lines +139 to +152
foreach (var expected in row.ExpectedRequests)
{
var matches = requests
.Where(r => string.Equals(r.Method, expected.Method, StringComparison.OrdinalIgnoreCase)
&& r.Path == expected.Path)
.ToList();

matches.Should().HaveCount(expected.Count,
$"{side} {row.ToolMethod}: {expected.Method} {expected.Path}");

foreach (var fragment in expected.BodyContains)
matches.Should().Contain(r => r.Body != null && r.Body.Contains(fragment),
$"{side} {row.ToolMethod}: payload must contain {fragment}");
}
Comment thread tests/SSW.TimePro.Cli.Integration/Mcp/McpStdioClient.cs
Comment on lines +64 to +65
await client.InitializeAsync(ct);
return client;
Comment thread scripts/e2e/mcp_smoke.py Outdated
Comment on lines +181 to +185
raise SmokeFailure(
f"{failure}\n"
" The MCP write path is not yet the CLI's: create_timesheet omits the sell price "
"the CLI resolves from the client rate, so TimePro rejects it."
) from failure
@jernejk jernejk mentioned this pull request Sep 12, 2026
Closed
jernejk and others added 3 commits September 13, 2026 00:53
@jernejk @claude
test(mcp): regression harness for tools/list, tool results, CLI parit…
7afec79 
…y and staging stdio

Locks the observable MCP contract before the CLI/MCP unification slices touch it.

- Golden tools/list taken from the real stdio server, accounting off (18) and on (47)
- Per-tool result goldens for all 47 tools plus generated empty and API-error cases
- tools/call envelopes over stdio for six tools, proven identical to the direct calls
- CLI/MCP parity runner with per-case permitted differences and write-traffic assertions
- Reflection test mapping every tool to a registered CLI command, with a shrink-only allowlist
- scripts/e2e/test-mcp-smoke.sh: staging stdio gate with non-production check and cleanup

ConfigPaths honours TIMEPRO_CLI_CONFIG_DIR so a child MCP host can run against an
isolated config instead of the developer's own tenants and feature flags.

Closes #63

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jernejk @claude
test(mcp): harden the smoke driver, pin the CLI binary, drop real pro…
8583666 
…ject ids

Review fixes on the regression harness.

- Smoke driver reads stdout on its own thread through a queue, so a silent MCP host times
  out and is terminated instead of blocking forever on an uninterruptible readline
- Cleanup moved into finally: locates the run's uniquely marked row, deletes only an
  unambiguous single match, verifies absence, and exits non-zero when unresolved. A dead
  host is recovered over a fresh session
- create_timesheet no longer has a canned sell-price diagnosis appended to every failure
- Smoke takes the project and iteration from the environment with no committed default,
  and no longer echoes project or iteration details
- CliBinary resolves the exact output for the test's own configuration and target
  framework instead of the newest assembly anywhere under bin
- UpdateLeave dry-run row forbids PUT /api/leave/, matching the CreateLeave row
- Parity notes carry the missing behaviour, not issue numbers
- CliMcpParityTests is serialised; CliRunner swaps the process-wide console

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jernejk @claude
test(mcp): follow main's leave services, and cover the new list filter
b6e888b 
Rebase fixes plus the goldens main's behaviour changes moved.

`LeaveUpdateService` now takes a `LeaveLookup` and `LeaveMcpTools` a `LeaveListService`, so
the harness host and the in-process CLI runner wire both.

Three golden changes, each caused by a behaviour change already on main:

- `check_week` description gained "and pendingSuggestions"
- `get_leave_entries` filter description gained ALL
- `CheckWeek` result and both parity sides gained a top-level `pendingSuggestions`, which
  appears identically on the CLI and MCP sides, so the parity verdict is unchanged

The list-filter change claimed the tool output is unaffected for the existing filters.
Verified rather than assumed: `GetLeaveEntries.past.json` is byte-identical to the UPCOMING
golden (same sha256). New cases cover ALL, whose merge and de-duplication now run against a
distinct PAST row, and an unknown filter, which returns the validation message without
touching the API.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jernejk
jernejk force-pushed the test/mcp-regression-harness branch from 39e35a1 to b6e888b Compare September 12, 2026 14:53
@jernejk jernejk mentioned this pull request Sep 12, 2026
Closed
@jernejk
jernejk merged commit 9797bdd into main Sep 12, 2026
1 check passed
@jernejk
jernejk deleted the test/mcp-regression-harness branch September 12, 2026 14:56
This was referenced Sep 12, 2026
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

MCP regression harness: golden tools/list, per-tool result goldens, CLI/MCP parity runner, staging stdio smoke

2 participants

@jernejk