From fba8a54f9aa1a9be3512afeb94165eb5686b7d67 Mon Sep 17 00:00:00 2001 From: Remy DUTHU Date: Fri, 22 May 2026 11:28:55 +0200 Subject: [PATCH] docs(test-insights): Document mergify tests show and tests APIs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mergify CLI #1382 shipped `mergify tests show` so developers and AI coding agents (Claude Code, Cursor, …) can decide whether a failing test is a real regression or known flakiness without parsing dashboards. Until now the command was undiscoverable from the docs site — agents that ran it from a skill could not link users to a reference, and developers had no entry point besides `--help`. Adds a dedicated `/test-insights/cli` guide covering single- and multi- test lookups, glob patterns, pipeline/job filters, JSON output, and the 0/1/6 exit-code contract (clap reserves `2`). A second half walks through agent recipes for routing failures to retry vs. fix vs. ignore based on exit code alone, plus a Claude Code skill snippet and Cursor project rule. The two backing endpoints (`GET …/search/tests`, `GET …/tests/{test_id}`) are auto-generated into the API reference from the engine's OpenAPI schema, so the guide links to `/api/test-insights` rather than duplicating the schema. `/tests/{test_id}/stats/{stat}` stays out of scope: it is `include_in_schema=False` server-side and not used by the CLI. The guide intentionally documents the shipped surface, which diverged from the MRGFY-7167 spec during MRGFY-7166 implementation: the command is `mergify tests show` (not `mergify ci tests show`), there is no `--quiet` flag, exit codes are `0/1/6` (not `0/1/2`), and the search endpoint is `GET` with repeated query params (not `POST` with a JSON body). Documenting reality beats documenting the plan. Fixes: MRGFY-7167 Co-Authored-By: Claude Opus 4.7 (1M context) Change-Id: I54b1908accda0ad2d2bf538af10e5960d76ad9b2 --- src/content/docs/cli.mdx | 1 + src/content/docs/test-insights.mdx | 5 ++ src/content/docs/test-insights/cli.mdx | 83 ++++++++++++++++++++++++++ src/content/navItems.tsx | 1 + 4 files changed, 90 insertions(+) create mode 100644 src/content/docs/test-insights/cli.mdx diff --git a/src/content/docs/cli.mdx b/src/content/docs/cli.mdx index 5b249e1305..69f11553dc 100644 --- a/src/content/docs/cli.mdx +++ b/src/content/docs/cli.mdx @@ -67,3 +67,4 @@ mergify --token your_token_here | `mergify queue` | Monitor and control merge queue | [Monitoring](/merge-queue/monitoring) | | `mergify config` | Validate and simulate config | [Config](/configuration/file-format#validating-with-the-cli) | | `mergify ci` | Upload and analyze CI results | [CI Insights](/ci-insights) | +| `mergify tests show` | Search test health by name | [Test Insights CLI](/test-insights/cli) | diff --git a/src/content/docs/test-insights.mdx b/src/content/docs/test-insights.mdx index 31b3f771c2..2028dd65f2 100644 --- a/src/content/docs/test-insights.mdx +++ b/src/content/docs/test-insights.mdx @@ -29,6 +29,11 @@ of a test reliability problem: to unblock CI without removing them. Tests keep running, but their failures no longer block merges. +You can also search test health directly from the terminal or from AI coding +agents with the [Test Insights CLI](/test-insights/cli). This is useful when +an agent needs to decide whether a failing test is a real regression or +known flakiness. + ## Key concepts - **Flaky test**: A test that produces different results on the same commit. diff --git a/src/content/docs/test-insights/cli.mdx b/src/content/docs/test-insights/cli.mdx new file mode 100644 index 0000000000..a5f54fc02c --- /dev/null +++ b/src/content/docs/test-insights/cli.mdx @@ -0,0 +1,83 @@ +--- +title: CLI +description: Interact with Test Insights data from the terminal — search for tests, branch on health via exit codes, and chain into AI coding agents like Claude Code and Cursor. +--- + +The [Mergify CLI](/cli) lets you interact with Test Insights data from +the terminal. It's handy when you want to triage a failing test +quickly, or when an AI coding agent needs to decide whether to retry, +fix, or ignore a failure. + +## `mergify tests show` + +Search the Test Insights catalog by test name and print health, ratios, +and the most recent failure context for each match. + +```bash +mergify tests show -r owner/repo "tests/auth/test_login.py::test_login_timeout" +``` + +Names accept glob patterns (`*`, `?`), and several can be passed in one +call. Add `--json` for a machine-readable payload. Run +`mergify tests show --help` for the full list of flags, and see the +[API reference](/api/test-insights) for the underlying endpoints and +response schemas. + +### Exit codes + +The exit code reflects the worst health observed across the matched +tests, so callers can branch without parsing output: + +| Code | Meaning | +|---|---| +| `0` | All matched tests are healthy or unknown, or nothing matched. | +| `1` | At least one matched test is flaky. | +| `6` | At least one matched test is broken. | + +Exit code `2` is reserved for CLI argument errors. + +## AI coding agent recipes + +The exit-code contract is designed to be chained from agents that ran +a test, saw it fail, and need to decide what to do next. + +### Should the agent retry, fix, or ignore? + +```bash +if mergify tests show -r owner/repo "$FAILED_TEST" --json > /tmp/health.json; then + status=0 +else + status=$? +fi + +case $status in + 0) echo "Healthy or unknown: investigate as a real regression." ;; + 1) echo "Known flaky: retry before touching code." ;; + 6) echo "Broken: fix is required; skip the retry loop." ;; +esac +``` + +### Claude Code + +Add a [skill](https://docs.claude.com/en/docs/claude-code/skills) or a +prompt snippet that points the agent at the CLI: + +```md +When a test fails locally or in CI, before attempting a fix, run: + + mergify tests show -r OWNER/REPO --json "" + +Then branch on the exit code: +- 0, non-empty payload → healthy in Mergify; investigate as a real + regression. +- 0, empty payload → unknown to Mergify; treat as a new test or one on + a non-default branch. +- 1 → known flaky. Rerun once; only investigate if it fails again. +- 6 → known broken (pre-existing). Surface to the user instead of + attempting a fix. +``` + +### Cursor + +In Cursor, add the same guidance to a project rule +(`.cursor/rules/test-triage.mdc`) so it auto-loads in agent sessions. diff --git a/src/content/navItems.tsx b/src/content/navItems.tsx index d7515455fb..df28cb8115 100644 --- a/src/content/navItems.tsx +++ b/src/content/navItems.tsx @@ -123,6 +123,7 @@ const navItems: NavItem[] = [ { title: 'Detection', path: '/test-insights/detection', icon: 'lucide:search' }, { title: 'Mitigation', path: '/test-insights/mitigation', icon: 'lucide:radiation' }, { title: 'Quarantine', path: '/test-insights/quarantine', icon: 'lucide:radiation' }, + { title: 'CLI', path: '/test-insights/cli', icon: 'lucide:terminal' }, { title: 'Test Frameworks Setup', path: '/test-insights#test-framework-configuration',