Skip to content

Reporters and Reports

Pavel Erokhin edited this page Nov 21, 2026 · 3 revisions

Русская версия

ctxray test shows its results in the terminal and can write them to files for people, for CI, and for other programs. This page names each reporter, the files it writes, and how ctxray keeps secrets out of all of them.

Reporters

Reporter Writes For
console nothing: the matrix of layers and harnesses, then the failures, in the terminal you, while you work
json report.json scripts and other tools
html index.html, plus report.json beside it a person reading the whole run, offline
junit junit.xml the test tab of GitLab, Jenkins, and other CI
allure allure-results/ Allure dashboards
markdown summary.md a merge request comment or a job summary

Pick them with --reporter, comma-separated:

ctxray test --reporter console,html,junit

The flag beats CTXRAY_REPORTER, which beats reporters in the config:

export default defineConfig({
  reporters: ["console", "html"],
});

With none of the three, a local run uses console alone and writes no files. In CI (see CI Integration) the default is console, json, junit, markdown, html, so a pipeline gets every file it can show. A config that sets reporters wins in CI too. An unknown name stops the run with exit code 2.

The short summary at the end, with the layer counts and the files written, always prints, whichever reporters you chose.

The report folder

Files go to .ctxray/report/ under the project root. --report-dir or report.dir in the config moves it, relative to the project root:

export default defineConfig({
  report: { dir: "build/ctxray" },
});
File Reporter
report.json json or html
index.html html
junit.xml junit
allure-results/<uuid>-result.json, categories.json, environment.properties allure
summary.md markdown

Each run overwrites the files, and allure empties allure-results/ first. ctxray init adds .ctxray/ to your .gitignore, so reports stay out of commits. Snapshots, cassettes, and ctxray.baseline.json live elsewhere, next to your tests and at the project root, because you commit them.

The HTML report

index.html is one self-contained file: styles, script, data, and messages are inside it. Open it from disk, attach it to a pipeline as an artifact, or mail it. It needs no server and no network.

The report has tabs: Overview, Tests, Context, and, when the run has them, Activation (the statistics of L6 and L7), Coverage, and Diff (with --base). A link like index.html#context opens a tab directly, and the hash names are overview, tests, context, evals, coverage, diff. Your choice of tab, harness, theme, and language stays in the browser's local storage.

Report languages

The HTML report can carry several languages and switch between them in place. --report-lang takes a comma-separated list, or all for every language ctxray speaks:

ctxray test --reporter html --report-lang en,ru

The flag beats CTXRAY_REPORT_LANG, which beats report.langs in the config. By default the report carries only the CLI language (Localization). With all, the CLI language comes first. An unknown language stops the run with exit code 2.

The first language matters most. The report opens in it, and it's the language of the texts ctxray writes into report.json, junit.xml, allure-results/, and summary.md. With more than one language, the report shows a language switcher.

show-report

ctxray show-report serves the report folder at http://127.0.0.1:<port>/, reachable only from your machine:

ctxray show-report --open

Leave it open while you run ctxray test --reporter html (or json) in another terminal. Each time a run writes a new report.json, every open page reloads. The page is built from report.json on each request, so even a run that wrote only json shows up. Stop the server with Ctrl+C. Without a report.json, it says so and exits with code 2. The options are on Commands.

The JSON report

report.json holds everything the other reporters show, so a script can build its own view. Its schema is versioned: schemaVersion is 1 today, and a breaking change raises it. The JSON Schema comes with every release:

https://github.com/MairwunNx/ctxray/releases/latest/download/report.schema.json

The top level:

{
  "schemaVersion": 1,
  "ctxray": "0.1.0",
  "createdAt": "2026-11-16T19:48:18.790Z",
  "langs": ["en"],
  "project": { "name": "…", "root": "…" },
  "run": { "backend": { "kind": "mock" }, "layers": ["L1", "L2", "L3"], "duration": 74 },
  "summary": { "passed": 2, "failed": 0, "skipped": 0, "inconclusive": 0, "flaky": 0, "byLayer": { … } },
  "harnesses": [ … ],  // id, title, modes, the starting context a capture saw, and the loadout L3 predicts at the root
  "tests": [ … ],      // each test with one cell per harness
  "broken": [ … ],     // test files that failed to load
  "coverage": [ … ]    // which activations your tests cover, per harness
  // "base": { … }     with --base: the loadout diff
  // "baseline": { … } with ctxray.baseline.json: how many findings it knew
}

A cell is one test on one harness: its status (passed, failed, skipped, inconclusive, flaky), the skip reason (unsupported, filtered, unavailable, backend, budget), attempts, duration in milliseconds, tokens, and the failure with its message. A statistical test adds stats, each with a Wilson 95% interval. The schema describes every field.

JUnit, Allure, and Markdown

  • JUnit. One <testsuite> per test file, and ctxray's own checks as ctxray lint, ctxray policy, and ctxray context. Each test on each harness is a <testcase> named [L3] <title> › <harness>. An inconclusive cell becomes <skipped>, and a flaky one passes with the number of attempts in <system-out>.
  • Allure. One result per test and harness, labelled with the file, the suite, the layer, and the harness. categories.json groups “Inconclusive statistics” and “Resolver drift”, and environment.properties records the ctxray version, the backend, and the harnesses.
  • Markdown. A heading with the counts, a table per layer, up to 20 failures, the starting context of each harness with its tokens, the diff against --base when there is one, and a link to the HTML report.

GitHub job summary

When GITHUB_STEP_SUMMARY is set and markdown is among the reporters, ctxray also appends summary.md to that file. On GitHub Actions the summary then shows on the run's page with no extra step.

Traces

L4 and up launch the harness and record every request it sends to the model (Capture Modes). Each exchange is one line in a JSONL trace, in a temporary folder outside your project:

$TMPDIR/ctxray-traces-<id>/<harness>-<n>.jsonl

ctxray deletes the folder when the run ends, unless a test at L4 or above failed. Then the trace stays, and the failure shows its path under “Trace”, so you can read exactly what the harness sent. The logs of MCP servers that ran behind ctxray get the same redaction.

Redaction

ctxray redacts before it writes anything to disk: reports, traces, cassettes, and MCP logs. The console output passes the same redaction, so a report holds what the terminal showed. A secret becomes a marker that names the pattern that caught it:

Authorization: ⟦redacted:secret-field⟧
token = ⟦redacted:github-token⟧

ctxray runs three sets of patterns, in this order.

Built-in patterns. The more specific ones run first:

Id Catches
private-key PEM private key blocks
jwt JSON Web Tokens
anthropic-key sk-ant-…
openai-key sk-…, sk-proj-…, sk-svcacct-…, sk-admin-…
stripe-key sk_live_…, rk_test_… and the like
google-api-key AIza…
github-token ghp_…, gho_…, ghu_…, ghs_…, ghr_…, github_pat_…
gitlab-token glpat-…
npm-token npm_…
aws-access-key AKIA…, ASIA…
slack-token xoxb-… and the other xox?- tokens
telegram-bot-token Telegram bot tokens
bearer Bearer <token>
url-credentials the user:password in https://user:password@host
email email addresses
phone phone numbers with a + country code, and the Russian 8 (XXX) XXX-XX-XX form, so version numbers don't match
card card numbers of 13 to 19 digits that pass the Luhn check

In JSON, the whole value of a field named authorization, proxy-authorization, cookie, set-cookie, password, passwd, client_secret, or anything ending in api_key / api-key becomes ⟦redacted:secret-field⟧, whatever it looks like.

gitleaks rules. ctxray ships the 217 rules of gitleaks v8.30.1 (MIT), with their keywords, entropy thresholds, and allowlists. Only the secret part of a match is replaced, and the marker carries the gitleaks rule id.

Your patterns. Add regular expressions in redact.patterns. Their marker is ⟦redacted:custom⟧:

export default defineConfig({
  redact: { patterns: [/ACME-[0-9]{8}/, "internal-[a-z]+\\.corp"] },
});

A string must be a valid regular expression, or the config fails to load. The built-in patterns can't be turned off, and a config that extends another can add patterns but never drop inherited ones. Configuration has the rest of the config.

Clone this wiki locally