Skip to content

Writing Tests

Pavel Erokhin edited this page Nov 16, 2026 · 2 revisions

Русская версия

A ctxray test states what your agent should have in context and checks it on every harness you use. You write it once, and ctxray runs it against Claude Code, Codex, and the rest, one cell per harness. This page starts with a single rule check and grows it into a test that asks a real model a hundred prompts. The reference at the end lists every function, option, and matcher.

Each kind of test checks one layer of the Loadout Stack: test.resolve checks L3, test.assemble L4, test.wire L5, test.activate L6, and test.scenario L7. L1 Lint and L2 Policy need no tests: ctxray runs them on every ctxray test, as Policies explains.

Test files

ctxray picks up every file named *.ctxray.ts in the repository. .js, .mjs, .cjs, .mts, and .cts work too, and *.ctxray.jsonc (or .json) holds declarative tests. Files your .gitignore hides, node_modules included, stay out. Run ctxray test from a subfolder, and only the tests under it run. ctxray reads the config from the folder it runs in, so point a subfolder run at the root one with --config ../ctxray.config.ts.

A test file imports its API from @mairwunnx/ctxray:

import { corpus, discover, expect, suite, test } from "@mairwunnx/ctxray";

ctxray serves this module to the file itself, so a test runs with no node_modules at all. The npm package only gives your editor the types:

npm install -D @mairwunnx/ctxray

When the installed package and the ctxray executable differ in version, the run says so: “node_modules holds @mairwunnx/ctxray 0.1.0, and the tests run on this ctxray, 0.2.0”. Your editor then checks the code against an API that ctxray may not run.

ctxray init writes a first test to ctxray/loadout.ctxray.ts.

Your first test: L3

L3 predicts what a harness loads from the project's files alone. It launches nothing and answers in milliseconds. The test below comes from ctxray's own fixtures: payment rules must load when the agent works on payment code.

import { expect, suite, test } from "@mairwunnx/ctxray";

suite("payments", { fixture: "../payments" }, () => {
  test.resolve("payments rules attach to payment sources", { requires: ["pathRules"] }, async ({ resolve }) => {
    const l = await resolve({ cwd: "app", files: ["app/src/main/kotlin/payments/Charge.kt"] });
    expect(l.rules).toInclude([".claude/rules/kotlin.md", ".claude/rules/payments.md"]);
    expect(l.rules).not.toInclude(".claude/rules/web.md");
    expect(l.rules.get(".claude/rules/payments.md").reason).toBe("glob:**/payments/**/*.kt");
  });
});

suite groups tests under a name. Its fixture names the project the tests look at, relative to the test file. Without it, the tests look at the repository itself, and nested suites inherit their parent's fixture.

resolve() takes the folder the agent starts in (cwd) and the files it works with (files), both relative to the project root. It returns the loadout the harness would build: rules, skills.listed, skills.loaded, mcp.servers, commands, and tokens. Every list also finds an entry by path or name: l.rules.get(".claude/rules/payments.md") returns the rule with its activation, reason, and tokens. An id the list doesn't hold fails the test and names what the list does hold.

Codex has no rules that load by path, so requires: ["pathRules"] skips it instead of failing it. Run the layer:

ctxray test --layer L3
 ctxray 0.1.0  ·  2 harnesses  ·  backend: mock

 payments  payments
   L3 resolve  payments rules attach to payment sources
               claude-code ✓   codex ○ needs pathRules

When an expectation fails, ctxray shows what it expected and what it found:

 ✗ claude-code › demo › web rule stays out
     expect(received).toInclude(expected)
     Missing   .claude/rules/web.md
     Received  CLAUDE.md, AGENTS.md, .claude/rules/style.md, app/CLAUDE.md, app/AGENTS.md, …

Different harnesses, different files

Harnesses read different files. Claude Code takes CLAUDE.md, Codex takes AGENTS.md. expect.on holds expectations for some harnesses only:

test.resolve("instructions of the app load in app", async ({ resolve }) => {
  const l = await resolve({ cwd: "app" });
  expect(l.rules).toInclude("app/AGENTS.md");
  expect.on("codex", () => expect(l.rules).toInclude("AGENTS.md"));
  expect.on("claude-code", () => expect(l.rules).toInclude(["CLAUDE.md", "app/CLAUDE.md"]));
});

The body also gets harness, the id of the harness it runs on. ctxray init uses it to pick each harness's root file:

const rootRules: Record<string, string> = { "claude-code": "CLAUDE.md", codex: "AGENTS.md" };

suite("loadout", () => {
  test.resolve("the root rules file loads for every file", async ({ harness, resolve }) => {
    const loadout = await resolve({ files: ["README.md"] });
    expect(loadout.rules).toInclude(rootRules[harness]);
  });
});

Harnesses lists which files each harness reads and which capabilities it has.

Snapshots

A snapshot holds a whole loadout, so any change to it shows up in review. toMatchLoadoutSnapshot() compares the loadout with the stored one:

const l = await resolve({ cwd: "app" });
expect(l).toMatchLoadoutSnapshot();

The first run fails with “Snapshot … isn't written yet”, because a test run never writes your files by itself. Write the snapshots on purpose and commit them:

ctxray test -u

They land next to the test file: payments.ctxray.ts keeps its snapshots in __snapshots__/payments.ctxray.snap.json. A key holds the suite, the test, the harness, and its version: payments › instructions of the app load in app › claude-code@static. L3 reads files only, so its version is static. An L4 or L5 snapshot carries the harness version and the model, so a harness upgrade changes the key by itself.

An L3 snapshot keeps each rule with its activation, reason, and tokens, each skill, the MCP servers, the commands, and the tokens per component. When the loadout changes, ctxray names each value that moved:

 ✗ claude-code › payments › instructions of the app load in app
     The loadout differs from snapshot “payments › instructions of the app load in app › claude-code@static”
     Changed  rules › app/CLAUDE.md › tokens  23 → 40
     Changed  tokens › rules  115 → 132

An L4 or L5 snapshot keeps names only: system prompt sections, rules, skills, MCP servers with their tools, and commands. Captured token counts move with the date and the paths in the request, the names don't.

L4: the context a session starts with

L4 launches the real harness against ctxray's mock model and takes its first request. You see what L3 can't: the system prompt, the built-in tools, and the MCP tool schemas. The mock model costs nothing and answers the same way every time. The harness has to be on your PATH or pinned in ctxray.lock, as Harnesses describes.

test.assemble("context budget", async ({ assemble }) => {
  const l = await assemble({ cwd: "app" });
  expect(l.tokens.total).toBeLessThan(25_000);
  expect(l.tokens.byMcpServer("jira")).toBeLessThan(3_000);
  expect(l.mcp.servers).toEqual(["github", "jira"]);
  expect(l).toMatchLoadoutSnapshot();
});

assemble() returns the captured loadout. On top of what L3 shows, it has system (the sections of the system prompt), mcp.tools, duplicates, the harness version and model, and tokens.total. tokens.byMcpServer(name) gives what one server costs once its tools load. tokens.exact is true only when a provider counted the tokens. Otherwise they're a local estimate, and the report marks them ≈.

toEqual compares an entry list with a list of ids in order, so ["github", "jira"] reads as the server names.

You don't need a test to hold the starting context to a limit. Each ctxray test with L4 in its layers launches every harness once and checks the starting context against the maxTokens of your policies.

L5: what loads during a session

In L5 your test plays the model. It tells the harness to read a file, waits for the next request, and checks what the harness added to the context:

test.wire("nested instructions load on read", { requires: ["pathRules"] }, async ({ session }) => {
  await session.model.calls("Read", { path: "app/src/main/kotlin/payments/Charge.kt" });
  const next = await session.nextRequest();
  expect(next.loadout.rules).toInclude("app/src/main/kotlin/payments/AGENTS.md");
  expect(next).toAgreeWithResolver();
});

session.model.calls(tool, input) answers the harness's latest request with a tool call. Read with a path and Skill work on every harness: ctxray turns them into the harness's own tool. session.model.says(text) answers with text. session.nextRequest() waits for the request that follows, and session.firstRequest() returns the first one.

A request holds loadout (the captured loadout at that point), added (the rules and skills this request brought in), and predicted (what L3 expects for the same folder and the files read so far). Two matchers work on it:

  • toLoadRule(rule) checks that the rule is in context at this request. With .not, it checks that this request didn't bring the rule in, so a rule an earlier read loaded doesn't fail a later read.
  • toAgreeWithResolver() checks that the harness loaded the rules and project skills L3 predicts. When they differ, ctxray reports resolver drift: the harness changed its behaviour, and ctxray's adapter for it needs an update.

session.events() returns the harness's own event stream, such as Claude Code's stream-json. startSession({ cwd, prompt }) opens one more session in the same test. ctxray ends every session when the test body returns. A harness that exits with an error, or that tries to reach the internet past ctxray, fails the test.

One test, many cases: .each

Every test kind has .each. Over a list of objects, it registers one test per item, and {field} in the name takes the item's value:

test.resolve.each([
  { file: "app/src/main/kotlin/payments/Charge.kt", rule: ".claude/rules/payments.md" },
  { file: "app/src/main/kotlin/web/Page.kt", rule: ".claude/rules/kotlin.md" },
])("{rule} loads for {file}", { requires: ["pathRules"] }, async ({ file, rule, resolve }) => {
  expect((await resolve({ files: [file] })).rules).toInclude(rule);
});

Rules ctxray finds for you

A list of rule paths goes stale the day someone adds a rule. discover.rules() selects rules and skills by what they are instead:

Selector Picks
owner Rules whose owner in CODEOWNERS matches, like "@acme/payments", or a list of owners. ctxray looks in .github/CODEOWNERS, .gitlab/CODEOWNERS, CODEOWNERS, and docs/CODEOWNERS
under Rules inside a folder or matching a glob, relative to the project root
activation always, path, intent, or manual, or a list of them

.where() narrows a selection further. The selection resolves inside each test, because which rules exist depends on the harness. A selection that matches nothing fails the test: a typo in a selector would otherwise pass as a test that checks nothing.

const payments = discover.rules({ owner: "@acme/payments" });

suite("payments rules", { fixture: "../payments" }, () => {
  test.resolve.each(payments.where({ activation: "path" }))(
    "{rule.path} attaches to matching files",
    { requires: ["pathRules"] },
    async ({ rule, resolve }) => {
      for (const file of rule.matchingFiles({ sample: 3 })) {
        expect((await resolve({ files: [file] })).rules).toInclude(rule);
      }
      const other = rule.nonMatchingFile();
      if (other) expect((await resolve({ files: [other] })).rules).not.toInclude(rule);
    },
  );
});

.each over a rule set gives each test a rule. rule.matchingFiles({ sample: 3 }) returns three project files the rule applies to, spread evenly over the list, and rule.nonMatchingFile() returns one it doesn't. toInclude takes the rule, a rule set, a path, or a list of paths.

The same selection works in L4 and L5. A rule set's root is the folder its rules live in, a good place to start the agent:

test.assemble("always rules present in payments zone", async ({ assemble }) => {
  const always = payments.where({ activation: "always" });
  const l = await assemble({ cwd: always.root });
  expect(l.rules).toInclude(always);
});

L6: does the model pick the skill?

A skill loads when the model decides it fits the prompt. Only a model that answers by itself can tell whether it does, so L6 runs the prompt several times and counts:

test.activate(
  "refund prompt → refunds skill",
  { prompt: "Make a partial refund of 40 EUR for order 123.", samples: 5, requires: ["skills"] },
  ({ runs }) => {
    expect(runs).skill("refunds").toActivate({ min: 0.5 });
    expect(runs).skill("payouts").not.toActivate({ max: 0.5 });
  },
);

Each sample is a whole session: the harness gets the prompt and works until it stops, or until it reaches the turns limit and ctxray answers in the model's place. Then the body sees every run. expect(runs).skill(name), .rule(path), and .anyOf(set) count the runs that loaded the skill, the rule, or any of a set.

L6 needs a model. With the default mock backend, the test shows “needs --backend replay or real” and skips. Backends and cassettes explains both.

The confidence interval

Ten runs out of ten don't prove a 90% rate. ctxray puts a 95% Wilson interval around the rate and judges the interval, not the bare rate:

Four results on a scale from 0 to 1. 16 of 16 passes min 0.8, because the interval’s lower bound is 0.81. 14 of 16 is inconclusive: the rate 0.88 is above 0.8, but the lower bound is 0.64. 9 of 16 fails, because the rate 0.56 is below 0.8. 0 of 16 passes max 0.2, because the upper bound is 0.19. A perfect run needs 4 runs for 0.5, 16 for 0.8, 35 for 0.9, and 73 for 0.95.

  • toActivate({ min }) passes when the interval's lower bound reaches min.
  • .not.toActivate({ max }) passes when its upper bound stays at or under max.
  • The test fails when the rate itself is on the wrong side of the threshold.
  • Otherwise the samples can't tell yet, and the test is inconclusive. ctxray names how many samples in all would settle it if the rate held.
skill(refunds) loaded at a rate of 1.00 [0.72–1.00, 95% CI]: too few samples to tell
  Samples in all to tell  skill(refunds): 35

Without a threshold, toActivate() means { min: 0.8 } and .not.toActivate() means { max: 0.2 }. Without samples, the test makes enough runs to prove them: 16 for each group of prompts, which is the single prompt, the positive list, and the negative list. ctxray splits a group's runs between its prompts and rounds up, so two positive prompts run 8 times each. An explicit samples wins and counts per prompt.

Count the runs before you pick a stricter threshold. Even when every run loads the skill, the lower bound reaches 0.5 at 4 runs, 0.8 at 16, 0.9 at 35, and 0.95 at 73. The activate section of the config sets the thresholds and the run count for the whole project. Raise its min to 0.9, and the default count grows to 35 by itself. A threshold you pass to toActivate({ … }) still beats the config.

An inconclusive test doesn't fail the run. Sessions that broke (the harness crashed, a cassette ran out) say nothing about the model, so ctxray leaves them out of the count. When every one of them broke, the first failure becomes the test's.

A corpus of prompts

One prompt tells you about one wording. A corpus gives the test many: prompts that should pick the skill (positive) and borderline ones that shouldn't (negative). Put them in a module whose default export holds both lists, or in a JSON file:

// tests/prompts/payments.ts
export default {
  positive: ["Refund order 123 in part", "Pay the seller out for last week"],
  negative: ["Rename the Android build flavour"],
};

corpus() loads the file, relative to the test file, before any test runs. runs.positive and runs.negative hold the runs of each list. Each list gets its own 16 runs, so below the two positive prompts run 8 times each and the negative one 16 times:

import { corpus, discover, expect, suite, test } from "@mairwunnx/ctxray";

const payments = discover.rules({ owner: "@acme/payments" });
const prompts = corpus("./prompts/payments.ts");

suite("payments rules", { fixture: "../payments" }, () => {
  test.activate(
    "payments intents activate payments skills",
    { positive: prompts.positive, negative: prompts.negative },
    ({ runs }) => {
      expect(runs.positive)
        .anyOf(payments.where({ activation: "intent" }))
        .toActivate();
      expect(runs.negative)
        .anyOf(payments.where({ activation: "intent" }))
        .not.toActivate();
    },
  );
});

Writing a hundred prompts by hand is slow. ctxray gen-prompts asks a model to write them from the description of each skill and each rule the model picks by description:

ctxray gen-prompts --from-descriptions --out tests/prompts/intents.json

It writes --count positive and as many negative prompts per rule or skill (5 by default) with claude-opus-5-5, through your upstream and key. Choose another model with --model, but not the one under test: a model that writes its own exam passes it too easily, and ctxray warns when it does. Read the file before a test uses it. The corpus stays as it is until you write it again, and an existing file needs --force.

{
  "model": "claude-opus-5-5",
  "generated": "2026-11-16",
  "positive": ["…"],
  "negative": ["…"],
  "by": { ".claude/skills/refunds/SKILL.md": { "positive": ["…"], "negative": ["…"] } }
}

corpus() reads the top-level positive and negative lists. by shows which prompts came from which file.

L7: scenarios

A scenario gives the agent a task and checks what it did. The body runs once per run, and gets that run:

test.scenario(
  "a partial refund goes through the refunds skill",
  { prompt: "Make a partial refund of 40 EUR for order 123.", samples: 3, min: 0.6, requires: ["skills"] },
  ({ run }) => {
    expect(run.skills).toInclude("refunds");
    expect(run.changed).not.toInclude("app/build.gradle.kts");
    // Tool names are the harness's own: Claude Code reads with Read, Codex with exec_command.
    expect.on("claude-code", () => expect(run.tools).toInclude("Read"));
  },
);

A run holds the prompt and its sample number, the skills and rules that reached the model, the tools it called in order, the files it changed, its turns, whether ctxray cut the last turn, and the loadout of its last request. turns defaults to 30 here. Without min, every run must pass. With min, the share of passing runs goes through the same interval as L6.

Backends and cassettes

A backend decides who answers the harness:

Backend Who answers Runs
mock Your test, or nobody. The default L3, L4, L5. L6 and L7 skip
real The model provider, or your gateway L3–L7
replay Answers recorded earlier with real L3–L7

L4 and L5 always run on the mock model, whatever backend you pick: it's free and deterministic.

A real run costs money, so record it once and replay it on every merge request:

ctxray test --layer L6-L7 --backend real --record
ctxray test --backend replay

--record writes the model's answers next to the test file, one file per harness: __cassettes__/activate.ctxray.ts.claude-code.json. Commit them. ctxray redacts secrets before a cassette reaches disk. Each tape hangs on the test's name, the prompt's position, and the sample number. Rename a test or reorder its prompts, and record again.

A replay runs the harness for real and answers each turn from the tape. When the harness goes another way than it did while recording, the run fails with a cassette miss, which usually means the harness or your files changed. Record again with --backend real --record.

A real run reads the provider key from CTXRAY_UPSTREAM_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY, and stops spending at the budget. Configuration covers upstream, budget, prices, and named backends, and Enterprise covers gateways.

Declarative tests

Simple checks don't need TypeScript. A *.ctxray.jsonc file holds suites of tests as data, and ctxray turns each into the same test the DSL registers:

{
  "$schema": "https://github.com/MairwunNx/ctxray/releases/latest/download/tests.schema.json",
  "suites": [
    {
      "name": "payments",
      "fixture": "../payments",
      "tests": [
        {
          "name": "kotlin rule on Kotlin files",
          "layer": "L3",
          "requires": ["pathRules"],
          "files": ["app/src/main/kotlin/payments/Charge.kt"],
          "expect": {
            "rules": { "include": [".claude/rules/kotlin.md"], "exclude": [".claude/rules/web.md"] },
            "tokens": { "total": { "lessThan": 25000 } }
          }
        },
        {
          "name": "nested instructions load on read",
          "layer": "L5",
          "requires": ["pathRules"],
          "steps": [
            {
              "read": "app/src/main/kotlin/payments/Charge.kt",
              "expect": { "rules": { "include": ["app/src/main/kotlin/payments/AGENTS.md"] }, "agreeWithResolver": true }
            }
          ]
        },
        {
          "name": "refund prompt → refunds skill",
          "layer": "L6",
          "corpus": "./prompts/intents.json",
          "expect": [{ "skill": "refunds", "min": 0.8, "runs": "positive" }]
        }
      ]
    }
  ]
}

Each test names its layer. Lists take include, exclude, and equal. Numbers take lessThan, atMost, moreThan, and atLeast. L3 and L4 expect rules, skills.listed, skills.loaded, mcp.servers, commands, tokens, and snapshot. An L5 step does one read, call, or say and may expect what the next request holds. An L6 expectation names a skill, rule, or anyOf with min or max, and runs picks all, positive, or negative. L7 expects changed, tools, skills, and rules. The tests.schema.json release asset holds the full schema, so your editor completes it.

Running tests

ctxray test                              # every layer the backend allows, every configured harness
ctxray test --layer L1-L3                # files only: no harness launches
ctxray test --harness claude-code        # one harness
ctxray test --filter "payments rules"    # tests whose name holds this text
ctxray test --watch                      # again after each change

--filter matches the full name, suites included, ignoring case. ctxray's own checks of L1, L2, and the L4 starting context run whatever the filter says, so limit them with --layer. --watch (-w) runs again a moment after you save a file, Enter runs right away, and q stops. Changes in .git, node_modules, .ctxray, __snapshots__, __cassettes__, and the report folder start no run.

Every cell ends with a status:

Status Means Fails the run
passed Every expectation held No
failed An expectation failed, the body threw, or the time ran out Yes
flaky It failed, then passed on a retry No
inconclusive L6 or L7 had too few samples to tell No
skipped filtered by --layer or --filter, unsupported by the harness (“needs pathRules”), unavailable here, waiting for a backend (“needs --backend replay or real”), or out of budget No

Commands lists every flag of ctxray test, and Reporters and Reports covers the HTML, JUnit, and Markdown output.

Reference

Exports of @mairwunnx/ctxray

Export What it is
suite(name, options?, body) Groups tests. options.fixture: the project folder, relative to the test file. The body must register its tests right away: an async body fails the file
test.resolve, test.assemble, test.wire, test.activate, test.scenario Register a test of L3, L4, L5, L6, or L7: (name, options?, body)
test.<kind>.each(items) (name, options?, body) once per object, or once per rule of a rule set
expect(value) Matchers, below
expect.on(harnesses, body) Runs body only on these harnesses: one id or a list. Returns what body returns, so await it when it's async
discover.rules(selector?) A rule set, selected inside each test
corpus(path) Prompts from a module or a JSON file, relative to the test file
defineConfig(config) Types for ctxray.config.ts, see Configuration

Types: TestOptions, SuiteOptions, Rule, RuleSet, RuleSelector, ResolvedLoadout, ResolveRequest.

Test kinds

Kind Layer The body gets Default timeout
test.resolve L3 resolve({ cwd?, files? }) 30 s
test.assemble L4 assemble({ cwd?, prompt? }) 120 s
test.wire L5 session, startSession({ cwd?, prompt? }) 120 s
test.activate L6 runs 300 s per sample
test.scenario L7 run 300 s per sample

Every body also gets harness and, under .each, the item's fields or rule.

TestOptions

Option Type Default Layers Means
requires Capability[] [] all A harness without one of these skips the test as unsupported: skills, pathRules, intentRules, mcp, hooks, headless, proxyable
retries number 0 all Runs again after a failure. A later pass makes the test flaky
timeout number (ms) see above all For L6 and L7 it limits each sample, not the test
cwd string project root L4–L7 Where the harness starts, relative to the project root
prompt string – L6, L7 The prompt
positive string[] [] L6, L7 Prompts that should load the subject, in runs.positive
negative string[] [] L6, L7 Prompts that shouldn't, in runs.negative
samples number 16 per group of prompts (L6), 1 (L7) L6, L7 Runs per prompt. Without it, L6 splits the config's activate.runs between the prompts of each group
turns number 2 (L6), 30 (L7) L6, L7 Model turns before ctxray answers in the model's place
min number every run L7 The share of runs that must pass

L6 and L7 need at least one of prompt, positive, and negative.

Loadouts

Field L3 resolve() L4 assemble(), L5 request.loadout
harness ✓ ✓
cwd, files ✓
version, model ✓
system ✓
rules, commands ✓ ✓
skills.listed, skills.loaded ✓ ✓
mcp.servers ✓ ✓
mcp.tools (id server/tool) ✓
duplicates ✓
tokens.known ✓
tokens.total, tokens.byMcpServer(name), tokens.framing ✓
tokens.byComponent, tokens.exact, tokens.method ✓ ✓

Every list is an array with .get(id), which finds an entry by path or name. A rule has path, activation, reason, and tokens. A skill has name, path, builtin, and tokens. L3 doesn't know the system prompt, the tools, or the MCP schemas, so tokens.known counts only rules, skills, and commands. ctxray's own plugin stays out of every captured list.

Session (L5)

Member Does
firstRequest() The first request of the session
nextRequest() The request that follows the latest answer
model.calls(tool, input?) Answers with a tool call. Read with path and Skill work on every harness
model.says(text) Answers with text
events() The harness's own events, like Claude Code's stream-json and Codex's --json
end() Ends the session. ctxray calls it after the body anyway

A request has index, loadout, added.rules, added.skills, predicted, raw, and snapshot.

Runs (L6, L7)

runs is an array of runs with runs.positive and runs.negative. A run has prompt, sample, skills, rules, tools ({ name, input }, oldest first), changed, turns, cut, loadout, added (what joined after the first request), and error when the session itself failed.

Rule and RuleSelector

Member Means
RuleSelector.owner A CODEOWNERS owner or a list of them
RuleSelector.under A folder or a glob, relative to the project root
RuleSelector.activation always, path, intent, manual, or a list
RuleSet.where(selector) A narrower set
RuleSet.rules The selected rules, inside a test
RuleSet.root The under folder, or the common folder of the rules
Rule.path, Rule.dir The file, and the folder it applies to
Rule.activation, Rule.globs How it loads, and its globs
Rule.matchingFiles({ sample? }) Project files it applies to, spread evenly
Rule.nonMatchingFile() One project file it doesn't apply to, or undefined

Instructions without globs count as always, with globs as path. Skills the harness lists count as intent, the others as manual.

Matchers

Matcher Takes Checks
toBe(value) anything Object.is
toEqual(value) anything Deep equality. An entry list equals a list of ids in the same order
toBeDefined(), toBeUndefined() anything
toBeLessThan(n), toBeLessThanOrEqual(n) a number
toBeGreaterThan(n), toBeGreaterThanOrEqual(n) a number
toHaveLength(n) anything with length
toMatch(text or RegExp) a string A substring or a match
toInclude(ids) a list Holds every id: a path, a name, a rule, a rule set, or a list. .not: holds none of them
toMatchLoadoutSnapshot() a loadout Equals the stored snapshot. No .not
toLoadRule(rule) an L5 request The rule is in context. .not: this request didn't bring it in
toAgreeWithResolver() an L5 request The harness loaded what L3 predicts. No .not
skill(name).toActivate({ min }) runs The interval's lower bound reaches min, by default 0.8 or the config's activate.min
rule(path).toActivate({ min }) runs The same for a rule
anyOf(set).toActivate({ min }) runs The same for any of a rule set or a list of paths and names
….not.toActivate({ max }) runs The upper bound stays at or under max, by default 0.2 or the config's activate.max

Matchers have a .not form unless the table says otherwise. The activation matchers put it before toActivate.

Files next to your tests

Path Written by Commit it
__snapshots__/<name>.ctxray.snap.json ctxray test -u Yes
__cassettes__/<name>.ctxray.ts.<harness>.json ctxray test --backend real --record Yes
a corpus, like prompts/intents.json ctxray gen-prompts or you Yes

Clone this wiki locally