Skip to content

The Loadout Stack

Pavel Erokhin edited this page Nov 16, 2026 · 1 revision

Русская версия

ctxray checks your agent setup in seven layers, from cheap to expensive. Each layer answers one question, and a cheaper layer catches whatever it can before a dearer one runs. L1–L3 only read files and finish in milliseconds. L4 and L5 launch the real harness against a mock model, which costs nothing. Only L6 and L7 ask a model, or replay the answers you recorded once.

The Loadout Stack: L1 Lint, L2 Policy, and L3 Resolve read files. L4 Assemble and L5 Wire launch the agent with a mock model. L6 Activate and L7 Scenario ask a model.

Layer Question Cost Who writes it
L1 Lint Do the files parse, link, and reach every harness? reads files ctxray runs it by itself
L2 Policy Does the setup follow your organisation's rules? reads files ctxray, from the policies of your config
L3 Resolve Which rules and skills load for this folder and these files? reads files you, with test.resolve
L4 Assemble What does the harness really send in its first request? launches the harness with a mock model you, with test.assemble, and ctxray's own starting-context check
L5 Wire What joins the context as the session goes on? launches the harness with a mock model you, with test.wire
L6 Activate Does the model pick the right skill or rule for a prompt? asks a model you, with test.activate
L7 Scenario Does a whole task go the way it should? asks a model you, with test.scenario

The examples below come from ctxray's payments fixture. Writing Tests covers the full test API.

L1 Lint

L1 checks every file each harness reads: frontmatter that doesn't parse, an @import that points nowhere, a glob no file matches, a skill whose folder and name disagree, a file over the size the harness accepts. It also runs the reach checks from Concepts: a rule one harness reads and another never gets. You write nothing for it.

 ctxray checks
   L1 lint      project files
                claude-code ✓ 1 warning   codex ✓

 ⚠ claude-code › L1 lint › project files
     .claude/rules/web.md  No file in the repository matches frontend/**/*.tsx, so the rule never loads  dead-rule

Errors fail the run, warnings show up without failing it. --error-on-warnings makes every warning fail too. Each finding ends with its check's id, here dead-rule, which is what the lint section of Configuration names to change its level. ctxray baseline records today's findings, so later runs fail only on new ones, see Commands.

L2 Policy

L2 checks the rules your organisation sets: token limits, MCP servers that must or mustn't be configured, files that must exist. You write them once in the config, and ctxray checks them for every harness without calling a model:

import { defineConfig } from "@mairwunnx/ctxray";

export default defineConfig({
  harnesses: { "claude-code": {}, codex: {} },
  policies: [
    { id: "no-jira", denyMcp: ["jira"] },
    { id: "lean-rules", maxTokens: { rules: 2_000 } },
  ],
});
   L2 policy    no-jira
                claude-code ✗   codex ✗
   L2 policy    lean-rules
                claude-code ✓   codex ✓

 ✗ claude-code › L2 policy › no-jira
     MCP server “jira” isn't allowed

A limit that files can't answer, like the total with the system prompt and MCP tools, moves to L4's starting-context check, where ctxray sees the real request. Policies covers every check and how a company config shares them.

L3 Resolve

L3 predicts the Loadout from files: which rules and skills load when the agent starts in a folder and works with some files. It needs no prompt and no harness, so it runs in milliseconds for every harness you test.

test.resolve("payments rules attach to payment sources", { requires: ["pathRules"] }, async ({ resolve }) => {
  const l = await resolve({ cwd: "app", files: ["app/src/main/kotlin/payments/Charge.kt"] });
  expect(l.rules).toInclude([".claude/rules/kotlin.md", ".claude/rules/payments.md"]);
  expect(l.rules).not.toInclude(".claude/rules/web.md");
  expect(l.rules.get(".claude/rules/payments.md").reason).toBe("glob:**/payments/**/*.kt");
});

requires: ["pathRules"] skips the test on Codex, which has no path rules, and the report says so.

L4 Assemble

L4 launches the real harness on a copy of your project, lets it build its first request, and answers with the mock model. Now ctxray sees what files can't show: the system prompt, the built-in tools, and every MCP tool with its schema.

test.assemble("context budget", async ({ assemble }) => {
  const l = await assemble({ cwd: "app" });
  expect(l.tokens.total).toBeLessThan(25_000);
  expect(l.tokens.byMcpServer("jira")).toBeLessThan(3_000);
  expect(l.mcp.servers).toEqual(["github", "jira"]);
  expect(l).toMatchLoadoutSnapshot();
});

ctxray also runs one L4 check of its own for every harness it can launch, the starting context, so you see the real total even without a test:

   L4 assemble  starting context
                claude-code ✓ ≈ 17,043 tokens   codex ✓ ≈ 8,232 tokens

L4 needs the harness on your PATH or pinned in ctxray.lock, and git. Capture Modes explains how ctxray launches it without touching your files or your home folder.

L5 Wire

L5 drives a session turn by turn. Your test plays the model: it reads a file, calls a tool, or says something, and then checks the next request. That's how you see a nested rule join the context after a read.

test.wire("nested instructions load on read", { requires: ["pathRules"] }, async ({ session }) => {
  await session.model.calls("Read", { path: "app/src/main/kotlin/payments/Charge.kt" });
  const next = await session.nextRequest();
  expect(next.loadout.rules).toInclude("app/src/main/kotlin/payments/AGENTS.md");
  expect(next).toAgreeWithResolver();
});

toAgreeWithResolver() compares what the harness really sent with what L3 predicts. When they part, ctxray's model of that harness version is out of date: the test fails and lists the rules and skills only the harness loaded, and those only L3 predicted.

L6 Activate

L6 asks a model a prompt several times and counts how often a skill or rule activated. The result comes with a confidence interval, so one lucky run can't pass the test.

test.activate(
  "refund prompt → refunds skill",
  { prompt: "Make a partial refund of 40 EUR for order 123.", samples: 5, requires: ["skills"] },
  ({ runs }) => {
    expect(runs).skill("refunds").toActivate({ min: 0.5 });
    expect(runs).skill("payouts").not.toActivate({ max: 0.5 });
  },
);

Record the answers once with --backend real --record, then replay them for free with --backend replay. ctxray gen-prompts drafts prompts from your rules' and skills' descriptions, see Commands.

L7 Scenario

L7 gives the agent a whole task and lets it work until it stops, 30 model turns at most by default. The test then checks the run: the skills and rules that reached the model, the tools the agent called, and the files it changed.

test.scenario(
  "a refund goes through the refunds skill",
  { prompt: "Make a partial refund of 40 EUR for order 123.", samples: 3, min: 0.6, requires: ["skills"] },
  ({ run }) => {
    expect(run.skills).toInclude("refunds");
    expect(run.changed).not.toInclude(".github/CODEOWNERS");
  },
);

min is the share of runs that must pass, every one by default.

Choosing layers

ctxray test runs every layer by default. --layer picks some of them, as a list, a range, or both:

ctxray test --layer L1-L3      # only what reads files: every commit, in milliseconds
ctxray test --layer L3,L5
ctxray test --layer 4-5

CTXRAY_LAYER does the same in CI. Tests outside the selection show as filtered.

L6 and L7 need a model that answers by itself. On the mock backend, the default, ctxray skips them and says what they need:

   L6 activate  refund prompt → refunds skill
                claude-code ○ needs --backend replay or real   codex ○ needs --backend replay or real

L4 and L5 always run against the mock model, whatever backend you pick. A test can also be skipped as unsupported, when the harness lacks something the test requires, or unavailable, when ctxray can't launch the harness here. Concepts describes the backends, and CI Integration which layers to run where.

Clone this wiki locally