Skip to content

CI Integration

Pavel Erokhin edited this page Nov 16, 2026 · 2 revisions

Русская версия

ctxray belongs in CI: every merge request then shows whether the agents still load the right rules, whether the context grew, and whether a policy broke. This page covers GitHub Actions, GitLab CI, and any other CI that runs a container or a shell script.

Every push runs ctxray test, layers L1 to L5 on the mock model. A pull request replays recorded answers for L6 and L7 and compares the context with the base branch. You record the answers once with a real model and commit them. A weekly job runs L6 against the real model with a budget. Every job writes reports and a job summary.

Before you start, commit two files: your config and ctxray.lock. The lockfile pins the harness builds CI runs, and without it every test from L4 up fails in CI (Harnesses explains why).

GitHub Actions

The action MairwunNx/ctxray-action installs ctxray, caches the harnesses ctxray.lock pins, and runs ctxray test:

- uses: actions/checkout@v7
- uses: MairwunNx/ctxray-action@v0

@v0 follows the newest 0.x release. Pin an exact tag, like @v0.1.0, when you want CI to change only on your say.

Inputs

Input Default What it does
version the action's own version The ctxray version to install, like 0.1.0
args test What to run, as arguments to ctxray: test --layer L1-L3, inspect, diff --base origin/master
working-directory . The project's folder, where ctxray.lock lies
cache true Cache the harnesses ctxray.lock pins between runs
download-base GitHub Releases Where the releases live, for a mirror. Same as CTXRAY_DOWNLOAD_BASE in install.sh

The action splits args on spaces, so a single argument can't hold a space. Pass such values through the config or an environment variable instead.

The action runs on Linux, macOS, and Windows runners, on x64 and arm64. It installs through install.sh (or install.ps1 on Windows) into the runner's temporary folder and adds it to PATH, so later steps can call ctxray too.

Tests on every push

name: ctxray

on:
  push:
  pull_request:

jobs:
  ctxray:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: MairwunNx/ctxray-action@v0
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: ctxray-report
          path: .ctxray/report

In CI, ctxray writes the console output plus JSON, JUnit, Markdown, and HTML reports into .ctxray/report/. On GitHub it also appends the Markdown summary to the job's summary page, so you see the result without opening the log. The artifact step keeps the HTML report for anyone who wants the details.

What a pull request changes

ctxray diff compares what each harness loads on the pull request with what it loaded on the base branch. It needs the base branch's history, so check out with fetch-depth: 0:

name: ctxray diff

on: pull_request

jobs:
  diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: MairwunNx/ctxray-action@v0
        with:
          args: diff --base origin/${{ github.base_ref }}

ctxray diff only reports: it exits with 0 whatever changed. To put the same comparison into the job summary and the HTML report next to the test results, run the tests with --base instead:

      - uses: MairwunNx/ctxray-action@v0
        with:
          args: test --base origin/${{ github.base_ref }}

Real models on a schedule

L6 Activate sends prompts to a real model many times, which costs money. Run it on a schedule with a key from your repository secrets, and cap the spend with --budget in US dollars:

name: ctxray activation

on:
  schedule:
    - cron: "0 4 * * 1"
  workflow_dispatch:

jobs:
  activation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: MairwunNx/ctxray-action@v0
        with:
          args: test --layer L6 --backend real --budget 5
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Once the budget is spent, the tests that haven't started yet are skipped with the reason “budget spent”, and the run doesn't fail on that alone. The harness itself never sees the key: ctxray's proxy adds it on the way to the model. Enterprise lists which variable serves which provider, and how to go through a gateway instead.

Replay on every pull request

Record the model's answers once with --record, commit the cassettes, and every pull request replays them for free. Writing Tests explains recording and where the cassettes live.

ctxray test --layer L6-L7 --backend real --record
name: ctxray

on: pull_request

jobs:
  ctxray:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: MairwunNx/ctxray-action@v0
        with:
          args: test --backend replay

Replay needs no key and no network: L1–L5 run as always, and L6–L7 play back the recorded answers.

GitLab CI

Each release carries a template, ctxray.gitlab-ci.yml. Include it and extend its hidden job .ctxray:

include:
  - remote: https://github.com/MairwunNx/ctxray/releases/latest/download/ctxray.gitlab-ci.yml

ctxray:
  extends: .ctxray

latest follows every release. To pin one, name its tag: https://github.com/MairwunNx/ctxray/releases/download/v0.1.0/ctxray.gitlab-ci.yml. In a closed network, keep a copy of the template in a project of your own and include it from there:

include:
  - project: platform/ci-templates
    ref: main
    file: ctxray.gitlab-ci.yml

ctxray:
  extends: .ctxray

What the template does

The .ctxray job runs ctxray test $CTXRAY_ARGS in the image ghcr.io/mairwunnx/ctxray:<version>-runtime, which holds ctxray, git, and Node. It keeps .ctxray/report/ as an artifact, even when the job fails, and hands junit.xml to GitLab as a JUnit report. GitLab then shows each failed test in the merge request's test summary.

Variable Default What it does
CTXRAY_IMAGE ghcr.io/mairwunnx/ctxray:<version>-runtime The image the job runs in. Point it at your registry's mirror, or at an image with the harnesses inside
CTXRAY_ARGS empty Extra arguments to ctxray test, like --layer L1-L3
CTXRAY_CACHE_DIR $CI_PROJECT_DIR/.ctxray-cache The harness cache. GitLab caches only inside the project's folder

The job caches .ctxray-cache/ with a key built from ctxray.lock, so a new lockfile starts a fresh cache. ctxray never counts .ctxray-cache/ and .ctxray/ as project files, so the cached harness builds stay out of L1 lint and out of the project copy L4 launches on. Add .ctxray-cache/ to your .gitignore anyway if you don't want git status to list it.

Merge requests

To compare with the merge request's target branch, fetch the whole history and pass the merge base:

ctxray:
  extends: .ctxray
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    GIT_DEPTH: "0"
    CTXRAY_ARGS: --base $CI_MERGE_REQUEST_DIFF_BASE_SHA

Real models on a schedule

Store the key as a masked CI/CD variable, and let the job run only in scheduled pipelines:

ctxray-activation:
  extends: .ctxray
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  variables:
    CTXRAY_ARGS: --layer L6 --backend real --budget 5

Any other CI

Two ways work everywhere: the Docker images, or install.sh.

The image. Run the job in ghcr.io/mairwunnx/ctxray:<version>-runtime for L4 and up, or in ghcr.io/mairwunnx/ctxray:<version> for L1–L3 alone. The image's entrypoint is ctxray, so reset it where your CI runs a script inside the container. Getting Started lists the tags.

The script. Pin the version, so every run installs the same build:

curl -fsSL https://github.com/MairwunNx/ctxray/releases/download/v0.1.0/install.sh | CTXRAY_VERSION=0.1.0 sh
export PATH="$HOME/.local/bin:$PATH"
ctxray test

Then do what the action and the template do for you:

  1. Cache the folder ctxray cache dir prints, with a key built from the hash of ctxray.lock.
  2. Keep .ctxray/report/ as an artifact, and hand .ctxray/report/junit.xml to your CI's test view.
  3. Make sure your CI sets CI, or pass --ci (below).

What changes in CI

ctxray takes a run for CI when CI, GITLAB_CI, or GITHUB_ACTIONS is set to anything other than empty, 0, or false. CI=false switches CI mode off by hand.

Locally In CI
Output live view in a terminal plain lines a log can keep
Reporters console console, json, junit, markdown, html
ctxray.lock a mismatch with the config is only mentioned --frozen: a mismatch fails the run
GitHub job summary – the Markdown summary is appended when GITHUB_STEP_SUMMARY is set

The CI reporters apply only while your config sets no reporters, and --reporter overrides both. Reporters and Reports describes each file.

The --ci flag does all of this on any machine, --frozen included. Pass it when your CI sets none of the variables.

Caching harnesses

The harnesses are the slow part of a cold run: Claude Code, Codex, and the others weigh hundreds of megabytes together. ctxray installs the builds ctxray.lock pins into its cache on first use and takes them from there afterwards. ctxray cache dir prints the folder, which is exactly what a CI cache step needs:

- id: harness-cache
  run: echo "dir=$(ctxray cache dir)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v6
  with:
    path: ${{ steps.harness-cache.outputs.dir }}
    key: ctxray-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('ctxray.lock') }}
- run: ctxray harness install

The action does exactly this when cache is true. ctxray harness install as its own step is optional: ctxray test installs on first use too, but a separate step shows the download apart from the tests. Harnesses covers the cache folders and CTXRAY_CACHE_DIRS.

A prebuilt image with the harnesses

A runner that may not reach npm, or a pipeline that shouldn't wait for downloads, needs the harnesses inside its image. Each release carries harnesses.Dockerfile for that. Put it into your project and build it there:

docker build -f harnesses.Dockerfile -t registry.example.com/ci/ctxray .

The file starts from ghcr.io/mairwunnx/ctxray:<version>-runtime, copies ctxray.config.ts and ctxray.lock, and runs ctxray harness install --frozen into /opt/ctxray/harnesses, which it names in CTXRAY_CACHE_DIRS. If your config is a .jsonc file or extends other files, change the COPY line to match. The harnesses come from your registry under your licences: ctxray's own images never carry them.

Rebuild the image whenever ctxray.lock changes. Then point CI at it: CTXRAY_IMAGE on GitLab, container: on GitHub, the job image elsewhere.

Exit codes

Scripts and pipelines can branch on these numbers, and they stay the same across releases:

Code Meaning
0 Everything passed
1 A test or a check failed
2 An unknown command, flag, or argument, or an unknown --base revision
3 The config is missing or invalid
4 A harness, the network, or the file system let the run down: ctxray doctor found an error, ctxray harness install couldn't install, or a command that needs the network runs offline
5 A bug in ctxray. The message links to the issue tracker
130 Ctrl-C or SIGINT

L4 and up need a sandbox

From L4 up, ctxray launches the real harness on a copy of your project. In headless mode a harness starts the project's MCP servers and hooks without asking, so the project's code runs. On macOS ctxray confines the run's writes with sandbox-exec. Linux and Windows have no such sandbox: there the CI container, or a runner that's thrown away after the job, is the sandbox.

Hosted runners on GitHub and GitLab are throwaway machines, so they're fine. On a self-hosted runner, run the job in a container. ctxray doctor says which case your machine is: “No sandbox on this system: run L4 and up in a container, as CI does”.

Clone this wiki locally