windtunnel

windtunnel is an experimental, solo-first CLI gym for practicing system architecture. You write a complete design without AI, fresh adversarial agents attack it, you defend the frozen design, and a separate model makes only item-level judgments. Go—not the model—computes the score.

The binary is wt. Windtunnel is a learning signal, not an authoritative hiring assessment or production-security certification.

Bundled challenge packs cover a distributed rate limiter, durable job queue, multi-region inventory reservations, and secure webhook delivery.

Install

Go install

go install github.com/AnishDe12020/windtunnel/cmd/wt@latest
wt version

This requires Go 1.24 or newer. Go installs wt into GOBIN, or $(go env GOPATH)/bin when GOBIN is unset; make sure that directory is on your PATH.

First run

wt init ~/windtunnel-gym
cd ~/windtunnel-gym
wt list
wt validate
wt start 001

start and defend are entirely local and invoke no AI. redteam, grade, calibrate, and progress require an authenticated Codex CLI or Claude Code. Provider invocations use your account and may consume paid model credits. The default configuration uses Codex for red-team and Claude for grading; either can be overridden.

Build from source

git clone https://github.com/AnishDe12020/windtunnel.git
cd windtunnel
go test ./...
go install ./cmd/wt

Release archives support macOS and Linux on amd64 and arm64. Windows currently cross-compiles but is not supported because editor launching relies on a POSIX shell. Codex red-team runs require macOS Seatbelt or an external filesystem sandbox for every Codex-powered command; see Isolation.

The loop

wt start 001
wt redteam 001
wt defend 001
wt grade 001
wt report 001
wt progress 001

1. Start solo

start prints the challenge, starts the timer, and opens design.md without invoking an agent. Vim and Neovim open the challenge beside the design. Known GUI editors including Zed, VS Code, Cursor, Windsurf, and Sublime receive their wait flag automatically.

2. Red-team

redteam launches three fresh processes: security attacker, scale pessimist, and 03:00 on-call. Context is never shared between rounds. Each sees only the frozen bytes of design.md, never the challenge rubric, references, or prior findings.

Each persona returns at most two strong, independent root findings, so a standard run produces no more than six defense questions. It must omit cosmetic, speculative, and merely nice-to-have concerns.

3. Defend

defend opens a readable Markdown Q&A sheet and invokes no AI. Every finding gets one decision:

• accept: the flaw exists in the original design;
• rebut: the original design already defeats it—proposed fixes do not count;
• out-of-scope: the challenge does not require the concern.

Rebut and out-of-scope answers require reasoning.

4. Grade and report

grade gives a fresh grader the hidden rubric, references, frozen design, findings, and defense. The grader makes binary rubric judgments, independently reclassifies finding severity, clusters findings by root cause, and judges which findings survive. It never emits a holistic score.

wt validates exact design citations and computes the integer score. It writes both verdict.json and a human-readable verdict.md; report prints the latter view from an existing verdict without invoking an agent.

The verdict also awards a deterministic run rating and small badges for things like clean survival, complete tier coverage, and preempting a flaw in the solo design. Models cannot award or influence these labels directly.

5. Track progress

progress compares completed dated attempts and reports newly caught thinking tiers, recurring blind spots, and whether solo-design preemption is trending up. It is read-only over attempts/.

Use --attempt YYYY-MM-DD (or a same-day -NN suffix) to select an older attempt.

Scoring

raw = addressed rubric weights
    - severity penalties for unique surviving root flaws
    + bonuses for unique preempted flaws

final = clamp(raw, 0, rubric maximum)

Windtunnel v0.2 reports completeness and correctness as separate 10-point axes. The correctness axis bottoms out at zero while retaining uncapped severity debt for diagnosis. The original raw formula remains authoritative, so presentation cannot forgive additional flaws; the report no longer presents a 40-point debt as though it were commensurate with the 10-point completeness axis. Earned preemption above the ceiling remains visible without producing an 11/10.

Rubric weights measure completeness; grader-confirmed severity controls only the correctness penalty. The default linear curve is low=1, medium=2, high=3, critical=5; the experimental exponential curve is 1, 3, 9, 27. One surviving critical can therefore cut an otherwise complete 10/10 design to 5/10.

Preemption is awarded only when the original design names and materially mitigates the risk before red-team. A defense cannot retroactively edit the design.

Challenge and artifacts

wt init extracts all bundled challenges, public JSON schemas, and the default configuration into a flat-file workspace. wt validate [challenge-id] checks rubric totals and tier coverage, references, calibration fixtures, configuration, and published schema documents before a challenge is used or contributed.

challenges/001-rate-limiter/
  challenge.md
  rubric.md
  references/
  .calibration/

schemas/
  attempt.schema.json
  redteam-round.schema.json
  redteam-manifest.schema.json
  defense.schema.json
  verdict.schema.json
  calibration-findings.schema.json
  calibration-report.schema.json
  planted-bugs.schema.json
  progress-report.schema.json
  windtunnel-config.schema.json

attempts/001/YYYY-MM-DD/
  attempt.json
  design.md
  redteam-manifest.json
  redteam-r1.json
  redteam-r2.json
  redteam-r3.json
  defense.md
  verdict.json
  verdict.md

The rubric is hidden by workflow convention, not DRM. This is a personal gym; opening the answer key only cheats yourself. Stable rubric hashes make attempts comparable.

Bundled challenges

ID	Challenge	Main pressure
001	Distributed rate limiter	shared state, clocks, overload
002	Durable multi-tenant job queue	leases, retries, fairness, durable scheduling
003	Multi-region inventory reservations	database invariants, contention, regional consistency
004	Multi-tenant webhook delivery	durable fan-out, SSRF, signing, tenant isolation

Every pack has a 10-point binary rubric spanning surface, structural, systemic, and emergent thinking, at least two annotated references, and a planted-bug calibration fixture.

The manifest hashes the solo design and every completed red-team round. defend and grade refuse changed inputs because a mitigation added after seeing findings is not preemption, and edited attack artifacts are not authentic agent output.

Configuration

windtunnel.json defaults to cross-provider review:

{
  "severity_curve": "linear",
  "preemption_bonus": 1,
  "redteam_model": "codex",
  "grade_model": "claude",
  "agent_timeout_seconds": 600
}

Override providers per command when only one CLI is authenticated:

wt redteam 001 --redteam-model claude
wt grade 001 --grade-model codex

Set WT_ROOT to use a workspace without changing directories.

Isolation

Designs and grading artifacts are sent to the selected external model provider. Do not submit confidential architecture unless that provider and account are approved for the data.

Every invocation uses a fresh temporary working directory, structured output, disabled session persistence, and no shared model context. Claude tools and MCP tools are disabled. On macOS, an outer Seatbelt profile denies the provider read or write access to the entire gym root; Codex child execution is also blocked.

Every Codex-powered command fails closed where that outer boundary is unavailable. When running wt inside a container or equivalent filesystem sandbox, set WT_EXTERNAL_SANDBOX=1 to attest that boundary. Malformed model JSON is retried once and raw failure output is preserved instead of producing a misleading artifact.

Calibration

wt calibrate 001 --grade-model codex

Calibration grades a planted-bug fixture three times, reports variance and catch rate, and compares linear with exponential penalties. Its recommendation is provisional until checked against acceptable-design controls and real attempts.

Contributing and security

See CONTRIBUTING.md, SECURITY.md, and the changelog. Windtunnel is available under the MIT License.