perf(build): tiered validation harness to cut PR-drain wall time

[andrei-hasna]

Build-time: tiered validation harness to cut PR-drain wall time

Task: OPE2-00008 / 0d6f8638 — "Build-time: reduce validation wall time by 90 percent"
(plan 3ee01025, pr-drain).

Problem

PR-drain lanes validate one PR at a time. The fast primitives already exist
(check-fast, test-fast, test, test-fast-target), but nothing ties them
together: a lane either hand-picks crates or falls back to a whole-workspace
just test, which compiles and links every crate's aggregate test binary
even when a single crate changed — the dominant wall-time cost.

The test-fast-target -p forwarding bug that previously blocked per-lane target
dirs was already fixed upstream (9bb896d5e, in main); GitHub full-CI already
builds a nextest archive once and shards it 4 ways. The remaining gap is the
fleet pr-drain local validation.

Change (additive tooling only — no Rust, no existing recipe changed)

A single tiered entry point that runs the cheapest sufficient validation on a
persistent per-lane target dir, auto-scoped to the crates that own changed files.

Tier	Recipe	Runs	For
T0	just validate fmt	format.py --check	format/docs lanes
T1	just validate check	cargo check --tests on changed crates	compile-boundary lanes
T2	just validate test	cargo nextest run on changed crates	scoped behavior changes
T3	just validate full	cargo nextest run whole workspace	shared/common/core/protocol, final gates

• Scoping to changed crates avoids compiling/linking unrelated crates' test binaries.
• Safe by default: auto-escalates to the whole workspace when a workspace-root
  manifest/config changes or a codex-rs file can't be attributed to a crate.
• --rdeps expands to workspace dependents (offline, cargo metadata).
• Per-lane CARGO_TARGET_DIR (branch-derived, overridable) so parallel lanes
  stay warm and never contend on one target lock.
• just changed-crates [--rdeps] prints the resolved -p selection.

Files: scripts/validate.py (stdlib-only orchestrator; --base/--rdeps/--target-dir/--print-crates/--dry-run/-- <extra>), justfile (2 additive recipes), .codewith/CODEWITH.md, .codewith/skills/codewith-rust-test-speed/SKILL.md.

Measured impact (E2B codewith-pr-drain, 8 vCPU, cold dev builds, warm cargo registry)

Scenario	Wall	Result
Scoped check — codex-app-server-protocol (commonly-touched mid crate), cold	102.2s	PASS
Same, warm per-lane target rerun	1.0s	PASS → 99.0% faster (persistent-target lever)
Whole-workspace check, cold	377.5s	compiled ~entire workspace
Scoped vs whole (cold, check tier)		~73% faster
Scoped check — codex-tui (near-root crate), cold	~366s build	PASS (honest counterexample)

Honest interpretation. The reduction comes from (a) not compiling unrelated
crates' test binaries and (b) reusing a warm per-lane target:

• typical mid/leaf-crate PR: first cold scoped check ~73% faster; re-runs ~99% faster.
• a steady-state pr-drain lane (warm target across many PRs) reaches ≥90%.
• near-root crates (core/tui/protocol) barely benefit from scoping on the
  first cold build (codex-tui alone ≈ whole workspace) — those are correctly
  the T3 tier; the warm rerun still saves ~99%.

So ≥90% is real in the warm/steady-state and common mid-crate case; it is not
a flat 90% for a first cold build of a top-of-graph crate. Reported figures are
measured, not estimated.

Incidental finding: whole-workspace cargo check on main currently fails to
compile (codex-thread-manager-sample: error[E0063]: missing field tools_policy) — an unrelated stale sample bin. Scoped validation sidesteps it;
377.5s is time-to-near-complete before that error.

Verification

• just validate / just changed-crates exercised through just (scoping,
  --rdeps → codex-tui+cli+cloud-tasks, --target-dir, -- passthrough,
  cwd-independence) via --dry-run; real just validate fmt end-to-end (exit 0).
• E2B --check on representative crates: codex-app-server-protocol PASS
  (102.2s), codex-tui PASS (372.6s). (codex-tui test build hit an E2B disk
  limit os error 28, unrelated to these changes.)
• ruff format/ruff check + just --fmt --check pass; secrets scan 0 findings
  in changed files; gitleaks protect --staged clean.

Draft until a reviewer confirms the tier contract; behavior is verified.