Skip to content

feat: benchmarking instrumentation (snapshots, gate logging, perf budgets)#20

Merged
saldestechnology merged 5 commits into
mainfrom
feat/benchmark-instrumentation
Jul 10, 2026
Merged

feat: benchmarking instrumentation (snapshots, gate logging, perf budgets)#20
saldestechnology merged 5 commits into
mainfrom
feat/benchmark-instrumentation

Conversation

@saldestechnology

Copy link
Copy Markdown
Collaborator

Implements Phase 0–1 of the benchmarking methodology plus the Program C performance budgets, all inside the ctx repo. Built to instrument the longitudinal field study (does enabling gates improve real quality trends?) and to enforce hook-path latency in CI.

What's included

Measurement instruments (Program B / Phase 0–1)

  • ctx snapshot — per-commit Parquet metric partitions (.ctx/snapshots/sha=<sha>/{symbols,files,dup_pairs,meta}.parquet) with per-symbol and per-file metrics, near-duplicate pairs, rule-violation counts, and churn. ctx snapshot backfill --since <REF> [--every N] builds history through temporary git worktrees (churn --until-anchored to each commit's date). Atomic partition writes, idempotent skips, --json envelopes.
  • ctx sql --snapshots[=DIR] — materializes snap.files/symbols/dup_pairs/meta tables before sandbox hardening (external access stays disabled at query time; regression-tested). Canned duplication/violation/hotspot-mass trend queries documented in the SQL schema reference. TIMESTAMP values now render as RFC3339.
  • Gate loggingCTX_GATE_LOG makes ctx score append one JSONL record per gate evaluation (.ctx/gate-log.jsonl by default). Opt-in, local-only, best-effort; never changes exit codes.
  • Blocking Stop hookCTX_GATE_BLOCKING=1 turns a failed ctx score --fail-on into a blocking stop (exit 2); default stays non-blocking, operational errors always fail open. Re-run ctx harness init to regenerate hooks.
  • Study scriptsscripts/rework-rate.sh (30-day line-survival rework rate via git blame) and scripts/revert-rate.sh; both POSIX sh, repo-agnostic, integration-tested against synthetic dated repos.
  • Run-record schema — versioned JSON Schema (draft 2020-12) + docs for benchmark run records, with a drift test keeping schema and example in sync.
  • snapshot.yml workflow — appends a partition per merge to the ctx-snapshots orphan data branch (bootstrap + steady-state + idempotent re-run verified against a scratch remote).

Performance budgets (Program C)

  • ctx::fixture — deterministic seeded repo generator (byte-identical trees and commit SHAs per seed; zipf-skewed fan-in so PageRank is non-trivial; repo_2k/repo_150k_loc presets; apply_change_set for identical per-run work). Ships in the crate for reuse by the future ctx-bench suite.
  • perf/ companion crate (not published, not a workspace member) — perf-harness spawns the built binary against generated fixtures: budgets of 300ms incremental index, 2s score, 1s check, 500ms map, 500ms sql (2,000-file fixture), 60s cold 150k-LOC index, 300MB RSS (via wait4/ru_maxrss), plus a 1.20× regression gate against committed baselines. Advisory perf CI job (continue-on-error) with CTX_PERF_BUDGET_SCALE=1.5; new thin-LTO [profile.perf]. Criterion microbenches for library hot paths.

Notes for review

  • No baseline is shipped yet — baselines are runner-class-specific; capture procedure is in perf/baselines/README.md. The compare logic warn-passes on missing entries during the interim.
  • First real perf finding: ctx score --against HEAD on the 2,000-file fixture measured ~4.2s / 379MB RSS locally (macOS, debug of the scenario protocol under [profile.perf]) — over both the 2s budget and the RSS ceiling. The advisory CI period will show ubuntu numbers; if they agree, score's hook path needs optimization before the job flips to required.
  • The branch passes ctx's own gates: ctx check clean, ctx score --against main shows new_duplication 0 after deduplicating test helpers into ctx::testutil::git_stdout.

Test plan

  • 440+ tests green with --all-features; 385 green with --no-default-features (Windows stub paths: ctx snapshot/ctx sql exit 2 with a clear message).
  • cargo clippy --all-features --all-targets -- -D warnings clean; cargo fmt --check clean; cargo publish --dry-run green (no perf/ leakage; cargo package --list verified).
  • Docs site builds with onBrokenLinks: 'throw'.
  • E2E: backfill → trend queries verified on scratch repos; blocking gate + gate log exercised through the actual generated stop.sh; snapshot workflow's branch-append rehearsed against a scratch bare remote.

🤖 Generated with Claude Code

- ctx snapshot: per-commit Parquet metric partitions (capture + backfill
  via temporary worktrees), duckdb parquet feature, gitutil helpers
- gate-evaluation logging: CTX_GATE_LOG appends JSONL records from ctx score
- opt-in blocking Stop hook via CTX_GATE_BLOCKING (default stays non-blocking)
- deterministic seeded fixture generator (src/fixture.rs) for perf benches
  and the future ctx-bench suite
- study scripts: rework-rate.sh, revert-rate.sh; benchmark run-record schema
- ctx sql --snapshots: materialized snap.* tables over snapshot partitions
  (loaded before sandbox hardening), canned trend queries in the schema docs
- perf/ companion crate: perf-harness with budgeted E2E scenarios over
  generated fixtures, RSS via wait4, baseline compare, criterion hotpath
  benches; [profile.perf]; advisory perf job in CI
- snapshot.yml workflow: append per-merge partitions to the ctx-snapshots
  data branch
- drop tautological test_server_compiles (clippy assert!(true) lint)
- docs: ctx snapshot command reference, gate-logging and blocking-gate
  sections, snapshot JSON envelopes, CI-enforced performance budget table
  in architecture docs, perf-gates contributing notes, changelog
- render DuckDB TIMESTAMP values as RFC3339 in sql output
- dedupe test git helpers into ctx::testutil::git_stdout and collapse
  near-identical tests (branch now passes its own new_duplication gate)
…rs, docs restructure)

Conflict resolutions: keep both new harness_cli test sections (stop-hook
gates + settings auto-wiring), place commands/snapshot in the new
Governance sidebar category beside sql, stack [Unreleased] above [0.3.1]
in the changelog.
@saldestechnology
saldestechnology merged commit 98e0638 into main Jul 10, 2026
8 checks passed
@saldestechnology
saldestechnology deleted the feat/benchmark-instrumentation branch July 10, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant