feat: benchmarking instrumentation (snapshots, gate logging, perf budgets)#20
Merged
Merged
Conversation
- ctx snapshot: per-commit Parquet metric partitions (capture + backfill via temporary worktrees), duckdb parquet feature, gitutil helpers - gate-evaluation logging: CTX_GATE_LOG appends JSONL records from ctx score - opt-in blocking Stop hook via CTX_GATE_BLOCKING (default stays non-blocking) - deterministic seeded fixture generator (src/fixture.rs) for perf benches and the future ctx-bench suite - study scripts: rework-rate.sh, revert-rate.sh; benchmark run-record schema
- ctx sql --snapshots: materialized snap.* tables over snapshot partitions (loaded before sandbox hardening), canned trend queries in the schema docs - perf/ companion crate: perf-harness with budgeted E2E scenarios over generated fixtures, RSS via wait4, baseline compare, criterion hotpath benches; [profile.perf]; advisory perf job in CI - snapshot.yml workflow: append per-merge partitions to the ctx-snapshots data branch - drop tautological test_server_compiles (clippy assert!(true) lint)
- docs: ctx snapshot command reference, gate-logging and blocking-gate sections, snapshot JSON envelopes, CI-enforced performance budget table in architecture docs, perf-gates contributing notes, changelog - render DuckDB TIMESTAMP values as RFC3339 in sql output - dedupe test git helpers into ctx::testutil::git_stdout and collapse near-identical tests (branch now passes its own new_duplication gate)
…rs, docs restructure) Conflict resolutions: keep both new harness_cli test sections (stop-hook gates + settings auto-wiring), place commands/snapshot in the new Governance sidebar category beside sql, stack [Unreleased] above [0.3.1] in the changelog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Phase 0–1 of the benchmarking methodology plus the Program C performance budgets, all inside the ctx repo. Built to instrument the longitudinal field study (does enabling gates improve real quality trends?) and to enforce hook-path latency in CI.
What's included
Measurement instruments (Program B / Phase 0–1)
ctx snapshot— per-commit Parquet metric partitions (.ctx/snapshots/sha=<sha>/{symbols,files,dup_pairs,meta}.parquet) with per-symbol and per-file metrics, near-duplicate pairs, rule-violation counts, and churn.ctx snapshot backfill --since <REF> [--every N]builds history through temporary git worktrees (churn--until-anchored to each commit's date). Atomic partition writes, idempotent skips,--jsonenvelopes.ctx sql --snapshots[=DIR]— materializessnap.files/symbols/dup_pairs/metatables before sandbox hardening (external access stays disabled at query time; regression-tested). Canned duplication/violation/hotspot-mass trend queries documented in the SQL schema reference. TIMESTAMP values now render as RFC3339.CTX_GATE_LOGmakesctx scoreappend one JSONL record per gate evaluation (.ctx/gate-log.jsonlby default). Opt-in, local-only, best-effort; never changes exit codes.CTX_GATE_BLOCKING=1turns a failedctx score --fail-oninto a blocking stop (exit 2); default stays non-blocking, operational errors always fail open. Re-runctx harness initto regenerate hooks.scripts/rework-rate.sh(30-day line-survival rework rate viagit blame) andscripts/revert-rate.sh; both POSIX sh, repo-agnostic, integration-tested against synthetic dated repos.snapshot.ymlworkflow — appends a partition per merge to thectx-snapshotsorphan data branch (bootstrap + steady-state + idempotent re-run verified against a scratch remote).Performance budgets (Program C)
ctx::fixture— deterministic seeded repo generator (byte-identical trees and commit SHAs per seed; zipf-skewed fan-in so PageRank is non-trivial;repo_2k/repo_150k_locpresets;apply_change_setfor identical per-run work). Ships in the crate for reuse by the future ctx-bench suite.perf/companion crate (not published, not a workspace member) —perf-harnessspawns the built binary against generated fixtures: budgets of 300ms incremental index, 2s score, 1s check, 500ms map, 500ms sql (2,000-file fixture), 60s cold 150k-LOC index, 300MB RSS (viawait4/ru_maxrss), plus a 1.20× regression gate against committed baselines. AdvisoryperfCI job (continue-on-error) withCTX_PERF_BUDGET_SCALE=1.5; new thin-LTO[profile.perf]. Criterion microbenches for library hot paths.Notes for review
perf/baselines/README.md. The compare logic warn-passes on missing entries during the interim.ctx score --against HEADon the 2,000-file fixture measured ~4.2s / 379MB RSS locally (macOS, debug of the scenario protocol under[profile.perf]) — over both the 2s budget and the RSS ceiling. The advisory CI period will show ubuntu numbers; if they agree, score's hook path needs optimization before the job flips to required.ctx checkclean,ctx score --against mainshowsnew_duplication 0after deduplicating test helpers intoctx::testutil::git_stdout.Test plan
--all-features; 385 green with--no-default-features(Windows stub paths:ctx snapshot/ctx sqlexit 2 with a clear message).cargo clippy --all-features --all-targets -- -D warningsclean;cargo fmt --checkclean;cargo publish --dry-rungreen (noperf/leakage;cargo package --listverified).onBrokenLinks: 'throw'.stop.sh; snapshot workflow's branch-append rehearsed against a scratch bare remote.🤖 Generated with Claude Code