Skip to content

v0.0.4: tak compare

Choose a tag to compare

@jdx jdx released this 27 Jul 01:04
Immutable release. Only release title and notes can be modified.
15da0fc

This release adds tak compare, which turns recorded measurements into a pass/fail gate for a pull request, and fixes a v0.0.3 regression where a broken tak.toml could abort commands that never read benchmarks.

Added

  • tak compare diffs git-notes measurements between a base revision (default origin/main) and a head (default HEAD), prints a markdown table, and exits non-zero when instruction counts rise more than gate_pct (default 1%). By @jdx in #22.

    $ tak compare v1.30.0 --rev v1.33.0
    

    Only instruction counts are gated. Wall clock is reported beside them for context and never gated, because on identical hardware it moves 4-20% run to run — a threshold tight enough to catch a real regression fires constantly. The comparison is keyed by benchmark, tool, and runner class: measurements from different runner types do not line up (counts shift between machine types by more than a real regression does), so they show up as added or removed rather than compared. Duplicate records reduce to their minimum, matching how tak reduces within a run, and a benchmark that stops running is called out rather than silently dropped from the table. Flags: global --gate-pct (also TAK_GATE_PCT), --no-credit, and --no-gate on compare.

Fixed

  • A broken tak.toml no longer aborts commands that ignore it. resolve_settings ran before dispatch for every subcommand and validated every benchmark, so an invalid [bench.x] would abort tak run -- ./mycli, tak push, and tak doctor even though none of them read benchmarks. Resolution is now lazy — only run, backfill, and settings consult the file — and reads only the settings tables, so an invalid benchmark no longer blocks an explicit command. A TOML syntax error still fails the measuring commands, since the file may carry [env] rules deciding what is scrubbed from a subject's environment and silently applying a weaker filter is worse than refusing. This fix was approved for v0.0.3 but a commit never landed on main. By @jdx in #20.

  • tak history and tak compare now resolve annotated tags correctly. git rev-parse v1.2.3 on an annotated tag returns the tag object, but notes hang off commits, so naming a tagged revision read as "nothing recorded" — exactly the revisions people are most likely to name. notes::rev_parse now peels to the underlying commit. By @jdx in #22.

Full Changelog: v0.0.3...v0.0.4