You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Breaking (library exports): package:bench_press/bench_press.dart now
exports only the benchmark-authoring (Benchmark, AsyncBenchmark, BenchmarkGroup, BenchmarkMatrix, BenchmarkVariant, BenchmarkConfig, Throughput, ByteThroughput, ElementThroughput, Blackhole), suite
entrypoint (mainBenchmark*), and .report() result (BenchmarkResult, BenchmarkMetrics, CalibratedBatch, WarmupResult) APIs. Internal CLI,
subprocess runner, compiler, statistical helper, and telemetry schema types
that were previously re-exported in 0.3.1 are now internal to lib/src/.
Breaking (config): removed the matrix.entrypoints key from bench_press.yaml. It never chose which file ran: each value re-ran the same
discovered benchmark files under a made-up entrypoint coordinate, so the
matrix table compared identical runs. Pass benchmark files as positional paths
to bench_press run instead. A leftover entrypoints: key is ignored.
Breaking (behavioral): ByteThroughput.formatRate now scales byte rates
by decimal 1000 (KB/s, MB/s, GB/s) instead of 1024, matching
hardware memory bandwidth, network I/O conventions, and ElementThroughput.
Breaking (behavioral): bench_press diff and bench_press run --diff now
pair cells by full key (name, target, and every coordinate, including group)
instead of by name and target alone. A name reused across groups was
previously compared against the first match in the other run. The Benchmark
column now shows coordinates, for example encode (group=A). Cells present in
only one run are listed in an Unmatched note, including when nothing pairs.
Breaking (behavioral): an explicitly configured Dart SDK (sdk matrix
axis in bench_press.yaml) is now authoritative and fails fast with a
specific diagnostic explaining why the path was rejected instead of silently
falling back to dart on PATH.
BenchmarkGroup.report, BenchmarkMatrix.report, and mainBenchmarkSuite
now interleave measurement trials across group variants in ABBA BAAB rounds
after every variant has finished setup, warmup, warmupComplete, and batch
calibration. Previously, all trials of variant A ran before variant B
started, so a linear thermal ramp or a short host contention window during B
produced a false speedup or regression while both variants still passed the
within-variant isStable gate. When maxTrials is set, lockstep rounds
continue across all variants in the group while any variant's CV exceeds 5%.
bench_press run now honors defaults.output in bench_press.yaml, both for
saving results and for the file run --diff looks up. An explicit --output
or --save still takes precedence, and benchmark_results.json remains the
fallback. A leading ~ expands to the home directory. The value must be a
non-empty string; anything else is now a config error, where it was previously
ignored. bench_press report and bench_press diff do not read it; pass them
the path.
Added --pin-cpu <cpu-list> to bench_press run to pin benchmark
subprocesses (VM JIT, AOT, Node.js, and D8) via taskset -c on Linux.
Added Fieller confidence interval and isStable speedup gating (gate: true
by default, configurable via --[no-]gate in bench_press run, report, and diff). Unstable or unbounded-CI comparisons render as unresolved and are
excluded from geometric mean rollups; a resolved comparison whose 95% CI
contains 1.00x renders as ➖ ⚪ Neutral.
Added Markdown report warning banners to flag benchmarks whose declared Throughput.bytes rate exceeds physical memory bandwidth (100 GB/s) or
whose latency remains invariant (0.9x–1.25x) across >=8x payload size
spreads for a shared benchmark name across groups.
Derived post-warmup calibration batch sizes directly from steady-state warmup
convergence latencies (recorded in warmup.estimated_op_ns), and added a
Markdown report footnote listing any cell whose trial median moved more than
25% from the warmup estimate that sized its batch.
Updated bench_press run and bench_press validate to discover and execute
all positional file and directory paths supplied on the command line rather
than silently ignoring arguments after the first path.
Invoked warmupComplete() lifecycle hook during BenchmarkVariant execution,
forwarded throughput and warmupComplete() for Benchmark and AsyncBenchmark instances run via mainBenchmark*, and preserved
programmatic BenchmarkConfig values unless explicitly overridden by CLI
flags.
Fixed Markdown suite reporting so suites mixing standalone benchmarks and BenchmarkGroup variants collate grouped variants into a single multi-row ### Group: ... comparison table with the baseline ordered first and a
geometric mean summary footer, and stopped wrapping Markdown tables in mdformat off / mdformat on HTML-comment guards.