Skip to content

package:bench_press v0.4.0

Choose a tag to compare

@kevmoo kevmoo released this 30 Sep 22:02
· 11 commits to main since this release
ff91b0e
  • Breaking (library exports): package:bench_press/bench_press.dart now
    exports only the benchmark-authoring (Benchmark, AsyncBenchmark,
    BenchmarkGroup, BenchmarkMatrix, BenchmarkVariant, BenchmarkConfig,
    Throughput, ByteThroughput, ElementThroughput, Blackhole), suite
    entrypoint (mainBenchmark*), and .report() result (BenchmarkResult,
    BenchmarkMetrics, CalibratedBatch, WarmupResult) APIs. Internal CLI,
    subprocess runner, compiler, statistical helper, and telemetry schema types
    that were previously re-exported in 0.3.1 are now internal to lib/src/.
  • Breaking (config): removed the matrix.entrypoints key from
    bench_press.yaml. It never chose which file ran: each value re-ran the same
    discovered benchmark files under a made-up entrypoint coordinate, so the
    matrix table compared identical runs. Pass benchmark files as positional paths
    to bench_press run instead. A leftover entrypoints: key is ignored.
  • Breaking (behavioral): ByteThroughput.formatRate now scales byte rates
    by decimal 1000 (KB/s, MB/s, GB/s) instead of 1024, matching
    hardware memory bandwidth, network I/O conventions, and ElementThroughput.
  • Breaking (behavioral): bench_press diff and bench_press run --diff now
    pair cells by full key (name, target, and every coordinate, including group)
    instead of by name and target alone. A name reused across groups was
    previously compared against the first match in the other run. The Benchmark
    column now shows coordinates, for example encode (group=A). Cells present in
    only one run are listed in an Unmatched note, including when nothing pairs.
  • Breaking (behavioral): an explicitly configured Dart SDK (sdk matrix
    axis in bench_press.yaml) is now authoritative and fails fast with a
    specific diagnostic explaining why the path was rejected instead of silently
    falling back to dart on PATH.
  • BenchmarkGroup.report, BenchmarkMatrix.report, and mainBenchmarkSuite
    now interleave measurement trials across group variants in ABBA BAAB rounds
    after every variant has finished setup, warmup, warmupComplete, and batch
    calibration. Previously, all trials of variant A ran before variant B
    started, so a linear thermal ramp or a short host contention window during B
    produced a false speedup or regression while both variants still passed the
    within-variant isStable gate. When maxTrials is set, lockstep rounds
    continue across all variants in the group while any variant's CV exceeds 5%.
  • bench_press run now honors defaults.output in bench_press.yaml, both for
    saving results and for the file run --diff looks up. An explicit --output
    or --save still takes precedence, and benchmark_results.json remains the
    fallback. A leading ~ expands to the home directory. The value must be a
    non-empty string; anything else is now a config error, where it was previously
    ignored. bench_press report and bench_press diff do not read it; pass them
    the path.
  • Added --pin-cpu <cpu-list> to bench_press run to pin benchmark
    subprocesses (VM JIT, AOT, Node.js, and D8) via taskset -c on Linux.
  • Added Fieller confidence interval and isStable speedup gating (gate: true
    by default, configurable via --[no-]gate in bench_press run, report, and
    diff). Unstable or unbounded-CI comparisons render as unresolved and are
    excluded from geometric mean rollups; a resolved comparison whose 95% CI
    contains 1.00x renders as ➖ ⚪ Neutral.
  • Added Markdown report warning banners to flag benchmarks whose declared
    Throughput.bytes rate exceeds physical memory bandwidth (100 GB/s) or
    whose latency remains invariant (0.9x–1.25x) across >=8x payload size
    spreads for a shared benchmark name across groups.
  • Derived post-warmup calibration batch sizes directly from steady-state warmup
    convergence latencies (recorded in warmup.estimated_op_ns), and added a
    Markdown report footnote listing any cell whose trial median moved more than
    25% from the warmup estimate that sized its batch.
  • Updated bench_press run and bench_press validate to discover and execute
    all positional file and directory paths supplied on the command line rather
    than silently ignoring arguments after the first path.
  • Invoked warmupComplete() lifecycle hook during BenchmarkVariant execution,
    forwarded throughput and warmupComplete() for Benchmark and
    AsyncBenchmark instances run via mainBenchmark*, and preserved
    programmatic BenchmarkConfig values unless explicitly overridden by CLI
    flags.
  • Fixed Markdown suite reporting so suites mixing standalone benchmarks and
    BenchmarkGroup variants collate grouped variants into a single multi-row
    ### Group: ... comparison table with the baseline ordered first and a
    geometric mean summary footer, and stopped wrapping Markdown tables in
    mdformat off / mdformat on HTML-comment guards.