Skip to content

package:bench_press v0.4.2

Latest

Choose a tag to compare

@kevmoo kevmoo released this 25 Oct 03:52
5fd6396
  • Added opt-in practical-significance and Schuirmann TOST equivalence gating
    (--threshold <delta> on bench_press run, report, and diff, plus
    defaults.threshold in bench_press.yaml, accepting relative fractions in
    [0.0, 1.0) such as 0.02 for ±2%). When threshold > 0, directional claims
    require the 95% Fieller CI to clear 1.0 + threshold or
    1.0 / (1.0 + threshold), ✅ Equivalent requires the 90% Fieller CI to lie
    completely inside [1.0 / (1.0 + threshold), 1.0 + threshold], and intervals
    straddling the equivalence boundary render as ➖ ⚪ Inconclusive (with a
    separate ⚪ N Inconclusive count in the comparison summary footer).
  • Added multi-process benchmark orchestration (--processes <P> and
    defaults.processes in bench_press.yaml), compiling each
    (coordinate, runtime) cell once upfront and interleaving process launches in
    ABBA BAAB order across rounds p = 0 .. P - 1.
  • Added BenchmarkMetrics.fromProcesses and ProcessVarianceComponents
    (exported from package:bench_press/bench_press.dart, serialized under
    metrics.variance_components, and surfaced as Processes and
    Between-Process Share columns in MarkdownReporter.renderSummaryTable when
    processes > 1) to decompose between-process and within-process variance
    using a drift-immune first-difference estimator (ms_within).
  • Added BenchmarkConfig.defaultTrials = 16 and updated the default trials
    count across BenchmarkConfig, bench_press.yaml, and bench_press run from
    15 to 16 so that the four ABBA BAAB interleaving visits divide evenly
    into [4, 4, 4, 4] trials per visit.
  • Added paired (bool, default false) to FiellerInterval and
    FiellerInterval.compute to compute covariance-adjusted Fieller confidence
    intervals (df = n - 1) when comparing equal-length paired sample vectors.
  • Exported FiellerInterval and EnvironmentInfo from
    package:bench_press/bench_press.dart (alongside BenchmarkMetrics).
    External benchmark harnesses—such as browser and UI drivers that collect trial
    samples out-of-process via CDP or WebDriver rather than running an in-process
    Benchmark loop—use bench_press directly for statistical aggregation
    (BenchmarkMetrics.fromSamples), ratio confidence intervals
    (FiellerInterval.compute), and host telemetry provenance
    (EnvironmentInfo.current).
  • bench_press run --target now narrows a matrix with a runtime axis to the
    requested targets instead of being silently overridden by it, and exits with a
    usage error when no coordinate matches. Coordinates without a runtime axis and
    runs that do not pass --target are unchanged.