You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added opt-in practical-significance and Schuirmann TOST equivalence gating
(--threshold <delta> on bench_press run, report, and diff, plus defaults.threshold in bench_press.yaml, accepting relative fractions in [0.0, 1.0) such as 0.02 for ±2%). When threshold > 0, directional claims
require the 95% Fieller CI to clear 1.0 + threshold or 1.0 / (1.0 + threshold), ✅ Equivalent requires the 90% Fieller CI to lie
completely inside [1.0 / (1.0 + threshold), 1.0 + threshold], and intervals
straddling the equivalence boundary render as ➖ ⚪ Inconclusive (with a
separate ⚪ N Inconclusive count in the comparison summary footer).
Added multi-process benchmark orchestration (--processes <P> and defaults.processes in bench_press.yaml), compiling each (coordinate, runtime) cell once upfront and interleaving process launches in ABBA BAAB order across rounds p = 0 .. P - 1.
Added BenchmarkMetrics.fromProcesses and ProcessVarianceComponents
(exported from package:bench_press/bench_press.dart, serialized under metrics.variance_components, and surfaced as Processes and Between-Process Share columns in MarkdownReporter.renderSummaryTable when processes > 1) to decompose between-process and within-process variance
using a drift-immune first-difference estimator (ms_within).
Added BenchmarkConfig.defaultTrials = 16 and updated the default trials
count across BenchmarkConfig, bench_press.yaml, and bench_press run from 15 to 16 so that the four ABBA BAAB interleaving visits divide evenly
into [4, 4, 4, 4] trials per visit.
Added paired (bool, default false) to FiellerInterval and FiellerInterval.compute to compute covariance-adjusted Fieller confidence
intervals (df = n - 1) when comparing equal-length paired sample vectors.
Exported FiellerInterval and EnvironmentInfo from package:bench_press/bench_press.dart (alongside BenchmarkMetrics).
External benchmark harnesses—such as browser and UI drivers that collect trial
samples out-of-process via CDP or WebDriver rather than running an in-process Benchmark loop—use bench_press directly for statistical aggregation
(BenchmarkMetrics.fromSamples), ratio confidence intervals
(FiellerInterval.compute), and host telemetry provenance
(EnvironmentInfo.current).
bench_press run --target now narrows a matrix with a runtime axis to the
requested targets instead of being silently overridden by it, and exits with a
usage error when no coordinate matches. Coordinates without a runtime axis and
runs that do not pass --target are unchanged.