Skip to content

perf: reduce benchmark suite runtime from 8min to 35s#20

Merged
bonega merged 1 commit intomasterfrom
perf/faster-benchmarks
Jan 19, 2026
Merged

perf: reduce benchmark suite runtime from 8min to 35s#20
bonega merged 1 commit intomasterfrom
perf/faster-benchmarks

Conversation

@bonega
Copy link
Copy Markdown
Owner

@bonega bonega commented Jan 19, 2026

Summary

  • Add size-based timing configuration to reduce benchmark overhead for small inputs
  • Reduce nresamples from 100k to 10k for faster bootstrap analysis
  • Achieves ~93% reduction in benchmark runtime while maintaining statistical validity

Changes

Input Size Warmup Measurement
≤64 bytes 100ms 200ms
65-512 bytes 100ms 300ms
>512 bytes 200ms 500ms

Variance Testing (3 runs)

Category Avg CV%
ASCII fast path 0.7%
Small sizes (8-64) 2.3%
Medium sizes (256-512) 3.8%
Large sizes (1024+) 3.2%
Total runtime 0.3%

Test plan

  • Run benchmarks 3 times to verify consistent ~35s runtime
  • Verify CV% remains acceptable (<5% for most benchmarks)

Add size-based timing configuration to benchmarks:
- Small inputs (≤64 bytes): 100ms warmup, 200ms measurement
- Medium inputs (65-512 bytes): 100ms warmup, 300ms measurement
- Large inputs (>512 bytes): 200ms warmup, 500ms measurement

Also reduce nresamples from 100k to 10k for faster bootstrap analysis.

Variance testing shows <5% CV for most benchmarks, with ASCII fast
path achieving <1% CV. Total runtime is consistent at ~35s (0.3% CV).
@bonega bonega enabled auto-merge January 19, 2026 20:51
@bonega bonega disabled auto-merge January 19, 2026 20:51
@bonega bonega enabled auto-merge (squash) January 19, 2026 20:51
@bonega bonega merged commit c8259c2 into master Jan 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant