ci(bench): bump per-target timeout_minutes to 45/60/45 (fixes #152)#153
Conversation
… shards finish The bench-matrix runs seven strategy shards per target. The heaviest shards exceed the previous per-target caps on every main push: * lazy shard runs 11 levels (5..15) sequentially through one bench binary invocation. On any target this takes 25-30+ minutes, hitting the cap before the benchmark frame even starts on the last level. * fast shard runs 8 levels (-7..-1, 1). On i686-gnu (32-bit, ~30% slower) this consistently exceeds the previous 30m cap. * btultra2 shard runs 3 expensive levels (20..22). On i686-gnu the matcher passes alone push it past 30m. When a single shard hits its job timeout the whole CI run is marked failed, so benchmark-aggregate and benchmark-pages never run and the dashboard at https://structured-world.github.io/structured-zstd/dev/bench/ gets no new data point for that commit. This affected every main push today (#148, #151, #138 → v0.0.21 release). Bump per-target timeout_minutes: x86_64-gnu 25 -> 45, i686-gnu 30 -> 60, x86_64-musl 25 -> 45. The 32-bit target gets a higher absolute cap because its bench loop is consistently ~30% slower. Closes #152
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe CI workflow configuration increases timeout limits for three benchmark targets in the ChangesBenchmark CI Timeout Configuration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~5 minutes Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
Adjusts the GitHub Actions benchmark matrix job time budgets so long-running benchmark shards (notably fast, lazy, and btultra2 on i686) complete successfully, allowing downstream aggregation and pages publishing to run reliably.
Changes:
- Increased per-target
timeout_minutesfor benchmark shards to reduce shard timeouts and unblockbenchmark-aggregate/benchmark-pages.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![sw-release-bot[bot]](https://avatars.githubusercontent.com/u/228206483?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'structured-zstd vs C FFI (x86_64-gnu)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.30.
| Benchmark suite | Current: db5b0a4 | Previous: 1e0f695 | Ratio |
|---|---|---|---|
compress/level_22_btultra2/small-4k-log-lines/matrix/c_ffi |
0.112 ms |
0.067 ms |
1.67 |
compress/level_22_btultra2/decodecorpus-z000033/matrix/c_ffi |
227.505 ms |
171.961 ms |
1.32 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @polaz
The 45/60/45 caps from PR #153 still aren't enough for the slowest shards (lazy, fast). Log inspection of cancelled jobs (job 76419168123, sha f3a6dad) shows linear progress with ~9s per criterion iteration and no stalls — the bench suite simply has more combinations (level × scenario × codec_side × stream_variant × {compress, decompress}) than fit in 45 min: ~11 levels × ~5 scenarios × 2 sides × 2 stream variants × 2 ops ≈ 440 iterations × 9s ≈ 66 min worst-case shard Uniform 120 min cap (GH-hosted limit is 360 min) gives ~50% headroom on the slowest shards and unblocks the publish chain so the dev/bench dashboard stays current.
Summary
Today's main pushes (#148, #151, #138 → v0.0.21) all failed to publish bench data to https://structured-world.github.io/structured-zstd/dev/bench/ because 7 of the 21 bench shards hit the per-target
timeout_minutescap. When any shard fails,benchmark-aggregateandbenchmark-pagesnever run, so no points land on the dashboard.Which shards timed out
The bench-matrix runs 7 strategy shards on 3 targets in parallel. Of the 21 shards:
lazyon all 3 targets (11 levels: 5..15)faston all 3 targets (8 levels: -7..-1, 1)btultra2on i686-gnu (3 levels but slow 32-bit target)Each level takes ~2-3 minutes through the bench binary; 11 levels exceed the 25-30 min caps.
Fix
Bump per-target
timeout_minutes:x86_64-gnu: 25 → 45i686-gnu: 30 → 60 (slower 32-bit target)x86_64-musl: 25 → 45This was user's explicit choice over splitting heavy shards (which would parallelize better but increase the CI surface). The bumped caps leave ~50% headroom over observed worst cases:
Test plan
benchmark-pagesjob publishes a new data point on the dashboardCloses #152
Summary by CodeRabbit