Skip to content

feat: add Throughput to all criterion benchmarks so critcmp shows ops/sec#78

Merged
deepjoy merged 1 commit into
mainfrom
bench-add-throughput
Mar 22, 2026
Merged

feat: add Throughput to all criterion benchmarks so critcmp shows ops/sec#78
deepjoy merged 1 commit into
mainfrom
bench-add-throughput

Conversation

@deepjoy
Copy link
Copy Markdown
Owner

@deepjoy deepjoy commented Mar 22, 2026

  • All criterion benchmarks now declare Throughput::Elements(N) so critcmp
    reports real throughput (tasks/sec, queries/sec) instead of ? ?/sec.
  • Standalone bench_function calls are converted to BenchmarkGroups since
    criterion only exposes throughput on groups. This changes benchmark names
    from flat (e.g. submit_1000_tasks) to structured (e.g. submit_tasks/1000).
  • Parameterized benchmarks (dependency chains, fan-in) set throughput per
    parameter to reflect the actual element count at each size.

…/sec

Standalone bench_function calls are converted to BenchmarkGroups since
criterion only supports Throughput on groups. Each benchmark now declares
the number of elements processed per iteration so critcmp can compute
and display meaningful throughput (e.g. tasks/sec, queries/sec) instead
of "? ?/sec".
@deepjoy deepjoy enabled auto-merge (squash) March 22, 2026 21:09
@deepjoy deepjoy merged commit 773c31d into main Mar 22, 2026
2 checks passed
@github-actions github-actions Bot mentioned this pull request Mar 22, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Comparison

Click to expand
group                                       main                                    pr
-----                                       ----                                    --
backoff_delay/constant                      1.05     50.1±0.09ns        ? ?/sec     1.00     47.9±0.11ns 397.9 MElem/sec
backoff_delay/exponential                   1.00    187.0±0.49ns        ? ?/sec     1.00    187.2±1.09ns 101.9 MElem/sec
backoff_delay/exponential_jitter            1.00    269.9±2.52ns        ? ?/sec     1.00    269.2±0.90ns 70.8 MElem/sec
backoff_delay/linear                        1.00     76.3±0.17ns        ? ?/sec     1.06     81.1±1.00ns 235.1 MElem/sec
batch_submit/1000                                                                   1.00     32.7±2.65ms 29.8 KElem/sec
batch_submit_1000                           1.00     34.0±2.40ms        ? ?/sec   
byte_progress/byte_reporting_500            1.03    242.2±6.15ms        ? ?/sec     1.00    234.3±5.53ms  2.1 KElem/sec
byte_progress/noop_500                      1.03    241.2±4.08ms        ? ?/sec     1.00    233.5±4.07ms  2.1 KElem/sec
byte_progress_snapshot/100_tasks                                                    1.00     78.3±2.25ms  1277 Elem/sec
byte_progress_snapshot_100_tasks            1.00     80.3±1.89ms        ? ?/sec   
concurrency_scaling/1                       1.03    374.3±4.28ms        ? ?/sec     1.00    363.2±3.79ms  1376 Elem/sec
concurrency_scaling/2                       1.03    310.4±3.40ms        ? ?/sec     1.00    301.5±3.86ms  1658 Elem/sec
concurrency_scaling/4                       1.03    267.8±4.60ms        ? ?/sec     1.00    259.8±5.00ms  1924 Elem/sec
concurrency_scaling/8                       1.03    241.8±4.12ms        ? ?/sec     1.00    235.0±4.09ms  2.1 KElem/sec
count_by_tags/100                           1.03    127.7±2.85µs        ? ?/sec     1.00    124.2±2.61µs  7.9 KElem/sec
count_by_tags/1000                          1.02    215.5±2.86µs        ? ?/sec     1.00    211.0±4.03µs  4.6 KElem/sec
count_by_tags/5000                          1.03    616.1±7.04µs        ? ?/sec     1.00    597.9±3.24µs  1672 Elem/sec
dep_chain_dispatch/10                       1.04     10.9±0.10ms        ? ?/sec     1.00     10.5±0.13ms   956 Elem/sec
dep_chain_dispatch/25                       1.05     26.6±0.62ms        ? ?/sec     1.00     25.2±0.44ms   990 Elem/sec
dep_chain_dispatch/50                       1.04     52.8±0.88ms        ? ?/sec     1.00     50.6±0.74ms   988 Elem/sec
dep_chain_submit/10                         1.05      3.1±0.13ms        ? ?/sec     1.00      2.9±0.10ms  3.3 KElem/sec
dep_chain_submit/200                        1.04     78.0±3.76ms        ? ?/sec     1.00     75.3±3.44ms  2.6 KElem/sec
dep_chain_submit/50                         1.06     17.0±0.76ms        ? ?/sec     1.00     16.1±0.78ms  3.0 KElem/sec
dep_fan_in_dispatch/10                      1.04      7.0±0.09ms        ? ?/sec     1.00      6.8±0.09ms  1625 Elem/sec
dep_fan_in_dispatch/100                     1.03     52.0±1.04ms        ? ?/sec     1.00     50.6±0.76ms  1997 Elem/sec
dep_fan_in_dispatch/50                      1.03     26.7±0.64ms        ? ?/sec     1.00     25.9±0.48ms  1969 Elem/sec
dispatch_and_complete/1000                                                          1.00    471.3±5.46ms  2.1 KElem/sec
dispatch_and_complete_1000                  1.00    485.0±5.97ms        ? ?/sec   
dispatch_group_scaling/1                    1.04    417.5±5.31ms        ? ?/sec     1.00    401.0±4.33ms  1246 Elem/sec
dispatch_group_scaling/10                   1.04    416.9±5.01ms        ? ?/sec     1.00    401.6±5.68ms  1244 Elem/sec
dispatch_group_scaling/100                  1.03    416.1±4.47ms        ? ?/sec     1.00    402.1±4.87ms  1243 Elem/sec
dispatch_group_scaling/50                   1.04    417.0±5.43ms        ? ?/sec     1.00    401.3±5.05ms  1245 Elem/sec
dispatch_no_groups/500                                                              1.00    235.0±4.17ms  2.1 KElem/sec
dispatch_no_groups_500                      1.00    243.2±3.65ms        ? ?/sec   
dispatch_one_group/500                                                              1.00    393.5±5.05ms  1270 Elem/sec
dispatch_one_group_500                      1.00    415.1±6.92ms        ? ?/sec   
dispatch_permanent_failure/500                                                      1.00    349.3±3.71ms  1431 Elem/sec
dispatch_permanent_failure_500              1.00    362.2±3.71ms        ? ?/sec   
history_by_type/100                         1.01   889.4±19.51µs        ? ?/sec     1.00   882.8±28.45µs  1132 Elem/sec
history_by_type/1000                        1.03   922.8±38.92µs        ? ?/sec     1.00   894.0±28.25µs  1118 Elem/sec
history_by_type/5000                        1.01   921.5±26.39µs        ? ?/sec     1.00   913.9±19.73µs  1094 Elem/sec
history_query/100                           1.02    543.1±8.12µs        ? ?/sec     1.00   534.7±16.11µs  1870 Elem/sec
history_query/1000                          1.02    545.1±7.17µs        ? ?/sec     1.00    537.0±8.08µs  1862 Elem/sec
history_query/5000                          1.02    559.8±6.39µs        ? ?/sec     1.00    548.0±7.18µs  1824 Elem/sec
history_stats/100                           1.03    148.2±0.73µs        ? ?/sec     1.00    143.9±0.54µs  6.8 KElem/sec
history_stats/1000                          1.02    367.3±1.28µs        ? ?/sec     1.00    359.7±0.69µs  2.7 KElem/sec
history_stats/5000                          1.00   1331.5±7.62µs        ? ?/sec     1.01   1340.6±2.49µs   745 Elem/sec
mixed_priority_dispatch/500                                                         1.00    259.2±4.99ms  1928 Elem/sec
mixed_priority_dispatch_500                 1.00    268.5±4.56ms        ? ?/sec   
peek_next/100                               1.03    119.8±2.98µs        ? ?/sec     1.00    116.6±2.57µs  8.4 KElem/sec
peek_next/1000                              1.03    120.1±2.98µs        ? ?/sec     1.00    116.2±1.99µs  8.4 KElem/sec
peek_next/5000                              1.02    119.6±2.77µs        ? ?/sec     1.00    117.1±2.49µs  8.3 KElem/sec
query_by_tags/100                           1.00  1238.0±140.88µs        ? ?/sec    1.03  1275.9±163.07µs   783 Elem/sec
query_by_tags/1000                          1.08     10.3±1.30ms        ? ?/sec     1.00      9.6±0.37ms   104 Elem/sec
query_by_tags/5000                          1.00     54.6±1.96ms        ? ?/sec     1.01     55.4±8.62ms    18 Elem/sec
retryable_dead_letter/constant              1.04    147.4±2.17ms        ? ?/sec     1.00    141.8±1.79ms   705 Elem/sec
retryable_dead_letter/exponential           1.03    147.1±1.90ms        ? ?/sec     1.00    142.3±2.35ms   702 Elem/sec
retryable_dead_letter/exponential_jitter    1.03    146.5±1.73ms        ? ?/sec     1.00    141.8±1.53ms   705 Elem/sec
retryable_dead_letter/linear                1.04    147.0±1.97ms        ? ?/sec     1.00    141.7±1.78ms   705 Elem/sec
submit_1000_tasks                           1.00    180.4±4.51ms        ? ?/sec   
submit_dedup_hit/1000                                                               1.00    232.6±7.03ms  4.2 KElem/sec
submit_dedup_hit_1000                       1.00    239.1±7.44ms        ? ?/sec   
submit_tasks/1000                                                                   1.00    176.5±4.73ms  5.5 KElem/sec
submit_with_tags/0                          1.04     90.3±3.36ms        ? ?/sec     1.00     86.9±2.62ms  5.6 KElem/sec
submit_with_tags/10                         1.03    240.1±8.14ms        ? ?/sec     1.00    233.0±9.63ms  2.1 KElem/sec
submit_with_tags/20                         1.03   395.4±17.26ms        ? ?/sec     1.00   383.8±16.63ms  1302 Elem/sec
submit_with_tags/5                          1.04    167.6±7.40ms        ? ?/sec     1.00    160.7±5.67ms  3.0 KElem/sec
tag_values/100                              1.02    133.1±3.84µs        ? ?/sec     1.00    131.0±3.84µs  7.5 KElem/sec
tag_values/1000                             1.01    195.4±2.89µs        ? ?/sec     1.00    194.2±3.73µs  5.0 KElem/sec
tag_values/5000                             1.02    470.6±4.60µs        ? ?/sec     1.00    461.2±5.31µs  2.1 KElem/sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant