Skip to content

test: verify the benchmark regression gate (DO NOT MERGE) - #1

Closed
juicycleff wants to merge 1 commit into
mainfrom
test/bench-gate-verification
Closed

test: verify the benchmark regression gate (DO NOT MERGE)#1
juicycleff wants to merge 1 commit into
mainfrom
test/bench-gate-verification

Conversation

@juicycleff

@juicycleff juicycleff commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Throwaway PR — do not merge. Close once the gate is confirmed.

Exists to prove .github/workflows/bench.yml actually fails a pull request when
benchmarks regress. A gate that has never been seen to fire is unverified.

What this changes

Four redundant passes over the input in histogramOp.Apply, marked with a
loud DO NOT MERGE comment. Nothing else.

Expected result

The Bench check should fail, with a comment on this PR showing roughly:

Pipe/histogram/n=100-16     +110% (p=0.002 n=6)
Pipe/histogram/n=1000-16    +166% (p=0.002 n=6)
Pipe/histogram/n=10000-16   +177% (p=0.002 n=6)

Measured locally against main at -benchtime=100ms -count=6.

If Bench passes, the gate is broken and needs fixing before it is trusted.

What this confirms

  • base and head checkout in a single job on one runner
  • benchstat installs and parses
  • scripts/bench-gate.sh reads the CSV and applies the 20% threshold
  • the result gets posted back as a PR comment
  • unrelated benchmarks stay quiet (no false positives across the other 16 operators)

🤖 Generated with Claude Code

@juicycleff
test: deliberately slow histogram to verify the bench gate
42d360f 
DO NOT MERGE. Adds four redundant passes over the input in histogramOp.Apply
so the pull-request benchmark gate has a real regression to catch.

Measured locally: +110% at n=100, +166% at n=1000, +177% at n=10000, all
significant at p=0.002. scripts/bench-gate.sh exits 1 on this.
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark comparison

goos: linux
goarch: amd64
pkg: github.com/xraph/dql/exec
cpu: AMD EPYC 7763 64-Core Processor                
                                  │ /tmp/old.txt │            /tmp/new.txt            │
                                  │    sec/op    │    sec/op     vs base              │
ExecuteEndToEnd/classic/n=100-4     50.47µ ± 18%   49.20µ ±  2%  -2.52% (p=0.009 n=6)
ExecuteEndToEnd/classic/n=1000-4    552.3µ ±  1%   546.9µ ±  3%       ~ (p=0.394 n=6)
ExecuteEndToEnd/classic/n=10000-4   5.731m ±  4%   5.711m ± 23%       ~ (p=0.818 n=6)
ExecuteEndToEnd/pipe/n=100-4        93.94µ ±  1%   91.94µ ±  2%  -2.12% (p=0.009 n=6)
ExecuteEndToEnd/pipe/n=1000-4       755.5µ ± 28%   749.5µ ±  1%       ~ (p=0.240 n=6)
ExecuteEndToEnd/pipe/n=10000-4      7.036m ±  2%   6.982m ±  2%       ~ (p=0.699 n=6)
geomean                             656.1µ         647.8µ        -1.26%

                                  │ /tmp/old.txt │            /tmp/new.txt            │
                                  │    rows/s    │    rows/s     vs base              │
ExecuteEndToEnd/classic/n=100-4     1.981M ± 15%   2.032M ±  2%  +2.58% (p=0.009 n=6)
ExecuteEndToEnd/classic/n=1000-4    1.811M ±  1%   1.829M ±  3%       ~ (p=0.394 n=6)
ExecuteEndToEnd/classic/n=10000-4   1.745M ±  5%   1.751M ± 19%       ~ (p=0.818 n=6)
ExecuteEndToEnd/pipe/n=100-4        1.065M ±  1%   1.088M ±  2%  +2.17% (p=0.009 n=6)
ExecuteEndToEnd/pipe/n=1000-4       1.324M ± 22%   1.334M ±  1%       ~ (p=0.240 n=6)
ExecuteEndToEnd/pipe/n=10000-4      1.421M ±  2%   1.432M ±  2%       ~ (p=0.699 n=6)
geomean                             1.524M         1.544M        +1.28%

                                  │ /tmp/old.txt │            /tmp/new.txt            │
                                  │     B/op     │     B/op      vs base              │
ExecuteEndToEnd/classic/n=100-4     56.76Ki ± 0%   56.76Ki ± 0%       ~ (p=0.729 n=6)
ExecuteEndToEnd/classic/n=1000-4    543.0Ki ± 0%   543.0Ki ± 0%       ~ (p=0.165 n=6)
ExecuteEndToEnd/classic/n=10000-4   5.410Mi ± 0%   5.410Mi ± 0%       ~ (p=0.619 n=6)
ExecuteEndToEnd/pipe/n=100-4        81.44Ki ± 0%   81.44Ki ± 0%       ~ (p=0.374 n=6)
ExecuteEndToEnd/pipe/n=1000-4       592.3Ki ± 0%   592.3Ki ± 0%       ~ (p=0.258 n=6)
ExecuteEndToEnd/pipe/n=10000-4      5.686Mi ± 0%   5.686Mi ± 0%       ~ (p=0.461 n=6)
geomean                             602.7Ki        602.7Ki       -0.00%

                                  │ /tmp/old.txt │            /tmp/new.txt             │
                                  │  allocs/op   │  allocs/op   vs base                │
ExecuteEndToEnd/classic/n=100-4       635.0 ± 0%    635.0 ± 0%       ~ (p=1.000 n=6) ¹
ExecuteEndToEnd/classic/n=1000-4     6.038k ± 0%   6.038k ± 0%       ~ (p=0.182 n=6)
ExecuteEndToEnd/classic/n=10000-4    60.05k ± 0%   60.05k ± 0%       ~ (p=1.000 n=6) ¹
ExecuteEndToEnd/pipe/n=100-4          994.0 ± 0%    994.0 ± 0%       ~ (p=1.000 n=6) ¹
ExecuteEndToEnd/pipe/n=1000-4        7.358k ± 0%   7.358k ± 0%       ~ (p=1.000 n=6) ¹
ExecuteEndToEnd/pipe/n=10000-4       70.45k ± 0%   70.45k ± 0%       ~ (p=0.455 n=6)
geomean                              7.010k        7.010k       +0.00%
¹ all samples are equal

pkg: github.com/xraph/dql/parser
                  │ /tmp/old.txt │           /tmp/new.txt            │
                  │    sec/op    │   sec/op     vs base              │
Parse/simple-4       5.358µ ± 2%   5.346µ ± 1%       ~ (p=0.513 n=6)
Parse/pipe-4         19.83µ ± 1%   20.03µ ± 2%  +1.02% (p=0.035 n=6)
Validate/simple-4    114.5n ± 1%   114.9n ± 9%       ~ (p=0.054 n=6)
Validate/pipe-4      8.014µ ± 2%   8.146µ ± 7%  +1.64% (p=0.041 n=6)
geomean              3.142µ        3.164µ       +0.69%

                  │ /tmp/old.txt │             /tmp/new.txt             │
                  │     B/op     │     B/op      vs base                │
Parse/simple-4      1.891Ki ± 0%   1.891Ki ± 0%       ~ (p=1.000 n=6) ¹
Parse/pipe-4        6.391Ki ± 0%   6.391Ki ± 0%       ~ (p=1.000 n=6) ¹
Validate/simple-4     16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=6) ¹
Validate/pipe-4     2.414Ki ± 0%   2.414Ki ± 0%       ~ (p=1.000 n=6) ¹
geomean               841.4          841.4       +0.00%
¹ all samples are equal

                  │ /tmp/old.txt │            /tmp/new.txt            │
                  │  allocs/op   │ allocs/op   vs base                │
Parse/simple-4        35.00 ± 0%   35.00 ± 0%       ~ (p=1.000 n=6) ¹
Parse/pipe-4          125.0 ± 0%   125.0 ± 0%       ~ (p=1.000 n=6) ¹
Validate/simple-4     1.000 ± 0%   1.000 ± 0%       ~ (p=1.000 n=6) ¹
Validate/pipe-4       61.00 ± 0%   61.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean               22.73        22.73       +0.00%
¹ all samples are equal

pkg: github.com/xraph/dql/pipe
                               │ /tmp/old.txt │             /tmp/new.txt              │
                               │    sec/op    │    sec/op      vs base                │
Pipe/filter/n=100-4              14.34µ ±  1%    14.72µ ±  2%    +2.61% (p=0.026 n=6)
Pipe/filter/n=1000-4             145.6µ ± 25%    146.5µ ±  2%         ~ (p=0.818 n=6)
Pipe/filter/n=10000-4            1.465m ±  1%    1.508m ±  3%    +2.96% (p=0.009 n=6)
Pipe/groupBy/n=100-4             1.874n ±  0%    1.904n ±  2%         ~ (p=0.106 n=6)
Pipe/groupBy/n=1000-4            1.871n ±  0%    1.896n ±  3%    +1.39% (p=0.006 n=6)
Pipe/groupBy/n=10000-4           1.893n ±  5%    1.872n ±  2%         ~ (p=0.229 n=6)
Pipe/aggregate/n=100-4           20.82µ ±  1%    20.99µ ±  3%         ~ (p=0.310 n=6)
Pipe/aggregate/n=1000-4          133.2µ ±  1%    136.1µ ±  2%    +2.16% (p=0.002 n=6)
Pipe/aggregate/n=10000-4         1.224m ±  2%    1.236m ±  2%    +1.00% (p=0.041 n=6)
Pipe/sort/n=100-4                5.538µ ±  5%    6.596µ ±  2%   +19.10% (p=0.002 n=6)
Pipe/sort/n=1000-4               56.64µ ±  2%    68.00µ ±  1%   +20.06% (p=0.002 n=6)
Pipe/sort/n=10000-4              787.1µ ±  2%    876.9µ ±  1%   +11.41% (p=0.002 n=6)
Pipe/distinct/n=100-4            11.06µ ±  2%    11.06µ ±  2%         ~ (p=0.937 n=6)
Pipe/distinct/n=1000-4           116.4µ ±  1%    118.6µ ±  4%    +1.88% (p=0.002 n=6)
Pipe/distinct/n=10000-4          1.172m ±  3%    1.188m ± 20%         ~ (p=0.132 n=6)
Pipe/dedupe/n=100-4              10.91µ ±  2%    10.86µ ±  1%         ~ (p=0.589 n=6)
Pipe/dedupe/n=1000-4             115.3µ ±  1%    115.6µ ±  1%         ~ (p=0.589 n=6)
Pipe/dedupe/n=10000-4            1.161m ±  3%    1.168m ±  2%         ~ (p=0.394 n=6)
Pipe/window/n=100-4              31.25µ ±  2%    30.89µ ±  1%         ~ (p=0.240 n=6)
Pipe/window/n=1000-4             580.7µ ±  2%    576.6µ ±  4%         ~ (p=0.180 n=6)
Pipe/window/n=10000-4            9.190m ±  9%   10.120m ±  9%   +10.12% (p=0.015 n=6)
Pipe/topPerGroup/n=100-4         27.71µ ±  2%    27.58µ ±  1%         ~ (p=0.180 n=6)
Pipe/topPerGroup/n=1000-4        503.6µ ±  2%    495.0µ ±  1%    -1.71% (p=0.002 n=6)
Pipe/topPerGroup/n=10000-4       7.825m ±  1%    7.669m ±  2%    -1.99% (p=0.004 n=6)
Pipe/histogram/n=100-4           5.548µ ±  1%   12.063µ ±  1%  +117.45% (p=0.002 n=6)
Pipe/histogram/n=1000-4          39.40µ ±  1%   107.67µ ±  1%  +173.27% (p=0.002 n=6)
Pipe/histogram/n=10000-4         385.5µ ±  1%   1109.7µ ±  3%  +187.87% (p=0.002 n=6)
Pipe/pivot/n=100-4               63.12µ ±  2%    62.08µ ±  2%    -1.65% (p=0.041 n=6)
Pipe/pivot/n=1000-4              521.1µ ±  2%    505.8µ ±  2%    -2.94% (p=0.015 n=6)
Pipe/pivot/n=10000-4             4.845m ±  1%    4.889m ±  2%    +0.92% (p=0.004 n=6)
Pipe/unpivot/n=100-4             21.77µ ±  2%    21.77µ ±  2%         ~ (p=0.937 n=6)
Pipe/unpivot/n=1000-4            281.6µ ±  4%    276.5µ ±  1%    -1.79% (p=0.004 n=6)
Pipe/unpivot/n=10000-4           2.617m ±  2%    2.613m ±  3%         ~ (p=0.818 n=6)
Pipe/gapfill/n=100-4             11.86µ ±  1%    12.04µ ±  1%    +1.50% (p=0.004 n=6)
Pipe/gapfill/n=1000-4            122.7µ ±  5%    123.6µ ±  0%         ~ (p=0.093 n=6)
Pipe/gapfill/n=10000-4           1.354m ±  2%    1.371m ±  1%    +1.29% (p=0.015 n=6)
PipeJoins/lookup/n=100-4         260.9µ ±  2%    264.8µ ±  2%         ~ (p=0.065 n=6)
PipeJoins/lookup/n=1000-4        1.735m ±  2%    1.751m ±  1%         ~ (p=0.240 n=6)
PipeJoins/lookup/n=10000-4       14.89m ±  2%    15.18m ±  2%    +1.91% (p=0.041 n=6)
PipeJoins/asofJoin/n=100-4       266.6µ ±  1%    257.5µ ±  1%    -3.43% (p=0.002 n=6)
PipeJoins/asofJoin/n=1000-4      1.883m ±  2%    1.887m ±  1%         ~ (p=0.240 n=6)
PipeJoins/asofJoin/n=10000-4     15.92m ±  3%    16.34m ± 15%         ~ (p=0.093 n=6)
PipeJoins/crossJoin/n=100-4      1.277m ±  3%    1.278m ±  2%         ~ (p=0.937 n=6)
PipeJoins/crossJoin/n=1000-4     14.65m ±  2%    14.71m ±  3%         ~ (p=0.394 n=6)
PipeJoins/crossJoin/n=10000-4    151.3m ±  2%    146.3m ±  6%         ~ (p=0.180 n=6)
PipeSetOps/except/n=100-4        112.8µ ±  2%    114.4µ ±  1%    +1.38% (p=0.009 n=6)
PipeSetOps/except/n=1000-4       1.256m ±  2%    1.260m ±  2%         ~ (p=0.937 n=6)
PipeSetOps/except/n=10000-4      12.97m ±  2%    12.93m ±  1%         ~ (p=0.818 n=6)
PipeSetOps/intersect/n=100-4     125.8µ ±  1%    123.3µ ±  1%    -1.95% (p=0.002 n=6)
PipeSetOps/intersect/n=1000-4    1.404m ±  2%    1.360m ±  2%    -3.14% (p=0.002 n=6)
PipeSetOps/intersect/n=10000-4   14.44m ±  2%    14.47m ±  3%         ~ (p=0.699 n=6)
geomean                          190.9µ          204.2µ          +6.99%

                               │ /tmp/old.txt  │            /tmp/new.txt             │
                               │    rows/s     │    rows/s     vs base               │
Pipe/filter/n=100-4               6.973M ±  1%   6.795M ±  3%   -2.54% (p=0.026 n=6)
Pipe/filter/n=1000-4              6.868M ± 20%   6.828M ±  2%        ~ (p=0.818 n=6)
Pipe/filter/n=10000-4             6.826M ±  1%   6.630M ±  3%   -2.87% (p=0.009 n=6)
Pipe/groupBy/n=100-4              53.36G ±  0%   52.53G ±  2%        ~ (p=0.093 n=6)
Pipe/groupBy/n=1000-4             534.6G ±  0%   527.4G ±  3%   -1.35% (p=0.004 n=6)
Pipe/groupBy/n=10000-4            5.284T ±  5%   5.342T ±  2%        ~ (p=0.310 n=6)
Pipe/aggregate/n=100-4            4.803M ±  1%   4.764M ±  3%        ~ (p=0.310 n=6)
Pipe/aggregate/n=1000-4           7.508M ±  1%   7.349M ±  2%   -2.11% (p=0.002 n=6)
Pipe/aggregate/n=10000-4          8.172M ±  2%   8.091M ±  2%   -0.99% (p=0.041 n=6)
Pipe/sort/n=100-4                 18.06M ±  4%   15.16M ±  2%  -16.03% (p=0.002 n=6)
Pipe/sort/n=1000-4                17.66M ±  2%   14.71M ±  1%  -16.71% (p=0.002 n=6)
Pipe/sort/n=10000-4               12.71M ±  2%   11.40M ±  1%  -10.24% (p=0.002 n=6)
Pipe/distinct/n=100-4             9.043M ±  2%   9.038M ±  3%        ~ (p=0.937 n=6)
Pipe/distinct/n=1000-4            8.588M ±  1%   8.430M ±  4%   -1.84% (p=0.002 n=6)
Pipe/distinct/n=10000-4           8.529M ±  3%   8.418M ± 17%        ~ (p=0.132 n=6)
Pipe/dedupe/n=100-4               9.168M ±  2%   9.206M ±  1%        ~ (p=0.589 n=6)
Pipe/dedupe/n=1000-4              8.671M ±  1%   8.654M ±  1%        ~ (p=0.589 n=6)
Pipe/dedupe/n=10000-4             8.614M ±  3%   8.565M ±  1%        ~ (p=0.394 n=6)
Pipe/window/n=100-4               3.200M ±  2%   3.237M ±  1%        ~ (p=0.240 n=6)
Pipe/window/n=1000-4              1.722M ±  2%   1.734M ±  4%        ~ (p=0.180 n=6)
Pipe/window/n=10000-4            1088.2k ±  8%   988.6k ±  8%   -9.14% (p=0.015 n=6)
Pipe/topPerGroup/n=100-4          3.608M ±  2%   3.626M ±  1%        ~ (p=0.180 n=6)
Pipe/topPerGroup/n=1000-4         1.986M ±  2%   2.020M ±  1%   +1.74% (p=0.002 n=6)
Pipe/topPerGroup/n=10000-4        1.278M ±  1%   1.304M ±  2%   +2.03% (p=0.004 n=6)
Pipe/histogram/n=100-4           18.027M ±  1%   8.290M ±  1%  -54.01% (p=0.002 n=6)
Pipe/histogram/n=1000-4          25.381M ±  1%   9.288M ±  1%  -63.41% (p=0.002 n=6)
Pipe/histogram/n=10000-4         25.942M ±  1%   9.012M ±  3%  -65.26% (p=0.002 n=6)
Pipe/pivot/n=100-4                1.584M ±  2%   1.611M ±  2%   +1.68% (p=0.041 n=6)
Pipe/pivot/n=1000-4               1.919M ±  2%   1.977M ±  2%   +3.03% (p=0.015 n=6)
Pipe/pivot/n=10000-4              2.064M ±  1%   2.045M ±  2%   -0.91% (p=0.004 n=6)
Pipe/unpivot/n=100-4              4.594M ±  2%   4.594M ±  2%        ~ (p=0.937 n=6)
Pipe/unpivot/n=1000-4             3.551M ±  4%   3.616M ±  1%   +1.83% (p=0.004 n=6)
Pipe/unpivot/n=10000-4            3.821M ±  2%   3.827M ±  3%        ~ (p=0.818 n=6)
Pipe/gapfill/n=100-4              8.431M ±  1%   8.307M ±  1%   -1.47% (p=0.004 n=6)
Pipe/gapfill/n=1000-4             8.151M ±  5%   8.094M ±  0%        ~ (p=0.093 n=6)
Pipe/gapfill/n=10000-4            7.387M ±  2%   7.293M ±  1%   -1.27% (p=0.015 n=6)
PipeJoins/lookup/n=100-4          383.2k ±  2%   377.6k ±  2%        ~ (p=0.065 n=6)
PipeJoins/lookup/n=1000-4         576.3k ±  1%   571.0k ±  1%        ~ (p=0.240 n=6)
PipeJoins/lookup/n=10000-4        671.4k ±  2%   658.8k ±  2%   -1.88% (p=0.041 n=6)
PipeJoins/asofJoin/n=100-4        375.1k ±  1%   388.4k ±  1%   +3.55% (p=0.002 n=6)
PipeJoins/asofJoin/n=1000-4       531.1k ±  2%   530.1k ±  1%        ~ (p=0.240 n=6)
PipeJoins/asofJoin/n=10000-4      628.2k ±  3%   612.0k ± 13%        ~ (p=0.093 n=6)
PipeJoins/crossJoin/n=100-4       78.28k ±  3%   78.23k ±  2%        ~ (p=0.937 n=6)
PipeJoins/crossJoin/n=1000-4      68.28k ±  2%   67.99k ±  2%        ~ (p=0.394 n=6)
PipeJoins/crossJoin/n=10000-4     66.09k ±  2%   68.35k ±  6%        ~ (p=0.180 n=6)
PipeSetOps/except/n=100-4         886.2k ±  2%   874.1k ±  1%   -1.36% (p=0.009 n=6)
PipeSetOps/except/n=1000-4        796.4k ±  2%   793.8k ±  2%        ~ (p=0.937 n=6)
PipeSetOps/except/n=10000-4       771.2k ±  2%   773.4k ±  1%        ~ (p=0.818 n=6)
PipeSetOps/intersect/n=100-4      795.1k ±  1%   810.9k ±  1%   +1.98% (p=0.002 n=6)
PipeSetOps/intersect/n=1000-4     712.4k ±  2%   735.5k ±  2%   +3.25% (p=0.002 n=6)
PipeSetOps/intersect/n=10000-4    692.7k ±  2%   691.3k ±  3%        ~ (p=0.699 n=6)
geomean                           5.240M         4.898M         -6.53%

                               │  /tmp/old.txt   │             /tmp/new.txt              │
                               │      B/op       │     B/op       vs base                │
Pipe/filter/n=100-4              2.125Ki ±  0%     2.125Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/filter/n=1000-4             20.50Ki ±  0%     20.50Ki ±  0%       ~ (p=0.242 n=6)
Pipe/filter/n=10000-4            205.0Ki ±  0%     205.0Ki ±  0%       ~ (p=0.978 n=6)
Pipe/groupBy/n=100-4               0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/groupBy/n=1000-4              0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/groupBy/n=10000-4             0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/aggregate/n=100-4           12.92Ki ±  0%     12.92Ki ±  0%       ~ (p=1.000 n=6)
Pipe/aggregate/n=1000-4          37.46Ki ±  0%     37.46Ki ±  0%       ~ (p=0.545 n=6)
Pipe/aggregate/n=10000-4         270.4Ki ±  0%     270.4Ki ±  0%       ~ (p=0.478 n=6)
Pipe/sort/n=100-4                  56.00 ±  0%       56.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/sort/n=1000-4                 56.00 ±  0%       56.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/sort/n=10000-4                56.00 ±  0%       56.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/distinct/n=100-4            5.070Ki ±  0%     5.070Ki ±  0%       ~ (p=1.000 n=6)
Pipe/distinct/n=1000-4           69.16Ki ±  0%     69.16Ki ±  0%       ~ (p=0.346 n=6)
Pipe/distinct/n=10000-4          584.8Ki ±  0%     584.8Ki ±  0%       ~ (p=0.368 n=6)
Pipe/dedupe/n=100-4              5.070Ki ±  0%     5.071Ki ±  0%       ~ (p=0.545 n=6)
Pipe/dedupe/n=1000-4             69.16Ki ±  0%     69.16Ki ±  0%       ~ (p=0.693 n=6)
Pipe/dedupe/n=10000-4            584.8Ki ±  0%     584.8Ki ±  0%       ~ (p=0.859 n=6)
Pipe/window/n=100-4              10.79Ki ±  0%     10.79Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/window/n=1000-4             63.84Ki ±  0%     63.84Ki ±  0%       ~ (p=0.589 n=6)
Pipe/window/n=10000-4            600.5Ki ± 77%     600.5Ki ± 72%       ~ (p=1.000 n=6)
Pipe/topPerGroup/n=100-4         8.165Ki ±  0%     8.165Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/topPerGroup/n=1000-4        39.83Ki ±  0%     39.83Ki ±  0%       ~ (p=0.766 n=6)
Pipe/topPerGroup/n=10000-4       344.7Ki ±  0%     344.7Ki ±  0%       ~ (p=0.307 n=6)
Pipe/histogram/n=100-4           3.594Ki ±  0%     3.594Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/histogram/n=1000-4          3.594Ki ±  0%     3.594Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/histogram/n=10000-4         3.672Ki ±  0%     3.672Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/pivot/n=100-4               56.55Ki ±  0%     56.55Ki ±  0%       ~ (p=1.000 n=6)
Pipe/pivot/n=1000-4              403.5Ki ±  0%     403.5Ki ±  0%       ~ (p=0.608 n=6)
Pipe/pivot/n=10000-4             3.733Mi ±  0%     3.733Mi ±  0%       ~ (p=0.970 n=6)
Pipe/unpivot/n=100-4             35.25Ki ±  0%     35.25Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/unpivot/n=1000-4            351.8Ki ±  0%     351.8Ki ±  0%       ~ (p=1.000 n=6)
Pipe/unpivot/n=10000-4           3.435Mi ±  0%     3.435Mi ±  0%       ~ (p=1.000 n=6)
Pipe/gapfill/n=100-4             2.922Ki ±  0%     2.922Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/gapfill/n=1000-4            32.67Ki ±  0%     32.67Ki ±  0%       ~ (p=1.000 n=6) ¹
Pipe/gapfill/n=10000-4           429.3Ki ±  0%     429.3Ki ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/lookup/n=100-4         147.0Ki ±  0%     147.0Ki ±  0%       ~ (p=0.626 n=6)
PipeJoins/lookup/n=1000-4        913.8Ki ±  0%     913.7Ki ±  0%       ~ (p=0.121 n=6)
PipeJoins/lookup/n=10000-4       8.379Mi ±  0%     8.379Mi ±  0%       ~ (p=0.485 n=6)
PipeJoins/asofJoin/n=100-4       103.7Ki ±  0%     103.7Ki ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/asofJoin/n=1000-4      877.2Ki ±  0%     877.2Ki ±  0%       ~ (p=1.000 n=6)
PipeJoins/asofJoin/n=10000-4     8.411Mi ±  0%     8.411Mi ±  0%       ~ (p=0.424 n=6)
PipeJoins/crossJoin/n=100-4      833.2Ki ±  0%     833.2Ki ±  0%       ~ (p=0.193 n=6)
PipeJoins/crossJoin/n=1000-4     8.132Mi ±  0%     8.132Mi ±  0%       ~ (p=0.589 n=6)
PipeJoins/crossJoin/n=10000-4    81.31Mi ±  0%     81.31Mi ±  0%       ~ (p=0.937 n=6)
PipeSetOps/except/n=100-4        77.62Ki ±  0%     77.62Ki ±  0%       ~ (p=0.513 n=6)
PipeSetOps/except/n=1000-4       765.3Ki ±  0%     765.3Ki ±  0%       ~ (p=0.175 n=6)
PipeSetOps/except/n=10000-4      7.628Mi ±  0%     7.628Mi ±  0%       ~ (p=0.472 n=6)
PipeSetOps/intersect/n=100-4     81.80Ki ±  0%     81.80Ki ±  0%       ~ (p=0.729 n=6)
PipeSetOps/intersect/n=1000-4    798.3Ki ±  0%     798.4Ki ±  0%       ~ (p=0.097 n=6)
PipeSetOps/intersect/n=10000-4   8.106Mi ±  0%     8.106Mi ±  0%       ~ (p=0.898 n=6)
geomean                                        ²                  +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                               │  /tmp/old.txt  │             /tmp/new.txt             │
                               │   allocs/op    │  allocs/op    vs base                │
Pipe/filter/n=100-4               201.0 ±  0%      201.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/filter/n=1000-4             2.001k ±  0%     2.001k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/filter/n=10000-4            20.00k ±  0%     20.00k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/groupBy/n=100-4              0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/groupBy/n=1000-4             0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/groupBy/n=10000-4            0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/aggregate/n=100-4            230.0 ±  0%      230.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/aggregate/n=1000-4          1.190k ±  0%     1.190k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/aggregate/n=10000-4         10.27k ±  0%     10.27k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/sort/n=100-4                 2.000 ±  0%      2.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/sort/n=1000-4                2.000 ±  0%      2.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/sort/n=10000-4               2.000 ±  0%      2.000 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/distinct/n=100-4             104.0 ±  0%      104.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/distinct/n=1000-4           1.006k ±  0%     1.006k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/distinct/n=10000-4          10.03k ±  0%     10.03k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/dedupe/n=100-4               104.0 ±  0%      104.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/dedupe/n=1000-4             1.006k ±  0%     1.006k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/dedupe/n=10000-4            10.03k ±  0%     10.03k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/window/n=100-4               231.0 ±  0%      231.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/window/n=1000-4             1.191k ±  0%     1.191k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/window/n=10000-4            15.15k ± 15%     15.15k ± 14%       ~ (p=1.000 n=6)
Pipe/topPerGroup/n=100-4          230.0 ±  0%      230.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/topPerGroup/n=1000-4        1.190k ±  0%     1.190k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/topPerGroup/n=10000-4       10.25k ±  0%     10.25k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/histogram/n=100-4            42.00 ±  0%      42.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/histogram/n=1000-4           42.00 ±  0%      42.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/histogram/n=10000-4          52.00 ±  0%      52.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/pivot/n=100-4                612.0 ±  0%      612.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/pivot/n=1000-4              4.272k ±  0%     4.272k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/pivot/n=10000-4             40.33k ±  0%     40.33k ±  0%       ~ (p=0.545 n=6)
Pipe/unpivot/n=100-4              301.0 ±  0%      301.0 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/unpivot/n=1000-4            3.001k ±  0%     3.001k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/unpivot/n=10000-4           30.00k ±  0%     30.00k ±  0%       ~ (p=1.000 n=6) ¹
Pipe/gapfill/n=100-4              11.00 ±  0%      11.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/gapfill/n=1000-4             15.00 ±  0%      15.00 ±  0%       ~ (p=1.000 n=6) ¹
Pipe/gapfill/n=10000-4            22.00 ±  0%      22.00 ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/lookup/n=100-4         2.310k ±  0%     2.310k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/lookup/n=1000-4        14.01k ±  0%     14.01k ±  0%       ~ (p=0.455 n=6)
PipeJoins/lookup/n=10000-4       131.0k ±  0%     131.0k ±  0%       ~ (p=0.455 n=6)
PipeJoins/asofJoin/n=100-4       1.217k ±  0%     1.217k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/asofJoin/n=1000-4      12.02k ±  0%     12.02k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/asofJoin/n=10000-4     120.0k ±  0%     120.0k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/crossJoin/n=100-4      12.00k ±  0%     12.00k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/crossJoin/n=1000-4     120.0k ±  0%     120.0k ±  0%       ~ (p=1.000 n=6) ¹
PipeJoins/crossJoin/n=10000-4    1.200M ±  0%     1.200M ±  0%       ~ (p=1.000 n=6)
PipeSetOps/except/n=100-4         968.0 ±  0%      968.0 ±  0%       ~ (p=1.000 n=6) ¹
PipeSetOps/except/n=1000-4       9.609k ±  0%     9.609k ±  0%       ~ (p=1.000 n=6) ¹
PipeSetOps/except/n=10000-4      96.04k ±  0%     96.04k ±  0%       ~ (p=1.000 n=6)
PipeSetOps/intersect/n=100-4     1.054k ±  0%     1.054k ±  0%       ~ (p=1.000 n=6) ¹
PipeSetOps/intersect/n=1000-4    10.41k ±  0%     10.41k ±  0%       ~ (p=1.000 n=6) ¹
PipeSetOps/intersect/n=10000-4   104.1k ±  0%     104.1k ±  0%       ~ (p=1.000 n=6)
geomean                                       ²                 +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/xraph/dql/planner
                │ /tmp/old.txt │           /tmp/new.txt            │
                │    sec/op    │   sec/op     vs base              │
Plan/pushdown-4    843.7n ± 2%   847.4n ± 2%       ~ (p=0.240 n=6)
Plan/inmemory-4    770.7n ± 2%   770.7n ± 1%       ~ (p=0.699 n=6)
Plan/groupBy-4     719.9n ± 1%   724.7n ± 2%       ~ (p=0.132 n=6)
geomean            776.5n        779.3n       +0.37%

                │ /tmp/old.txt │             /tmp/new.txt             │
                │     B/op     │     B/op      vs base                │
Plan/pushdown-4   1.594Ki ± 0%   1.594Ki ± 0%       ~ (p=1.000 n=6) ¹
Plan/inmemory-4   1.562Ki ± 0%   1.562Ki ± 0%       ~ (p=1.000 n=6) ¹
Plan/groupBy-4    1.016Ki ± 0%   1.016Ki ± 0%       ~ (p=1.000 n=6) ¹
geomean           1.362Ki        1.362Ki       +0.00%
¹ all samples are equal

                │ /tmp/old.txt │            /tmp/new.txt            │
                │  allocs/op   │ allocs/op   vs base                │
Plan/pushdown-4     9.000 ± 0%   9.000 ± 0%       ~ (p=1.000 n=6) ¹
Plan/inmemory-4     9.000 ± 0%   9.000 ± 0%       ~ (p=1.000 n=6) ¹
Plan/groupBy-4      8.000 ± 0%   8.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean             8.653        8.653       +0.00%
¹ all samples are equal

pkg: github.com/xraph/dql/processor
                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │    sec/op    │    sec/op     vs base              │
Process/passthrough/n=100-4     81.65n ±  3%   78.20n ±  1%  -4.21% (p=0.002 n=6)
Process/passthrough/n=1000-4    97.74n ±  1%   95.98n ±  2%  -1.80% (p=0.026 n=6)
Process/passthrough/n=10000-4   94.33n ±  2%   94.18n ±  2%       ~ (p=0.699 n=6)
Process/aggregate/n=100-4       22.05µ ±  1%   22.12µ ±  5%       ~ (p=0.699 n=6)
Process/aggregate/n=1000-4      153.2µ ± 11%   155.5µ ± 22%  +1.49% (p=0.028 n=6)
Process/aggregate/n=10000-4     1.461m ±  3%   1.462m ±  4%       ~ (p=1.000 n=6)
Process/sort/n=100-4            5.166µ ±  1%   5.169µ ±  0%       ~ (p=0.937 n=6)
Process/sort/n=1000-4           52.54µ ±  7%   53.54µ ±  1%       ~ (p=0.065 n=6)
Process/sort/n=10000-4          743.8µ ±  2%   752.2µ ±  0%       ~ (p=0.240 n=6)
Process/computed/n=100-4        3.696µ ±  2%   3.624µ ±  1%  -1.96% (p=0.002 n=6)
Process/computed/n=1000-4       40.81µ ±  1%   40.98µ ±  0%  +0.41% (p=0.041 n=6)
Process/computed/n=10000-4      405.9µ ±  1%   409.3µ ±  0%  +0.84% (p=0.002 n=6)
geomean                         13.76µ         13.73µ        -0.17%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │    rows/s    │    rows/s     vs base              │
Process/passthrough/n=100-4     1.225G ±  3%   1.279G ±  1%  +4.39% (p=0.002 n=6)
Process/passthrough/n=1000-4    10.23G ±  1%   10.42G ±  2%  +1.83% (p=0.026 n=6)
Process/passthrough/n=10000-4   106.0G ±  2%   106.2G ±  2%       ~ (p=0.699 n=6)
Process/aggregate/n=100-4       4.536M ±  1%   4.521M ±  4%       ~ (p=0.699 n=6)
Process/aggregate/n=1000-4      6.527M ± 10%   6.431M ± 18%  -1.47% (p=0.026 n=6)
Process/aggregate/n=10000-4     6.844M ±  2%   6.840M ±  4%       ~ (p=1.000 n=6)
Process/sort/n=100-4            19.36M ±  1%   19.35M ±  0%       ~ (p=0.937 n=6)
Process/sort/n=1000-4           19.03M ±  6%   18.68M ±  1%       ~ (p=0.065 n=6)
Process/sort/n=10000-4          13.45M ±  2%   13.29M ±  0%       ~ (p=0.240 n=6)
Process/computed/n=100-4        27.06M ±  2%   27.60M ±  1%  +1.99% (p=0.002 n=6)
Process/computed/n=1000-4       24.50M ±  1%   24.40M ±  0%  -0.41% (p=0.041 n=6)
Process/computed/n=10000-4      24.64M ±  1%   24.43M ±  0%  -0.83% (p=0.002 n=6)
geomean                         72.70M         72.82M        +0.17%

                              │ /tmp/old.txt │             /tmp/new.txt             │
                              │     B/op     │     B/op      vs base                │
Process/passthrough/n=100-4       168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/passthrough/n=1000-4      168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/passthrough/n=10000-4     168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/aggregate/n=100-4       13.24Ki ± 0%   13.24Ki ± 0%       ~ (p=1.000 n=6) ¹
Process/aggregate/n=1000-4      37.78Ki ± 0%   37.77Ki ± 0%       ~ (p=0.892 n=6)
Process/aggregate/n=10000-4     270.6Ki ± 0%   270.6Ki ± 0%       ~ (p=0.675 n=6)
Process/sort/n=100-4              224.0 ± 0%     224.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/sort/n=1000-4             224.0 ± 0%     224.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/sort/n=10000-4            224.0 ± 0%     224.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=100-4          168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=1000-4         168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=10000-4        168.0 ± 0%     168.0 ± 0%       ~ (p=1.000 n=6) ¹
geomean                           759.3          759.3       -0.00%
¹ all samples are equal

                              │ /tmp/old.txt │            /tmp/new.txt             │
                              │  allocs/op   │  allocs/op   vs base                │
Process/passthrough/n=100-4       2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/passthrough/n=1000-4      2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/passthrough/n=10000-4     2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/aggregate/n=100-4         252.0 ± 0%    252.0 ± 0%       ~ (p=1.000 n=6) ¹
Process/aggregate/n=1000-4       1.212k ± 0%   1.212k ± 0%       ~ (p=1.000 n=6) ¹
Process/aggregate/n=10000-4      10.27k ± 0%   10.27k ± 0%       ~ (p=1.000 n=6) ¹
Process/sort/n=100-4              4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/sort/n=1000-4             4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/sort/n=10000-4            4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=100-4          2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=1000-4         2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
Process/computed/n=10000-4        2.000 ± 0%    2.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                           12.37         12.37       +0.00%
¹ all samples are equal

pkg: github.com/xraph/dql/sqlgen
                       │ /tmp/old.txt │           /tmp/new.txt            │
                       │    sec/op    │   sec/op     vs base              │
GenerateSQL/simple-4      715.7n ± 3%   721.1n ± 1%       ~ (p=0.485 n=6)
GenerateSQL/compound-4    1.020µ ± 1%   1.026µ ± 1%       ~ (p=0.084 n=6)
geomean                   854.4n        859.9n       +0.65%

                       │ /tmp/old.txt │            /tmp/new.txt            │
                       │     B/op     │    B/op     vs base                │
GenerateSQL/simple-4       520.0 ± 0%   520.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSQL/compound-4     536.0 ± 0%   536.0 ± 0%       ~ (p=1.000 n=6) ¹
geomean                    527.9        527.9       +0.00%
¹ all samples are equal

                       │ /tmp/old.txt │            /tmp/new.txt            │
                       │  allocs/op   │ allocs/op   vs base                │
GenerateSQL/simple-4       13.00 ± 0%   13.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSQL/compound-4     16.00 ± 0%   16.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                    14.42        14.42       +0.00%
¹ all samples are equal

juicycleff added a commit that referenced this pull request Aug 3, 2026
@juicycleff
ci: raise the bench gate threshold to 30%
697b587 
Verification on pull request #1 flagged Pipe/sort/n=1000 at +20.06% on a change
that never touched sort. Same-runner comparison holds most benchmarks to
+/-1-3%, but the tail reaches 20, so a 20% gate cries wolf.

The deliberate regression in that same run measured +117% to +188%, so 30 costs
no detection power. Also records the measured gate runtime of 2m48s, well under
the estimate — -benchtime bounds each benchmark by wall clock, so total time
tracks benchmark count rather than machine speed.
@juicycleff

juicycleff commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Verification complete — the gate works as intended.

It correctly failed on the deliberate regression (+117% to +188%, p=0.002) and the whole run took 2m48s.

It also surfaced one false positive: Pipe/sort/n=1000 at +20.06% on a change that never touched sort. Threshold raised 20% → 30% in 697b587 as a result, and the findings are recorded in the design spec.

Closing without merging — this branch only ever existed to make the gate fire.

@juicycleff juicycleff closed this Aug 3, 2026
@juicycleff
juicycleff deleted the test/bench-gate-verification branch August 3, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@juicycleff