use octet_length() instead of len() in clickbench#8267
Merged
Conversation
myrrc
force-pushed
the
myrrc/clickbench-strlen
branch
from
156ac17 to
7a376a7
Compare
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | compare[63] |
245.2 µs | 360.9 µs | -32.06% |
| ❌ | Simulation | compare[56] |
230.1 µs | 332.7 µs | -30.83% |
| ❌ | Simulation | compare[62] |
255.3 µs | 369.1 µs | -30.83% |
| ❌ | Simulation | compare[60] |
248.8 µs | 358.8 µs | -30.66% |
| ❌ | Simulation | compare[61] |
256 µs | 368 µs | -30.41% |
| ❌ | Simulation | compare[58] |
246.1 µs | 352.4 µs | -30.16% |
| ❌ | Simulation | compare[59] |
251.5 µs | 359.8 µs | -30.08% |
| ❌ | Simulation | compare[57] |
246.7 µs | 351.1 µs | -29.75% |
| ❌ | Simulation | compare[54] |
236.5 µs | 335.4 µs | -29.49% |
| ❌ | Simulation | compare[55] |
242.2 µs | 343 µs | -29.38% |
| ❌ | Simulation | compare[52] |
230.5 µs | 325.7 µs | -29.23% |
| ❌ | Simulation | compare[48] |
212.9 µs | 300.6 µs | -29.18% |
| ❌ | Simulation | compare[53] |
236.6 µs | 333.6 µs | -29.08% |
| ❌ | Simulation | compare[50] |
227.7 µs | 319.2 µs | -28.65% |
| ❌ | Simulation | compare[51] |
232.7 µs | 326 µs | -28.63% |
| ❌ | Simulation | compare[49] |
227.9 µs | 317.5 µs | -28.22% |
| ❌ | Simulation | compare[47] |
223.3 µs | 309.1 µs | -27.78% |
| ❌ | Simulation | compare[46] |
218.4 µs | 302.4 µs | -27.78% |
| ❌ | Simulation | compare[44] |
212.2 µs | 292.4 µs | -27.44% |
| ❌ | Simulation | compare[45] |
218.9 µs | 301 µs | -27.29% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/clickbench-strlen (20f6f2e) with develop (f127357)
myrrc
force-pushed
the
myrrc/clickbench-strlen
branch
from
7a376a7 to
08efe51
Compare
github-actions
Bot
removed
the
action/benchmark
joseph-isaacs
approved these changes
Jun 5, 2026
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (2)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals ProfilingVortex (geomean): 1.002x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.002x ➖, 1↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.047x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.036x ➖, 0↑ 0↓)
datafusion / parquet (1.051x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.044x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.055x ➖, 0↑ 1↓)
duckdb / parquet (1.066x ➖, 0↑ 1↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.005x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.002x ➖, 0↑ 0↓)
datafusion / parquet (0.983x ➖, 2↑ 1↓)
datafusion / arrow (1.011x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.008x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.003x ➖, 0↑ 0↓)
duckdb / parquet (0.998x ➖, 1↑ 2↓)
duckdb / duckdb (1.003x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -0.0% overall, 4↑ 5↓)
Totals:
|
Contributor
Benchmarks: Clickbench on NVMEVerdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.973x ➖, 2↑ 2↓)
datafusion / parquet (0.983x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.994x ➖, 1↑ 3↓)
duckdb / parquet (0.986x ➖, 1↑ 0↓)
duckdb / duckdb (1.002x ➖, 1↑ 1↓)
File Size Changes (106 files changed, -0.0% overall, 51↑ 55↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.990x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.989x ➖, 3↑ 1↓)
datafusion / parquet (0.989x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed (0.995x ➖, 1↑ 2↓)
duckdb / vortex-compact (0.998x ➖, 1↑ 2↓)
duckdb / parquet (1.006x ➖, 0↑ 3↓)
duckdb / duckdb (0.994x ➖, 1↑ 1↓)
File Size Changes (6 files changed, -0.0% overall, 1↑ 5↓)
Totals:
|
myrrc
force-pushed
the
myrrc/clickbench-strlen
branch
from
08efe51 to
8bd76d7
Compare
myrrc
changed the title
Contributor
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.116x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.116x ➖, 0↑ 2↓)
datafusion / parquet (1.211x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.143x ➖, 0↑ 2↓)
duckdb / vortex-compact (1.075x ➖, 0↑ 0↓)
duckdb / parquet (1.082x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.007x ➖, 0↑ 0↓)
duckdb / parquet (1.002x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.129x ❌, 0↑ 7↓)
datafusion / vortex-compact (1.016x ➖, 0↑ 0↓)
datafusion / parquet (1.014x ➖, 0↑ 0↓)
datafusion / arrow (1.027x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed (1.013x ➖, 0↑ 2↓)
duckdb / vortex-compact (1.005x ➖, 0↑ 0↓)
duckdb / parquet (1.027x ➖, 0↑ 2↓)
duckdb / duckdb (1.010x ➖, 0↑ 0↓)
File Size Changes (27 files changed, -0.0% overall, 11↑ 16↓)
Totals:
|
Contributor
Benchmarks: Appian on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.003x ➖, 0↑ 0↓)
datafusion / parquet (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.012x ➖, 0↑ 0↓)
duckdb / parquet (1.006x ➖, 0↑ 0↓)
duckdb / duckdb (1.006x ➖, 0↑ 0↓)
File Size Changes (4 files changed, -0.0% overall, 0↑ 4↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.265x ➖, 0↑ 9↓)
datafusion / vortex-compact (1.184x ➖, 0↑ 7↓)
datafusion / parquet (1.192x ➖, 0↑ 6↓)
duckdb / vortex-file-compressed (1.097x ➖, 0↑ 2↓)
duckdb / vortex-compact (1.056x ➖, 0↑ 0↓)
duckdb / parquet (1.095x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Random AccessVortex (geomean): 0.890x ✅ How to read Verdict and Engines
unknown / unknown (0.972x ➖, 13↑ 1↓)
|
Contributor
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.100x ➖, 1↑ 5↓)
datafusion / vortex-compact (1.180x ➖, 0↑ 5↓)
datafusion / parquet (1.065x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.062x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.093x ➖, 0↑ 2↓)
duckdb / parquet (1.044x ➖, 0↑ 0↓)
|
myrrc
force-pushed
the
myrrc/clickbench-strlen
branch
from
8bd76d7 to
d6e08fc
Compare
Comment on lines
+180
to
+184
| ctx.register_tables(&*benchmark, format)?; | ||
|
|
||
| // Duckdb doesn't support octet_length for strings but we need this | ||
| // in ClickBench. | ||
| ctx.execute_query_result("create macro octet_length(a) as strlen(a)")?; |
Contributor
There was a problem hiding this comment.
we really should move this into a benchmark specific setup.sql function. I think that other benchmarks might be affected
Contributor
Author
There was a problem hiding this comment.
No, because duckdb doesn't support octet length so there were no queries with it.
Contributor
There was a problem hiding this comment.
this is a bug waiting to happen
github-actions
Bot
removed
the
action/benchmark
Contributor
Benchmarks: CompressionVortex (geomean): 1.003x ➖ How to read Verdict and Engines
unknown / unknown (1.007x ➖, 0↑ 2↓)
|
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
myrrc
force-pushed
the
myrrc/clickbench-strlen
branch
from
d6e08fc to
20f6f2e
Compare
github-actions
Bot
removed
the
action/benchmark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clickhouse's length() is non-standard SQL which returns number of bytes and not
Unicode code points. Replace this for ClickBench to Duckdb's octet_length() to match
original intent. As Duckdb doesn't support octet_length for strings and uses non-standard strlen(), add a macro to convert octet_length to strlen().
We can't use strlen() directly because DataFusion's strlen() counts code points, not bytes.