duckdb: propagate per-file "partition" stats - #9639
Conversation
Merging this PR will degrade performance by 7.16%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.5 µs | 20.4 µs | -33.48% |
| ❌ | WallTime | arrow_checked_add_u32_avx2[16384] |
17.7 µs | 21.3 µs | -16.8% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.6 µs | +20.65% |
| ⚡ | WallTime | mul_u32_nonnull_avx512 |
6.3 µs | 5.7 µs | +11.25% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-partition-stats (ca83a4e) with develop (50cd6d7)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Pull request was converted to draft
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.010x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.978x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.027x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.978x ➖, 1↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 52↑ 48↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 3↑ 3↓)
duckdb / parquet / ns (0.984x ➖, 8↑ 2↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.134x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.913x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.694x ✅, 5↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.994x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.909x ➖, 2↑ 2↓)
duckdb / parquet / ns (0.978x ➖, 2↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.937x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.957x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.961x ➖, 0↑ 0↓)
|
|
Please explain this code with comments |
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
8b3244b to
270427a
Compare
|
@joseph-isaacs Done |
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
8ba4fd3 to
d03363e
Compare
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
This couldn't be done previously since we didn't have ordering of files
for threads. Since migration to MultiFileReader, we now have well-defined
batch indices.
Propagate per-file partition stats to duckdb. MultiFileSession already caches
footers by file path, so we can use this information. This change makes
Clickbench Q0 and Q6 constant (supplied from metadata) from second iteration.
Use crate's REGISTRY instead of own one.
Resolves: #3393