Add selectable io_uring buffer backends - #3
Open
whzruc wants to merge 1 commit into
Open
Conversation
whzruc
force-pushed
the
rebuild/cpp-io-uring-multi-backend
branch
from
September 2, 2026 10:01
9235824 to
c50bb08
Compare
whzruc
force-pushed
the
rebuild/cpp-io-uring-multi-backend
branch
from
September 2, 2026 10:05
c50bb08 to
cccc4fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pixel.bufferpool.modeStack
This PR is based on
rebuild/cpp-dynamic-buffer-pooland should be reviewed after PR #2.Performance conclusions
The committed reports summarize the results in English and Chinese. Raw outputs and per-run measurements are intentionally excluded from Git.
ClickBench q24 end-to-end
The test used 24 NVMe SSDs, 12/24/48 threads, one warm-up and seven measured runs per backend. All 63 measured runs returned identical results without errors.
The largest difference was 2.75%, below the predefined 5% practical-significance threshold. q24 therefore shows no material end-to-end regression from the selectable backends, but its filtering, sorting, and materialization dilute differences in the I/O submission path.
Raw I/O scan
The raw scanner bypasses DuckDB, PixelsReader, decoding, and materialization. Each measured run scanned 15,360 files and 1,916,931,234,624 bytes with 1,048,576-byte requests and queue depth 32 per thread.
Registered fixed buffers were about 4% faster at 12 threads, before platform saturation. At 24 and 48 threads, all backends converged within 0.5% at an aggregate ceiling of roughly 110 GB/s.
With 24 threads and 24 devices, forcing one thread per device produced 108.79–109.15 GB/s and was 0.38%–0.83% slower than the shared interleaved queue. The shared queue is not a material bottleneck and slightly improves load balancing.
A single SSD reached 6.14 GB/s, while all 24 averaged about 4.55 GB/s. The 1/4/8/12/24-device scaling curve and the four PCIe root groups indicate that the aggregate ceiling is in the shared PCIe/CPU I/O, DMA, or memory path rather than in buffer registration or file scheduling.
Interpretation
Validation
pixels_extension,PixelsIoUringScanBenchmark,DirectUringNonFixedTest,DynamicBufferPoolTest, andGlobalStaticBufferPoolTestgit diff --checkpass