feat(parquet): batch RLE runs in level encoder via scan-ahead#9830
Merged
alamb merged 2 commits intoapache:mainfrom Apr 28, 2026
Merged
feat(parquet): batch RLE runs in level encoder via scan-ahead#9830alamb merged 2 commits intoapache:mainfrom
alamb merged 2 commits intoapache:mainfrom
Conversation
Add `is_accumulating_rle()` and `extend_run()` methods to `RleEncoder` that allow callers to detect when the encoder is in RLE accumulation mode and bulk-extend runs without per-element overhead. Upgrade `put_with_observer()` in `LevelEncoder` to exploit this: after each `put()`, it checks whether the encoder entered accumulation mode. If so, it scans ahead for the rest of the run, calls `extend_run()` to batch it in O(1), and fires the observer once with the full run length. This turns the previous O(n) per-value encoding + observation into O(1) amortized per RLE run, which is a significant improvement for sparse columns where long runs of identical levels are common. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>
Contributor
Author
|
@alamb this one is short and sweet, and should yield some nice perf wins |
Contributor
|
run benchmarks arrow_reader arrow_reader_clickbench |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing batch_rle_run (415b496) to 4fa8d2f (merge-base) diff File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing batch_rle_run (415b496) to 4fa8d2f (merge-base) diff File an issue against this benchmark runner |
alamb
approved these changes
Apr 27, 2026
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Looks good to me -- thank you @HippoBaro
FYI @etseidl and @jhorstmann
| bit_packed_max_size.max(rle_max_size) | ||
| } | ||
|
|
||
| /// Returns `true` if the encoder is currently in RLE accumulation mode |
Contributor
There was a problem hiding this comment.
I am always a bit confused about what part of the parquet module is (actually) part of its public API. For anyone else that is also confused:
RleEncoder is only reachable through the encodings module, and parquet/src/lib.rs:144 makes that module public only when the experimental feature is enabled. In the default build it is private, and even when exposed it is explicitly marked as having “no stability guarantees.” The docs.rs crate page for parquet 58.0.0 also omits enc odings from the normal public API surface, which is consistent with that setup: https://docs.rs/parquet/58.0.0/parquet/ and the feature list notes experimental as unstable: https://docs.rs/crate/parquet/latest
Contributor
Author
There was a problem hiding this comment.
The difference between public and public-experimental always trips me up!
Contributor
|
I'll wait for benchmark confirmation before merging this one |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
etseidl
approved these changes
Apr 27, 2026
Contributor
etseidl
left a comment
etseidl
left a comment
There was a problem hiding this comment.
Looks like a win to me. 😄
etseidl
reviewed
Apr 27, 2026
Contributor
Author
|
@alamb you ran the benchmarks for the reader path, but this code relates to the write path 😅 |
Contributor
|
run benchmark arrow_writer |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing batch_rle_run (415b496) to 4fa8d2f (merge-base) diff File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
That's more like it! :noice: |
Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>
Contributor
|
I did do a double check on some of the string regressions and they disappeared on my workstation. |
Contributor
|
As I am still waiting on a few PRs before I cut the 58.2.0 release I figured I'll just merge this one in to make the release that much better |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
See #9731
What changes are included in this PR?
Add
is_accumulating_rle()andextend_run()methods toRleEncoderthat allow callers to detect when the encoder is in RLE accumulation mode and bulk-extend runs without per-element overhead.Upgrade
put_with_observer()inLevelEncoderto exploit this: after eachput(), it checks whether the encoder entered accumulation mode. If so, it scans ahead for the rest of the run, callsextend_run()to batch it in O(1), and fires the observer once with the full run length.This turns the previous O(n) per-value encoding + observation into O(1) amortized per RLE run, which is a significant improvement for sparse columns where long runs of identical levels are common.
Are these changes tested?
All tests passing + added coverage around RLE accumulation mode trigger.
Are there any user-facing changes?
None.