Issue #1746: Moving QUANTILE #1761
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Move ResizeState into QuantileState. Eventually this will store the index for moving quantile instead of the values (which will be passed in.)
Simplify quantile to to leverate temporal struct types.
…h/duckdb into hawkfish-moving-quantile
Merge as of 2021-04-29
Merge as of 2021-05-03
Code and tests. Waiting on simpler MODE changes.
…h/duckdb into hawkfish-moving-quantile
… into hawkfish-moving-quantile
Move reused objects into the class to improve framed performance by 2x.
Rework the infrastructure for `Frame` function support.
…h/duckdb into hawkfish-moving-quantile
Reduce frame data to a union of both frames. Refactor QUANTILE to remove Value.
Fix std:min/max vs Win64.
Rename Frame to Window. Add support for NULLs to Window API Check for oversized frames.
Add missing NULL detection and test it.
Reduce calls to Reset. Removes one of the two main hot spots in the fixed frame size test computation. This code is shared with the segment tree implementation, so it should help there too.
|
The latest change is just a perf change, but it impacts all windowed aggregates by removing unnecessary memory allocations from most segment tree computations. The perf improvement for windowed quantile (10M rows, fixed frame) is about 20%: ./build/reldebug/test/unittest test/sql/window/test_moving_quantile.test_slow 7.42s user 0.30s system 99% cpu 7.728 total
./build/reldebug/test/unittest test/sql/window/test_moving_quantile.test_slow 6.10s user 0.26s system 96% cpu 6.582 total |
Mytherin
reviewed
May 17, 2021
added 2 commits
May 17, 2021 07:53
Add more tests for scattered NULLs. Fix corner case in MODE Finalize when the hash table is present but empty.
|
Thanks, looks great now! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First pass at moving quantile implementation. There is more to do, but this has an important fix for the API by passing in
NULLmasks, which was a bug.