Refactor of rolling_window implementation. - #8158
Merged
rapids-bot[bot] merged 27 commits intoMay 24, 2021
Merged
Conversation
…e rolling_window interface.
…ad of unique_ptrs.
…ter the finalizer. Refactor groupby to use this mechanism. Change finalizer to not be an abstract class in the interest of keeping it extensible for rolling window and other users.
nvdbaranec
added
libcudf
mythrocks
reviewed
May 13, 2021
mythrocks
reviewed
May 13, 2021
mythrocks
requested changes
May 14, 2021
mythrocks
left a comment
mythrocks
left a comment
Contributor
There was a problem hiding this comment.
I think I've gotten my head around this change now. Sorry, it took a little time.
I had a couple of nitpicks, around DeviceRolling*::is_supported() and if constexpr.
I haven't fully grokked where the dictionary specific code in cudf::detail::rolling_window() might be moved. I'm looking forward to reviewing that change.
hyperbolic2346
approved these changes
May 17, 2021
mythrocks
reviewed
May 18, 2021
mythrocks
reviewed
May 18, 2021
mythrocks
approved these changes
May 18, 2021
Contributor
Author
|
rerun tests |
jrhemstad
reviewed
May 20, 2021
mythrocks
reviewed
May 21, 2021
…ded a note about removing some code once is_valid_aggregation<> gets cleaned up a bit.
jrhemstad
approved these changes
May 24, 2021
Contributor
Author
|
@gpucibot merge |
rapids-bot Bot
pushed a commit
that referenced
this pull request
May 28, 2021
Fixes the rolling-window part of #7611. All the rolling window functions return empty results when the input aggregation column is empty, just as they should. But the column types are incorrectly set to match the input type. While this is alright for `[MIN(), MAX(), LEAD(), LAG()]`, it is incorrect for some aggregations: Aggregation | Input Types | Output Type | --------------|----------------------|-----------------------------------| COUNT_VALID | All types | INT32 | COUNT_ALL | All types | INT32 | ROW_NUMBER | All types | INT32 | SUM | Numerics (e.g. INT8) | 64-bit promoted type (e.g. INT64) | SUM | Chrono | Same as input type | SUM | All else | Unsupported | MEAN | Numerics | FLOAT64 | MEAN | Chrono | FLOAT64 | MEAN | All else | Unsupported | COLLECT_LIST | All types T | LIST with child of type T | This mapping is congruent with `cudf::target_type_t` from `<cudf/detail/aggregation/aggregation.hpp>`. This commit corrects the type of the output column that results from an empty input. It adds test for all the combinations listed above. Note: This is dependent on #8158, and should be merged after that is committed. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Nghia Truong (https://github.com/ttnghia) - https://github.com/nvdbaranec - Vyas Ramasubramani (https://github.com/vyasr) URL: #8274
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an attempt to significantly reduce the complexity of the logic of the SFINAE and various functors/functions inside of rolling_detail.cuh. There are 2 major components:
It introduces the idea of device "rolling operators". These operators are essentially just the implementations of what were formerly the
process_rolling_window()functtions. However, they provide they key mechanism for removing the complex SFINAE out of the core logic. They do this by providing their own logic that can throw for invalid aggregation/type pairs at construction time, internally.It refactors the type and aggregation-dispatched functors to use the collector/finalize paradigm used by groupby. Specifically, the rolling operation is broken down into three parts. 1.) Preprocess incoming aggregation/type pairs, potentially transforming them into different operations. 2.) Perform the rolling window operation on the transformed inputs. 3.) Postprocess the output from the rolling rolling window operation to obtain the final result.
Combined, these two changes dramatically reduce the amount of dispatch and gpu rolling implementation code one has to read through.
The implementation of the collect list rolling operation has been moved into
rolling_collect_list.cuhThere are a couple of other things worth mentioning:
Each device rolling operator implements an
is_supported()constexpr function which are stripped down, type-specific versions of the oldis_rolling_supported()global function. It might be possible to eliminate this with further fundamental type traits. Looking for opinions here.is_rolling_supported()has been removed from the code, however the various tests relied on it pretty heavily. So for now I just transplanted it into the test code in a common place. It's definitely not an ideal solution, but maybe ok for now.It might be worth moving the device rolling operators into their own module to further shrink
rolling_detail.cuh. Also looking for opinions here.