FEAT-#6803: Enable range-partitioning impl for 'groupby.apply()' by default #6804
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…apply()' by default Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev
commented
Dec 6, 2023
| @@ -3735,14 +3735,6 @@ def groupby( | |||
| skip_on_aligning_flag = "__skip_me_on_aligning__" | |||
|
|
|||
| def apply_func(df): # pragma: no cover | |||
| if any( | |||
There was a problem hiding this comment.
categorical are now always being caught at the query compiler level
dchigarev
commented
Dec 6, 2023
| by_dtypes = self._modin_frame._dtypes.lazy_get(by).get() | ||
| else: | ||
| by_dtypes = self.dtypes[by] | ||
| if any(isinstance(dtype, pandas.CategoricalDtype) for dtype in by_dtypes): |
There was a problem hiding this comment.
We're now materializing 'by' dtypes to consistently catch unsupported cases and fallback to an older implementation. Previously, if the dtypes weren't materialized we were raising an exception in the kernel which caused groupby to fail. Since we're now moving this implementation out of experimental mode, we want more stability here in terms of falling back to an implementation that has more coverage
dchigarev
requested review from
devin-petersohn,
mvashishtha,
RehanSD,
YarShev,
vnlitvinov,
anmyachev and
a team
as code owners
YarShev
reviewed
Dec 6, 2023
| # 'group_wise' means 'groupby.apply()'. We're certain that range-partitioning groupby | ||
| # always works better for '.apply()', so we're using it regardless of the 'ExperimentalGroupbyImpl' | ||
| # value | ||
| if how == "group_wise" or ExperimentalGroupbyImpl.get(): |
There was a problem hiding this comment.
Let's rename ExperimentalGroupbyImpl to RangePartitioningGroupby in a separate PR as we discussed offline.
YarShev
approved these changes
Dec 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
It's believed that range-partitioning implementation is always better for
groupby.apply(), so this PR makes the new implementation to be a default one.flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sgroupby.apply()automatically #6803docs/development/architecture.rstis up-to-date