Clean up percentiles_summary logic
#11094
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rjzamora
added
dataframe
enhancement
rjzamora
commented
May 3, 2024
| # `data.values` doesn't work for cudf, so we need to | ||
| # use `quantile(..., method="table")` as a fallback | ||
| # Series.quantile doesn't work with some data types (e.g. strings) | ||
| if PANDAS_GE_150: |
There was a problem hiding this comment.
Hopefully this will ensure that the unnecessary _percentile code path gets removed when pandas<1.5 is no longer supported.
rjzamora
commented
May 3, 2024
Comment on lines
+457
to
+464
| # Convert to array if necessary (and possible) | ||
| if is_series_like(vals): | ||
| try: | ||
| vals = vals.values | ||
| except (ValueError, TypeError): | ||
| # cudf->cupy won't work if nulls are present, | ||
| # or if this is a string dtype | ||
| pass |
There was a problem hiding this comment.
My sense is that this entire code block can/should be removed for the sake of simplicity. However, I'm currently leaving the .values operation in place to keep the scope if this PR small.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ±0 15 suites ±0 3h 24m 15s ⏱️ +39s Results for commit 4f29798. ± Comparison against base commit efb4a62. |
phofl
approved these changes
May 6, 2024
|
thx |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is essentially a revision of #10551, which generalized the partition-wise
quantileslogic used inpercentiles_summaryto work with bothcudfandpandas-backed data. It turns out that those changes do not quite cover the case that a numerical column contains null values.An additional problem with the current logic is that error handling for the
.valuesoperation is mixed with error handling for thequantileoperation itself. This PR separates these steps a bit to make the logic less confusing.Overview of the partition-wise
quantileslogic after this PR:Series.quantilemethod (without calling.valuesyet)DataFrame.quantile(..., method="table")method as ifpandas>=1.5pandas<1.5, we fall back topercentile(this entire code path can be removed when pandas2+ is required).values. Sincecudf->cupywill fail to do this for string columns and/or null values, we simplypassif this step fails..valueshere at all. I believe NEP18 covers all necessarySerieslogic anyway. Perhaps we can remove this step?