Preserve order if necessary when deduping categoricals internally #11597
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## branch-22.10 #11597 +/- ##
===============================================
Coverage ? 86.41%
===============================================
Files ? 145
Lines ? 22998
Branches ? 0
===============================================
Hits ? 19874
Misses ? 3124
Partials ? 0 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
brandon-b-miller
added
bug
|
rerun tests |
| .drop_duplicates(ignore_index=True) | ||
| ._column | ||
| ) | ||
| new_cats = dedup_preserve_order(cudf.Series(new_cats))._column |
There was a problem hiding this comment.
Can we have dedup_preserve_order operate on a Column rather than a Series? As much as possible, we want to avoid this pattern of constructing a Series out of a column, invoking a Series operation on it, and then getting the column back out of the result.
Having it operate on a column would have the added benefit that we can then make it a method of CategoricalColumn.
There was a problem hiding this comment.
@shwina those are all good points, I'll update as such.
There was a problem hiding this comment.
I added this to ColumnBase in the end as I ended up not being able to see any reason all columns shouldn't share the capability. Let me know if this should be changed.
| @@ -217,6 +217,16 @@ def dropna(self, drop_nan: bool = False) -> ColumnBase: | |||
| # The drop_nan argument is only used for numerical columns. | |||
| return drop_nulls([self])[0] | |||
|
|
|||
| def _dedup_preserve_order(self): | |||
There was a problem hiding this comment.
Should this just be merged with the unique() method? I'm thinking col.unique() returns unique values in arbitrary order, and col.unique(preserve_order=True) does this.
There was a problem hiding this comment.
Discussed this offline and came to the conclusion that long term we should implement #11638 to back this.
There was a problem hiding this comment.
Can we still collapse this _dedup_preserve_order() method into unique()?
|
@gpucibot merge |
Closes #11486