PERF: df.groupby(categorical)

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.0.0.rst file if fixing a bug or adding a new feature.

Perf improvement in Categorical.reorder_categories with user facing performance improvements likely to be DataFrame.groupby(categorical).

import pandas as pd
import pandas._testing as tm

vals = pd.Series(tm.rands_array(10, 10**6), dtype="string")
df = pd.DataFrame({"cat": vals.astype("category")})

%timeit df.groupby("cat").size()

1.21 s ± 4.71 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     <- main
15.1 ms ± 274 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR

existing ASVs:

       before           after         ratio
     [d69e63d7]       [382245ad]
     <main>           <cat-reorder-categories>
-      3.91±0.1ms       3.08±0.1ms     0.79  groupby.Categories.time_groupby_nosort
-     3.54±0.04ms      2.31±0.02ms     0.65  groupby.Categories.time_groupby_extra_cat_nosort
-     3.85±0.05ms       2.39±0.1ms     0.62  groupby.Categories.time_groupby_ordered_nosort
-     2.05±0.01ms          514±6μs     0.25  groupby.Categories.time_groupby_extra_cat_sort
-      2.21±0.1ms          536±4μs     0.24  groupby.Categories.time_groupby_sort
-      2.23±0.1ms         523±30μs     0.23  groupby.Categories.time_groupby_ordered_sort

[topper-123]

Wow, very nice!

One question, that's only semi-related to this: why do we even use reorder_categories in this groupby? it does seem unrelated especially as we're only using one dtype (if we were combining two dtypes, I would understand it).

Could we get an additional perf. improvement by avoiding reorder_categories in groupby in cases like this?

[lukemanley]

I'm going to guess the rationale for matching the original order of the categories is to keep the result categories dtype equivalent to the input categories dtype. I think we could add a short-circuit that checks to see if they happen to be equal already and if so skip reorder_categories. However, I just tried that locally and could not measure any noticeable improvement in performance.

[topper-123]

LGTM

[topper-123]

Ok. Thanks for trying it out. This performance improvement is already really great as is :-)

[topper-123]

Thanks, @lukemanley. Great stuff.