PERF: Categorical indexing performance regression

[jorisvandenbossche]

Recent regression in the categoricals.CategoricalSlicing.time_getitem_list benchmark: https://pandas.pydata.org/speed/pandas/#categoricals.CategoricalSlicing.time_getitem_list?commits=6efc2379-b9de33e3

Reproducible example for this benchmark:

N = 10 ** 6
categories = ["a", "b", "c"]
values = [0] * N + [1] * N + [2] * N
data = pd.Categorical.from_codes(values, categories=categories)

list_ = list(range(10000))

%timeit data[list_]

Now, this slowdown is due to the changes in #30308. Categorical __getitem__ now checks if the key is a boolean indexer: https://github.com/pandas-dev/pandas/pull/30308/files#diff-f3b2ea15ba728b55cab4a1acd97d996d

So this slowdown is of course expected, and also only for Categorical itself (eg pd.Series indexing already handles this boolean checking). So in that light, we can certainly ignore this regression.
But, this led me think: maybe the ExtensionArrays are a good place to start not supporting object dtype as boolean indexer? (and so not add support for it now, which also avoids this performance regression)

[jorisvandenbossche]

Actually, I think I am wrong in the fact that it is the object dtype support that is the additional overhead in this case. It rather is the conversion of the list to a numpy array, to then only see it is an integer array and not a boolean array, and throw the converted array away (and later have to do the conversion again, when actually indexing with the integer list)

[TomAugspurger]

So IIUC, the best thing to do is convert list inputs into array inputs as early as possible? And then re-use that (hopefully well-typed) input later on?

[jorisvandenbossche]

Yes, that's correct. But seeing that basically every internal ExtensionArray and also external ExtensionArray would want to do this, I am wondering if we rather want to expose something like check_array_indexer instead of the boolean specific one we exposed now. That could also avoid those avoidable extra conversions.

[jorisvandenbossche]

Such a common function might also help with #30738