ValueError in Categorical Constructor with empty data and boolean categories

[TomAugspurger]

This works,

In [15]: pd.Categorical([], categories=['a', 'b'])
Out[15]: [], Categories (2, object): [a, b]

This doesn't

In [16]: pd.Categorical([], categories=[True, False])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-8e79cd310199> in <module>()
----> 1 pd.Categorical([], categories=[True, False])

~/sandbox/pandas/pandas/core/arrays/categorical.py in __init__(self, values, categories, ordered, dtype, fastpath)
    426
    427         else:
--> 428             codes = _get_codes_for_values(values, dtype.categories)
    429
    430         if null_mask.any():

~/sandbox/pandas/pandas/core/arrays/categorical.py in _get_codes_for_values(values, categories)
   2449     (_, _), cats = _get_data_algo(categories, _hashtables)
   2450     t = hash_klass(len(cats))
-> 2451     t.map_locations(cats)
   2452     return coerce_indexer_dtype(t.lookup(vals), cats)
   2453

~/sandbox/pandas/pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.map_locations()
   1330             raise KeyError(key)
   1331
-> 1332     def map_locations(self, ndarray[object] values):
   1333         cdef:
   1334             Py_ssize_t i, n = len(values)

ValueError: Buffer dtype mismatch, expected 'Python object' but got 'unsigned long'

the values there is array([], dtype=object). It should be int dtype by this point.

[pganssle]

I haven't bisected on this to check exactly when it first started manifesting, but this issue is not present in version 0.20.3, but it is present in version 0.21.0.

[pganssle]

Turns out the first bad commit is 7818486859d1aba53ce359b93cfc772e688958e5.

[pganssle]

Ok, so the problem is that in this function, you have:

    if not is_dtype_equal(values.dtype, categories.dtype):
        values = ensure_object(values)
        categories = ensure_object(categories)

    (hash_klass, vec_klass), vals = _get_data_algo(values, _hashtables)
    (_, _), cats = _get_data_algo(categories, _hashtables)

For boolean categories, the category's dtype is already object I guess, so the if not is_dtype_equal(values.dtype, categories.dtype) check is True and the no coercion to object takes place, but inside _get_data_algo, there is a call to _ensure_data, which detects that categories is all booleans and converts it to uint64, which invalidates the assumption that values.dtype and categories.dtype are the same.

You would think that this would be a problem for pd.CategoricalIndex([], categories=[1, 2]) as well, since integers should also be coerced, but evidently _ensure_data treats Index and ndarray differently, and _ensure_object gives an ndarray while categories is an Index. PR incoming.

[TomAugspurger]

Sorry about that :/ Thanks for debugging.

…

On Fri, Sep 14, 2018 at 8:48 AM Paul Ganssle ***@***.***> wrote: Ok, so the problem is that in this function <https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/categorical.py#L2444>, you have: if not is_dtype_equal(values.dtype, categories.dtype): values = ensure_object(values) categories = ensure_object(categories) (hash_klass, vec_klass), vals = _get_data_algo(values, _hashtables) (_, _), cats = _get_data_algo(categories, _hashtables) For boolean categories, the category's dtype is already object I guess, so the if not is_dtype_equal(values.dtype, categories.dtype) check is True and the no coercion to object takes place, but inside _get_data_algo, there is a call to _ensure_data <https://github.com/pandas-dev/pandas/blob/master/pandas/core/algorithms.py#L48>, which detects that categories is all booleans and converts it to uint64, which invalidates the assumption that values.dtype and categories.dtype are the same. You would think that this would be a problem for pd.CategoricalIndex([], categories=[1, 2]) as well, since integers should *also* be coerced, but evidently _ensure_data treats Index and ndarray differently. PR incoming. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22702 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgHT5gpKqc1XbybhBA5x99Y5xZscks5ua7OugaJpZM4WoMaH> .