astype errors with Categorical

[jbrockmendel]

import pandas as pd
cat = pd.Categorical(['CA', 'AL'])
df = pd.DataFrame([['CA', 'CA'], ['AL', 'CA']], index=['foo', 'bar'], columns=range(2))
df[1].astype(cat)

with 0.20.1, this raises:

    if dtype == CategoricalDtype():
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This doesn't appear to be quite the intended usage. .astype("category", categories=cat) also fails, though .astype("category", categories=cat.categories) is OK.

I suspect this is related to similar errors in trying to identify which columns of a DataFrame are categorical (possible repeat of #16659):

df.dtypes[colname] == 'category' evaluates as True for categorical columns and raises TypeError: data type "category" not understood for np.float64 columns.

df.dtypes == pd.Categorical raises TypeError: Could not compare <type 'type'> type with Series

Also related: #15078

[chris-b1]

The correct api is the one that works - .astype("category", categories=cat.categories)

cat is a categorical array, not type, so what you are trying is in effect is something like this:

    In [15]: pd.Series([1,2,3]).astype(np.array([1., 2.,  3.]))
    TypeError: data type not understood

So we could give a better message, but I think this is basically working as it should.

[jreback]

this is purely a user error. You are incorrectly passing things as shown by @chris-b1 .

diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index f2a7ac7..b4c3e16 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -138,8 +138,11 @@ class Block(PandasObject):
         returns a boolean if we are a categorical
         """
         if is_categorical_dtype(dtype):
-            if dtype == CategoricalDtype():
-                return True
+            try:
+                if dtype == CategoricalDtype():
+                    return True
+            except:  # no pramga
+                pass
 
             # this is a pd.Categorical, but is not
             # a valid type for astypeing

fixes. this is a bit tricky as categorical is right now a singleton type.

[jbrockmendel]

Makes sense, thanks. Two follow-up questions.

Is there a canonical way to pick out categorical columns within a DataFrame?

Is there a way to pin a pre-existing Category to a Series? Here I'm thinking of cases where multiple columns of a DataFrame should be explicitly tied to the same dtype.

[jorisvandenbossche]

Is there a canonical way to pick out categorical columns within a DataFrame?

You can use select_dtypes

The problem with df.dtypes == 'category' is that this fails (as you have noted) for other dtypes (numpy datatypes), which is beyond pandas to fix (it is due to a specification how numpy dtype comparison works).

Is there a way to pin a pre-existing Category to a Series?

You mean a pre-existing set of categories? Then you can do the .astype("category", categories=cat.categories)

[rafiqhasan]

Simple solution , which works for me always:

str( df.dtypes[colname] ) == 'category'