API: relax categorical equality when comparing against object

[jreback]

from SO

In [1]: a = pd.Series(['a','b','c'],dtype="category")
In [2]: b = pd.Series(['a','b','c'],dtype="object")
In [3]: c = pd.Series(['a','b','cc'],dtype="object")
In [5]: a==b
TypeError: Cannot compare a Categorical for op <built-in function eq> with type <type 'numpy.ndarray'>. If you want to 
compare values, use 'series <op> np.asarray(cat)'.

In [6]: A = pd.DataFrame({'A':a,'B':[1,2,3]})
In [7]: B = pd.DataFrame({'A':b,'C':[4,5,6]})

In [9]: A.merge(B,on='A') 
Out[9]: 
   A  B  C
0  a  1  4
1  b  2  5
2  c  3  6

In [10]: A.merge(B,on='A').dtypes
Out[10]: 
A    object
B     int64
C     int64
dtype: object

In [11]: A.dtypes
Out[11]: 
A    category
B       int64
dtype: object

In [12]: B.dtypes
Out[12]: 
A    object
C     int64
dtype: object

[jreback]

cc @JanSchulz
@jorisvandenbossche
cc @shoyer
cc @immerrr

I think the merging is the correct behavior (@immerrr and I worked on this for a while to make it correct).

as a single categorical merging with either an equal categorical (meaning the same categories) or an object array is OK, though you do lose the 'character', meaning the Categorical of the dtypes. But this is like downgrading a type, e.g. going from integer to float when you are merging a float. So I don't think this is that big of a deal.

Note that we could try to infer the categorical if possible after merging (which is possible if the object happens to not contain new categories).

[shoyer]

Agreed, the first example should work.

[immerrr]

Yup, it seems natural to enable that.

[jankatins]

The problem is when there is a custom ordering, then a object array with "a"<"b"<"c" is not the same as an categorical with "a">"b">"c", so what should "a" < "b" comparison do: use the object order or the categorical one?

In my head I think about categorical elements as a new type and under python 3, different types are not comparable per default and I went with that thought in the implementation.

[jreback]

@JanSchulz I agree with you on the ordering case and < or >. Clearly they are not comparable.

However, merging and equality are are about positional equality.

and we already support comparisons against a scalar

In [2]: s = Series(list('abc'),dtype='category')

In [3]: s
Out[3]: 
0    a
1    b
2    c
dtype: category
Categories (3, object): [a < b < c]

In [4]: s=='a'
Out[4]: 
0     True
1    False
2    False
dtype: bool

I'll put up a PR

[jankatins]

Ah, ok, now I understand: more or less merge treats each value in the "to-be-merged" object array as a single value, so the scalar cases applies.