[WIP]API: CategoricalType for specifying categoricals

[TomAugspurger]

closes #14676

This adds a new top-level class pd.CategoricalType for representing the type information on a Categorical, without the values. Users can pass this anywhere they would previously pass 'category' if they need to specify non-default values for categories or ordered (you can still use 'category' of course):

In [1]: from pandas import *

In [2]: paste
    >>> t = CategoricalType(categories=['b', 'a'], ordered=True)
    >>> s = Series(['a', 'a', 'b', 'b', 'a'])
    >>> c = s.astype(t)
    >>> c

## -- End pasted text --
Out[2]:
0    a
1    a
2    b
3    b
4    a
dtype: category
Categories (2, object): [b < a]

This is the simplest possible change for now. A bigger change would be to make c.dtype return a CategoricalType instead of 'category'. We could probably do that in a way that's backwards compatible, but I'll need to think on it a bit.

Other places this should work

• select_dtypes (currently returns all categoricals; should just be ones with same CategoricalType?)
• read_csv: need to implement, see API: Expand read_csv dtype for categoricals #14503
• Series(..., dtype=...)
• probably others.

Implementation-wise I need to document, more tests, clean up some things like the repr. But I wanted to get this out there for discussion. @JanSchulz you might be interested.

[jreback]

this should be in with the rest of the dtypes

to be honest i wouldn't create this; just add it optionally into the existing

[TomAugspurger]

Is that OK with the caching and stuff that is done on CategoricalDType? This is on CategoricalyDtype:

    def __new__(cls):

        try:
            return cls._cache[cls.name]
        except KeyError:
            c = object.__new__(cls)
            cls._cache[cls.name] = c
            return c

I haven't messed with extension types much. We could make the keys of that internal dict reflect the categories and ordered attributes.

[jreback]

yes you would had optional attributes and cache based on them

[TomAugspurger]

K, this seems to be working. Just have to convert the categories to a tuple so that they're hashable. Thanks.

[TomAugspurger]

What would the ideal __repr__ of CategoricalDtype be? Currently it's just category. For places like DataFrame.info it's probably best to keep that as is, but that's not as useful if you're inpsecing CategoricalDtype itself.

Also, for equality semantics, this is what I have right now

    An instance of ``CategoricalDtype`` compares equal with any other
    instance of ``CategoricalDtype``, regardless of categories or ordered.
    In addition they compare equal to the string ``'category'``.
    To check whether two instances of a ``CategoricalDtype`` match,
    use the ``is`` operator.

    >>> t1 = CategoricalDtype(['a', 'b'], ordered=True)
    >>> t2 = CategoricalDtype(['a', 'c'], ordered=False)
    >>> t1 == t2
    True
    >>> t1 == 'category'
    True
    >>> t1 is t2
    False
    >>> t1 is CategoricalDtype(['a', 'b'], ordered=True)
    True

though I don't expect people to be working with these objects that much.

[jreback]

this has been around for quite some time, you are adding parameter support in 0.20.0

[jreback]

list-like, use similar to whats in Categorical now

[jreback]

these examples are not relevant. This should be not include Series. This is a self-contained type.

[jreback]

this needs all of the validation logic (from Categorical). on the actual categories.

[jreback]

you need to move over the repr (of the categories) and the validation logic on the categories to this class.

Further Categorical should use this internally for storage of categories / ordered.

This is a fairly invasive change and needs some thought.

[jreback]

I would split off the actual issue change and make that a follow up PR. Just putting in place the correct infrastructure can be the scope of this PR>

[TomAugspurger]

This is a fairly invasive change and needs some thought.

Yep. I had hoped to do the minimal change of just providing a new API, but let's do it right the first time. I'll spend some time on this the next couple weeks and ping when I have something.

[TomAugspurger]

Closing for now. Will reopen a new PR on top of a PR fixing #14711