API: Standardize ExtensionDtype subtype name

[TomAugspurger]

We have a few extension arrays that build off one or more ndarrays. You'll often want to get the underlying NumPy dtype for that array.

• IntegerArray uses .type
• IntervalArray uses .subtype

IntervalArray can't use .type, since that has to be Interval. So IntegerArray will need to alias subtype to type.

Are we happy with the name subtype?

[jorisvandenbossche]

+1 for subtype

[jorisvandenbossche]

IntegerArray actually has numpy_dtype

[TomAugspurger]

We also set subdtype on the base PandasExtensionDtype, but don't use it anywhere. That seems to be mirroring https://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.subdtype.html, which I don't really understand.

We should first agree on what exactly this thing (call it subtype for now) is supposed to mean.

• IntervalDtype: the dtype used for the left and right arrays
• SparseDtype: the dtype of the non-fill-value values
• IntegerDtype: the dtype that would be used, if only NumPy supported nullable integer arrays
• PeriodDtype: could be int for the ordinals
• DatetimeTZDtype: could be datetime64[ns]

So something like "the type this would correspond to in NumPy, if NumPy supported this directly"?

Given all that, I think I like numpy_type as the best name.

[jorisvandenbossche]

So something like "the type this would correspond to in NumPy, if NumPy supported this directly"?

But this is not what you listed for all dtypes, so maybe we should not have a single name. Eg for intervals, even if numpy supported interval dtype, this would not be the same dtype as the interval bounds itself (the same for period).
And for categorical, you have the dtype of the codes, and the dtype of the categories.

So maybe we shouldn't standardize too much. numpy_dtype could also be kept for #22791

[TomAugspurger]

Yes, you're correct... So if we can't pin down the concept we're expressing here, standardizing won't be that useful.

…

On Mon, Oct 8, 2018 at 8:10 AM Joris Van den Bossche < ***@***.***> wrote: So something like "the type this would correspond to in NumPy, if NumPy supported this directly"? But this is not what you listed for all dtypes, so maybe we should not have a *single* name. Eg for intervals, even if numpy supported interval dtype, this would not be the same dtype as the interval bounds itself (the same for period). And for categorical, you have the dtype of the codes, and the dtype of the categories. So maybe we shouldn't standardize too much. numpy_dtype could also be kept for #22791 <#22791> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIm9kJt4ekT3huCwz6-32zVitK2FSks5ui07fgaJpZM4VxDX-> .

[jorisvandenbossche]

Let's do this for 0.24.0 ? Otherwise we need to start deprecating things if we still want to change it after the initial release, which is a bit stupid for this.

I don't think it is much work, it is mainly deciding what we want to do

[TomAugspurger]

Yeah, not much work, aside from the hard problem of "what do we want". I haven't really come to a decision on what's best here.

What we can relatively easily do for 0.24 is decide if anything should be assigned a common name

1. IntervalDtype.subtype
2. SparseDtype.subtype
3. Int*Dytpe.numpy_dtype
4. CaegoricalDtype.codes.dtype or CategoricalDtype.categories.dtype

I'm really not sure here... Maybe there all OK as is.

[jorisvandenbossche]

In the meantime, the PandasDtype (for PandasArray holding numpy array) also has a numpy_dtype attribute.

IntervalDtype has a subtype (but also subdtype which is defined in PandasExtensionDtype, not sure what is that for. For numpy compat?).

I'm really not sure here... Maybe there all OK as is.

I think the main question might be what we want to do with #22791. Because a name there might clash with some of the names used here.

[jreback]

yeah I ran into this when fixing up to use pandas_dtype. We also have a .base attribute that need some rationalization.

We ought to make these NotImplemented on the base dtype to catch errors.

I think subtype is ok where it is now for Sparse, maybe add to Categorical in the same capacity?
numpy_dtype is also pretty clear.

I would allow having one of [subtype|numpy_dtype] for dtypes.

[jorisvandenbossche]

@TomAugspurger about your list above: I think SparseDtype actually uses subtype and not numpy_dtype?

[TomAugspurger]

Yes, sorry. Updated.

[TomAugspurger]

Does anyone have a concrete proposal for what to update, if anything? I've struggled to form an opinion here.

[TomAugspurger]

Pushing to 0.24.2.

[JBGreisman]

To follow up on this old thread, is there a currently supported way to get numpy-compatible dtype inference for an ExtensionDtype?

It seems that one should be able to define a custom ExtensionDtype._get_common_dtype() method. This ends up working in some cases where a DataFrame is populated with a non-unique set of custom ExtensionDtypes. Unfortunately, if one has a DataFrame with homogeneous ExtensionDtypes these lines get hit:

pandas/pandas/core/dtypes/cast.py

Lines 1789 to 1792 in ce3bac9

	# workaround for find_common_type([np.dtype('datetime64[ns]')] * 2)
	# => object
	if all(is_dtype_equal(first, t) for t in types[1:]):
	return first

This in turn causes the BlockManager to fall back to a numpy object dtype:

pandas/pandas/core/internals/managers.py

Lines 1462 to 1469 in ce3bac9

	# TODO: https://github.com/pandas-dev/pandas/issues/22791
	# Give EAs some input on what happens here. Sparse needs this.
	if isinstance(dtype, SparseDtype):
	dtype = dtype.subtype
	elif isinstance(dtype, ExtensionDtype):
	dtype = np.dtype("object")
	elif is_dtype_equal(dtype, str):
	dtype = np.dtype("object")

Are there any thoughts on modifying that isinstance(dtype, ExtensionDtype) clause to do something like this to allow an ExtensionDtype to avoid falling back to object-dtypes in numpy?

elif isinstance(dtype, ExtensionDtype):
    if hasattr(dtype, "numpy_dtype"):
        dtype = dtype.numpy_dtype
    elif hasattr(dtype, "subtype"):
        dtype = dtype.subtype
    else:
        dtype = np.dtype("object")

[jbrockmendel]

The issue here is standardizing idiosyncratic naming across different internal dtypes, not defining an API we expect 3rd parties to implement? If so, I'm really not bothered by the existing names not matching. They mean different things for different dtypes right?

The closest thing I have to an opinion is maybe they should be privatized?