ENH: add in extension dtype registry #21185

jreback · 2018-05-24T00:35:49Z

precursor to #21160

codecov · 2018-05-24T02:23:17Z

Codecov Report

Merging #21185 into master will increase coverage by <.01%.
The diff coverage is 92.98%.

@@            Coverage Diff             @@
##           master   #21185      +/-   ##
==========================================
+ Coverage    91.9%   91.91%   +<.01%     
==========================================
  Files         154      154              
  Lines       49659    49673      +14     
==========================================
+ Hits        45640    45657      +17     
+ Misses       4019     4016       -3

Flag	Coverage Δ
#multiple	`90.29% <92.98%> (ø)`	⬆️
#single	`41.95% <75.43%> (+0.05%)`	⬆️

Impacted Files	Coverage Δ
pandas/io/formats/format.py	`98.25% <ø> (ø)`	⬆️
pandas/core/algorithms.py	`94.85% <100%> (ø)`	⬆️
pandas/core/dtypes/base.py	`92.3% <100%> (+0.2%)`	⬆️
pandas/core/series.py	`94.19% <100%> (ø)`	⬆️
pandas/core/indexes/interval.py	`93.13% <100%> (ø)`	⬆️
pandas/core/dtypes/common.py	`95.07% <100%> (+0.44%)`	⬆️
pandas/core/arrays/base.py	`87.85% <100%> (+0.26%)`	⬆️
pandas/core/dtypes/cast.py	`88.36% <50%> (-0.13%)`	⬇️
pandas/core/internals.py	`95.52% <81.81%> (-0.07%)`	⬇️
pandas/core/dtypes/dtypes.py	`95.98% <96.77%> (+0.03%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad76ffc...d2c91d7. Read the comment docs.

jbrockmendel · 2018-05-24T02:55:03Z

doc/source/whatsnew/v0.24.0.txt

+^^^^^^^^^^^^^^^^^^^^^
+
+- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` and ``.append()``, and attribute ``array_type`` (:issue:`21185`)
+- ``ExtensionDtype`` has gained the ability to instantiate from string dtypes, e.g. ``decimal`` would instaniate a registered ``DecimalDtype`` (:issue:`21185`)


typo instantiate

Dr-Irv

It seems to me that there are changes in this PR that go beyond just adding in the extension dtype registry. Wasn't your goal to just make this PR about the registry?

jreback · 2018-05-24T11:46:16Z

split out ops to a separate PR #21191

jorisvandenbossche

Didn't look at all the tests yet.

In addition:

We need to discuss what is the exact contract for this registry and array_type: where is it used, and how is it used?
I think now it is used in constructors or astype if you specify a (string) dtype? But then it calls the class constructor of the extension array? Shouldn't this rather be the _from_sequence method ?
Can you add some explanation of the registry to the extension array documentation?
I think this also gives an enhancement for the existing types (interval, datetimetz, period, ..) that you can now specify string dtypes? (since they are added to the registry) In that case, can you add tests for this and a whatsnew notice?

jorisvandenbossche · 2018-05-24T16:14:19Z

doc/source/whatsnew/v0.24.0.txt

+ExtensionType Changes
+^^^^^^^^^^^^^^^^^^^^^
+
+- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` and ``.append()``, and attribute ``array_type`` (:issue:`21185`)


Are the addition of dropna and append orthogonal to the registry, or are they in some way needed for that?

I think we should be hesitant in adding more and more methods to the ExtensionArray if they are not strictly needed

no, these are necessary for followons. The api is not fleshed out enough.

jorisvandenbossche · 2018-05-24T16:24:14Z

pandas/core/arrays/base.py

        """Construct a new ExtensionArray from a sequence of scalars.

        Parameters
        ----------
        scalars : Sequence
            Each element will be an instance of the scalar type for this
            array, ``cls.dtype.type``.
+        copy : boolean, default True
+            if True, copy the underlying data


What does this exactly mean?
Often, the scalars objects are either not stored as is, and then copy or not copy makes no difference. Or if they are stored, they don't necessarily have a 'copy' method.

same guarantee with copy we have already, if its True, then copy if possible.

Can you add that to the docstring?

jorisvandenbossche · 2018-05-24T16:25:27Z

pandas/core/arrays/base.py

    * _concat_same_type
+    * array_type


This is not an attribute of the array class I think?

jorisvandenbossche · 2018-05-24T16:28:32Z

pandas/core/dtypes/cast.py

+
+    # dispatch on extension dtype if needed
+    if is_extension_array_dtype(dtype):
+        return dtype.array_type._from_sequence(arr, copy=copy)


should there be any validation about arr ? Eg that it is 1D?

this validation is a contract of the Array itself

jorisvandenbossche · 2018-05-24T17:16:56Z

pandas/core/dtypes/dtypes.py

+        """
+        Parameters
+        ----------
+        dtype : PandasExtension Dtype


Can you document constructor parameter?
and "PandasExtension Dtype" -> "ExtensionDtype"

jorisvandenbossche · 2018-05-24T17:17:16Z

pandas/core/dtypes/dtypes.py

+        ----------
+        dtype : PandasExtension Dtype
+        """
+        if not issubclass(dtype, (PandasExtensionDtype, ExtensionDtype)):


Shouldn't ExtensionDtype be enough? (our internals one subclass both I think?)

not yet, are extension dtypes are not all ExtensiDtypes yet (this is why we have the whole PandasExtensionDtype bizness)

jorisvandenbossche · 2018-05-24T17:21:29Z

pandas/core/dtypes/dtypes.py

+        if string.startswith('period[') or string.startswith('Period['):
+            # do not parse string like U as period[U]
+            return PeriodDtype.construct_from_string(string)
+        raise TypeError("could not construct PeriodDtype")


Should we consider rather changing construct_from_string itself? As those don't really adhere to the ExtensionDtype requirements I think?

If we keep both, I would make this one private.

jorisvandenbossche · 2018-05-24T17:29:07Z

pandas/tests/extension/base/ops.py

+
+class BaseOpsTests(BaseExtensionTests):
+    """Various Series and DataFrame ops methos."""
+    pass


This is a left-over from splitting it from the previous PR?

jreback · 2018-05-24T23:04:34Z

all fixed up @jorisvandenbossche (except for docs on Registry in Extension section)

jreback · 2018-05-25T13:58:31Z

see if can make this dispatch to create_array_type as a function

jreback · 2018-06-03T22:35:26Z

updated docs

@jorisvandenbossche @TomAugspurger if any final comments.

jorisvandenbossche

Added some more comments.

From my previous review:

We need to discuss what is the exact contract for this registry and array_type: where is it used, and how is it used?

Can you add some explanation about this to the docs/docstrings? (eg that it relies on _from_sequence)

jorisvandenbossche · 2018-06-04T09:10:10Z

pandas/core/arrays/base.py

        """Construct a new ExtensionArray from a sequence of scalars.

        Parameters
        ----------
        scalars : Sequence
            Each element will be an instance of the scalar type for this
            array, ``cls.dtype.type``.
+        copy : boolean, default True
+            if True, copy the underlying data


Can you add that to the docstring?

jorisvandenbossche · 2018-06-04T09:11:32Z

pandas/core/dtypes/base.py

+        type
+        """
+        if array is None:
+            return cls


I think cls is never the correct return? (since it is a dtype class, while this should return an array class?)

jorisvandenbossche · 2018-06-04T09:19:56Z

pandas/core/dtypes/dtypes.py

+    """ Registry for dtype inference
+
+    We can directly construct dtypes in pandas_dtypes if they are
+    a type; the registry allows us to register an extension dtype


The sentence "We can directly construct dtypes in pandas_dtypes if they are a type" is not really clear to me. What does "if they are a type" mean? To which context does this refer?

jorisvandenbossche · 2018-06-04T09:44:14Z

pandas/tests/dtypes/test_dtypes.py

+    'dtype, expected',
+    [('int64', None),
+     ('interval', IntervalDtype()),
+     ('interval[int64]', IntervalDtype()),


can you replace this (or also add) one that is not the default?
Eg ('interval[datetime64[ns]]', IntervalDtype('datetime64[ns]')),

(although that is maybe covered with the period or datetime64 one below. Is 'interval[datetime64[ns]]' already tested for IntervalDtype?)

jorisvandenbossche · 2018-06-04T10:05:04Z

pandas/tests/extension/base/constructors.py

+        dtype = data.dtype
+
+        expected = pd.Series(data)
+        result = pd.Series(np.array(data), dtype=dtype)


Do we have a restriction on what the result of __array__ should be? Maybe not necessarily the scalars?

So maybe better to do list(data) or data.astype(object)

jorisvandenbossche · 2018-06-04T10:12:00Z

pandas/tests/extension/decimal/test_decimal.py

+    @pytest.mark.xfail(reason="not implemented constructor from dtype")
+    def test_from_dtype(self, data):
+        # construct from our dtype & string dtype
+        pass


Can you actually implement this? (only change needed would be to register DecimalDtype I think?)
As that way we actually have a test for external dtypes registering?

I do in subsequent PR's trying to keep this diff down.

jorisvandenbossche · 2018-06-04T10:13:32Z

pandas/tests/extension/decimal/test_decimal.py

+# TODO(extension)
+@pytest.mark.xfail(reason=(
+    "raising AssertionError as this is not implemented, "
+    "though easy enough to do"))


I think you can change the AssertionError to a ValueError in the code, and then we can still test this.

this is ok for now

jorisvandenbossche · 2018-06-04T10:14:53Z

pandas/tests/extension/json/test_json.py

+    @pytest.mark.xfail(reason="not implemented constructor from dtype")
+    def test_from_dtype(self, data):
+        # construct from our dtype & string dtype
+        pass


Can we test here that an appropriate error is raised (instead of skipping the test) ?

jorisvandenbossche · 2018-06-04T10:17:31Z

pandas/core/dtypes/base.py

@@ -109,6 +109,11 @@ class ExtensionDtype(_DtypeOpsMixin):
    * name
    * construct_from_string

+    Optionally one can override construct_array_type for construction
+    with the name of this dtype via the Registry


I would give some additional explanation of what the Registry is, because now this is not explained here?

jorisvandenbossche · 2018-06-04T10:18:48Z

doc/source/whatsnew/v0.24.0.txt

+ExtensionType Changes
+^^^^^^^^^^^^^^^^^^^^^
+
+- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` and ``.append()`` (:issue:`21185`)


As I said before, I am not convinced we should necessarily add those methods (certainly append), so would personally leave this for another PR to discuss

Agreed with not adding append.

Could go either way on dropna.

i need them for subsquent PR's I guess will move them.

TomAugspurger · 2018-06-04T12:48:49Z

doc/source/whatsnew/v0.24.0.txt

+ExtensionType Changes
+^^^^^^^^^^^^^^^^^^^^^
+
+- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` and ``.append()`` (:issue:`21185`)


Agreed with not adding append.

Could go either way on dropna.

TomAugspurger · 2018-06-04T12:51:57Z

doc/source/whatsnew/v0.24.0.txt

+
+- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` and ``.append()`` (:issue:`21185`)
+- ``ExtensionDtype`` has gained the ability to instantiate from string dtypes, e.g. ``decimal`` would instantiate a registered ``DecimalDtype``; furthermore
+  the dtype has gained the ``construct_array_type`` (:issue:`21185`)


'the dtype' -> ExtensionDtype has gained :meth:`ExtensionDtype.construct_array_type`

TomAugspurger · 2018-06-04T12:52:40Z

doc/source/whatsnew/v0.24.0.txt

 .. _whatsnew_0240.api.other:

 Other API Changes
 ^^^^^^^^^^^^^^^^^

-
+- Invalid consruction of ``IntervalDtype`` will now always raise a ``TypeError`` rather than a ``ValueError`` if the subdtype is invalid (:issue:`21185`)


consruction -> construction.

TomAugspurger · 2018-06-04T12:55:10Z

pandas/core/arrays/base.py

@@ -379,6 +383,16 @@ def fillna(self, value=None, method=None, limit=None):
            new_values = self.copy()
        return new_values

+    def dropna(self):
+        """ Return ExtensionArray without NA values


Remove the leading space. Add a trailing .

TomAugspurger · 2018-06-04T12:57:42Z

pandas/core/dtypes/base.py

+
+        Parameters
+        ----------
+        array : array-like, optional


@jorisvandenbossche by "this" you mean the array argument right? I'm also wondering that.

TomAugspurger · 2018-06-04T13:02:29Z

pandas/core/dtypes/dtypes.py

@@ -8,6 +8,64 @@
 from .base import ExtensionDtype, _DtypeOpsMixin


+class Registry(object):


Without looking at the uses yet, could we simplify this a by just allowing string lookup? Ideally, registry would be a simple class holding a dict mapping dtype.name -> ExtensionDtype.

I think the current code also supports finding the dtype for eg 'interval[int64]' and not just interval (so parametrized versions of the strings)

Oh, and I suppose we want that to support .astype('interval[int64]'). That's fair...

TomAugspurger · 2018-06-04T13:04:17Z

pandas/core/indexes/interval.py

@@ -800,7 +800,7 @@ def astype(self, dtype, copy=True):
    @cache_readonly
    def dtype(self):
        """Return the dtype object of the underlying data"""
-        return IntervalDtype.construct_from_string(str(self.left.dtype))
+        return IntervalDtype(str(self.left.dtype))


left.dtype.name? I'm not sure when these differ, but using .name seems safer.

jreback · 2018-06-19T12:36:44Z

ok all fixed up. @jorisvandenbossche @TomAugspurger if you want to look.

Dr-Irv · 2018-06-19T13:27:00Z

@jreback I'm wondering if there is a meta issue with the registry that should be thought about. Let's say that pandas registers 10 types - they are always in the registry for all pandas users. Now let's say you have 2 different people who independently extend pandas with the EA capabilities creating libraries ABC and DEF. But they happen to both call their extension array type MyEAType . Now if someone wants to use both libraries ABC and DEF, there will be a collision in the registry of MyEAType. So does there need to be some "master" registry in the documentation containing the names of all EA types that people have registered?

I realize this is a highly unlikely occurrence.

Put another way, shouldn't there be code for the registry that says "this type has already been registered"?

Also, the implementation of the registry is using a list. Shouldn't you use a dict or set?

jreback · 2018-06-19T20:17:20Z

@Dr-Irv you don;t need to register the EA types, this is only for the purpose of translating a string dtype name -> EA dtype (e.g. 'categorical' -> Categorical), basically for convenience.

so sure you could have a collision, but then user would have to deal with that (this wouldn't show up on pandas because we won't have colliding names).

jreback · 2018-06-19T20:19:05Z

@Dr-Irv I had this an OrderdedDict but its actually not necessary, and is actually not correct. If we have a string, we have to ask the dtypes can you construct a type from it. The issue is that multiple strings can map to a dtype when its parameterized, e.g. datetime64[ns, US/Eastern] is a valid string dtype as is datetime64[ns, Asia/Tokyo] and w/o imbuing the knowledge to say its 'good' inside the Register (bad idea), we can't know something works unless we duck type it.

Dr-Irv · 2018-06-19T20:25:20Z

@jreback Thanks for the clarification of the need versus require of registration of EA types.

On the implementation, wouldn't OrderedDict be faster once there are a lot of types?

TomAugspurger

Aside from adding EA.append, I'm +1 on this.

Given that NumPy doesn't implement it, I don't think we should either.

jreback · 2018-06-20T21:02:27Z

I had removed append FYI (it should not longer be in this PR)

This reverts commit 6d5c67b.

jreback · 2018-07-02T23:18:07Z

any final comments. going to rebase.

jreback · 2018-07-02T23:20:47Z

@jorisvandenbossche @TomAugspurger

jreback · 2018-07-03T14:24:55Z

merging.

Closes pandas-dev#21185

jreback added the ExtensionArray Extending pandas with custom dtypes or arrays. label May 24, 2018

jreback added this to the 0.24.0 milestone May 24, 2018

jreback mentioned this pull request May 24, 2018

ENH: Integer NA Extension Array #21160

Merged

jbrockmendel reviewed May 24, 2018

View reviewed changes

Dr-Irv reviewed May 24, 2018

View reviewed changes

jreback force-pushed the eat branch from 78abef9 to 9b2cdb0 Compare May 24, 2018 11:04

jreback mentioned this pull request May 24, 2018

ENH: extension ops #21191

Closed

jorisvandenbossche reviewed May 24, 2018

View reviewed changes

jreback force-pushed the eat branch from 9b2cdb0 to a854f06 Compare May 24, 2018 23:03

jreback force-pushed the eat branch from a854f06 to f5a0c24 Compare May 25, 2018 11:33

jreback force-pushed the eat branch 3 times, most recently from 92a1322 to 10aab3c Compare May 31, 2018 10:36

jreback mentioned this pull request Jun 3, 2018

Series[EA].astype always casts intermediate to NumPy array #21296

Closed

jreback force-pushed the eat branch from 10aab3c to ff34f01 Compare June 3, 2018 22:34

jorisvandenbossche reviewed Jun 4, 2018

View reviewed changes

TomAugspurger reviewed Jun 4, 2018

View reviewed changes

jreback force-pushed the eat branch from ff34f01 to 558a639 Compare June 19, 2018 12:35

TomAugspurger reviewed Jun 20, 2018

View reviewed changes

jreback added 21 commits June 28, 2018 20:41

ENH: add in extension dtype registry

4b132cf

review comments

90fccb0

remove ops

80e15fd

move array_type -> construct_array_type

1c3d023

interval dtypes

4b05a1c

doc in construct_array_type

9cfbc07

add repr tests

72e1010

make construct_array_type arg optional

59b3510

small doc enhancements

07d1038

review comments

7dbd2f3

remove arg from construct_array_type as no longer needed

7a2cb6a

doc comments

96d5d09

lint

5119672

remove doc note

e22d4c7

remove append

7cc1a0a

remove dropna text

930cec5

Revert "remove dropna text"

bd5dcd3

This reverts commit 6d5c67b.

remove arg from construct_array_type in base class

407d7b3

use simpler dtype for intervals

9084906

rebase

f560ea1

Merge branch 'master' into eat

5fabd51

jreback force-pushed the eat branch from dd74832 to 5fabd51 Compare July 2, 2018 10:49

Merge branch 'master' into eat

d2c91d7

jreback merged commit 2f14faf into pandas-dev:master Jul 3, 2018

TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Aug 14, 2018

API: dispatch to EA.astype

a7ba8f6

Closes pandas-dev#21185

TomAugspurger mentioned this pull request Aug 14, 2018

API: dispatch to EA.astype #22343

Merged

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

ENH: add in extension dtype registry (pandas-dev#21185)

3699007

		@@ -8,6 +8,64 @@
		from .base import ExtensionDtype, _DtypeOpsMixin


		class Registry(object):

ENH: add in extension dtype registry #21185

ENH: add in extension dtype registry #21185

Conversation

jreback commented May 24, 2018

codecov bot commented May 24, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Dr-Irv left a comment

Choose a reason for hiding this comment

jreback commented May 24, 2018

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented May 24, 2018

jreback commented May 25, 2018

jreback commented Jun 3, 2018

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche Jun 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 19, 2018

Dr-Irv commented Jun 19, 2018

jreback commented Jun 19, 2018

jreback commented Jun 19, 2018 • edited Loading

Dr-Irv commented Jun 19, 2018

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback commented Jun 20, 2018

jreback commented Jul 2, 2018

jreback commented Jul 2, 2018

jreback commented Jul 3, 2018

codecov bot commented May 24, 2018 •

edited

Loading

jorisvandenbossche Jun 4, 2018 •

edited

Loading

jreback commented Jun 19, 2018 •

edited

Loading