Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REF: DatetimeLikeArray #24024

Merged
merged 220 commits into from Jan 2, 2019

Conversation

@TomAugspurger
Copy link
Contributor

commented Nov 30, 2018

TODO:

  • DatetimeLikeArrayMixin.astype(int) ignores sign and size?
  • _from_sequence too permissive?
  • OK with DatetimeArray.view?
  • tz vs. dtype in DatetimeArray.__init__: #24024 (comment)
  • Index._simple_new consistency: #24024 (comment)

n.b.: right now I have a few other PRs in this branch (#23601, #23990, #24023). The eventual diff is at TomAugspurger/pandas@disown-tz-only-target...TomAugspurger:disown-tz-only-rebased.

Also, I would say this isn't quite ready for review yet, but I've been promising it for a while. I'm going to try to split off a bit more into separate PRs.


This implements DatetimeArray and TimedeltaArray, arrays for storing datetime
(tz naive or aware) and timedelta data.

High-level Design:

Everything here applies equally to TimedeltaIndex / TimedeltaArray, other than
references to DatetimeTZDtype. We don't have an ExtensionType for Timedelta.

DatetimeIndex's data is now a DatetimeArray, rather than an ndarray, and its dtype
is dtype::Union['datetime64[ns]', DatetimeTZDtype].

Class Hierarchy: We've split DatetimeIndex to compose a DatetimeArray,
rather than inherit it.

Old - DatetimeIndex: dti_master

New - DatetimeIndex: dti_pr

New - DatetimeArray: dta_pr

Internals I've mostly given up here... Things are basically as they were
before, with the exception that DatetimeTZBlock now extends ExtensionBlock. In
particular
tz-naive data is still stored in a DatetimeBlock, and is still
consolidatable. This isn't elegant, but we've lived with it for this long.
Future releases can clean it up, possibly when we rewrite / cythonize the block
manager, possibly before.

Types of changes

  1. Some changes from x.view('i8') to x.astype('i8', copy=False) to
    support getting i8 values from {ndarray, DatetimeIndex, DatetimeArray}.
    We could revert those if we implemented DatetimeArray.view, but I don't
    think we should. However, this is slower than .view for ndarrays, so
    I'm going to find another way.
  2. Many places where we did series.values instead of series._values.
  3. Moves from PeriodArray / PeriodIndex to a base class. I've annotated these
    with NB: Moved from...
  4. Test skips / xfails. I intended to resolve all of these over the next few
    days.

TODO:

  • Split out DatelikeOps and TimelikeOps changes (#24038)
    • independent tests for DatelikeOps & TimelikeOps methods on arrays.
  • re-clean up the constructors. I may have some cruft from rebasing
  • identify shared code in PeriodArray / PeriodIndex that can be moved up
  • all the xfailed tests
  • identify code that was special casing Datetime-dtyps (like is_extension_type).
  • API: Series[datetimetz].unique: ndarray[object] or DatetimeArray?

Closes #23185
Closes #23932
closes #24465

TomAugspurger added some commits Nov 30, 2018

Squashed commit of the following:
commit c23c9e2
Merge: 50e1aeb dc8d35a
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 09:10:00 2018 -0600

    Merge remote-tracking branch 'upstream/master' into dtype-only

commit 50e1aeb
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 13:00:35 2018 -0600

    PeriodDtype needs freq

commit 10d2c8a
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:30:29 2018 -0600

    refactor construct_from_string

commit c14b45f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:19:29 2018 -0600

    fix unpickling

commit 7ab2a74
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:16:32 2018 -0600

    Remove _coerce_to_dtype

commit 6cc9ce5
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 08:54:35 2018 -0600

    Fixed tz name

commit e0b7b77
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 07:05:04 2018 -0600

    Updates

    * Use pandas_dtype
    * removed cache_readonly

commit ad2723c
Merge: 9e4faf8 d9a037e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 06:39:25 2018 -0600

    Merge remote-tracking branch 'upstream/master' into dtype-only

commit 9e4faf8
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 06:14:05 2018 -0600

    cache readonly

commit 7e6d8ea
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 21:45:21 2018 -0600

    Restore construct_array_type

commit 2fa4bb0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 21:14:10 2018 -0600

    unxfail test, remove caching bit

commit 1ca7fa4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 17:04:40 2018 -0600

    REF/API: DatetimeTZDtype

    * Remove magic constructor from string
    * Remove Caching

    The remaining changes in the DatetimeArray PR will be to

    1. Inherit from ExtensionDtype
    2. Implement construct_array_type
    3. Register
Squashed commit of the following:
commit e7cc2ac
Merge: 740f9e5 30c1290
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 16:16:03 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 740f9e5
Merge: a35399e db8d33e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 07:42:38 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit a35399e
Merge: d9df6bf 6fad5a0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:07:41 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit d9df6bf
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:04:56 2018 -0600

    correct boxing

commit d84cc02
Merge: 2b5fe25 deb7b4d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:02:02 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 2b5fe25
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:38:03 2018 -0600

    BUG: Fixed SparseArray formatter

    We want to fall back to the implementation in formats.

commit ef390fc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:29:42 2018 -0600

    Updates: misc

    * whatsnew
    * docstrings

commit 5c253a4
Merge: 27db397 2946745
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:22:28 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 27db397
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 15 09:05:47 2018 -0600

    simplify formatter

commit fc4279d
Merge: a926dca 8af7637
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 15 08:06:48 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit a926dca
Merge: 62b1e2f e413c49
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 14 15:21:09 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 62b1e2f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 11:19:07 2018 -0600

    remove bytes

commit 2364546
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:11:27 2018 -0600

    fixup! fixup! use repr

commit 439f2f8
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:05:50 2018 -0600

    fixup! use repr

commit 221cee9
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:04:31 2018 -0600

    use repr

commit e5f6976
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:19:01 2018 -0600

    wip

commit ebadf6f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:11:12 2018 -0600

    FutureWarning -> DeprecationWarning

commit 9116930
Merge: 0f4083e 011b79f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:10:53 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 0f4083e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 06:59:20 2018 -0600

    remove periodarray

commit 708dd75
Merge: 1b93bf0 3592a46
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 06:18:59 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 1b93bf0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sun Nov 11 14:52:27 2018 -0600

    update repr tests

commit 5b291d5
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:33:20 2018 -0600

    lint

commit 4d343ea
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:30:55 2018 -0600

    unicode

commit baee6b2
Merge: 5d8d2fc 383d052
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:27:49 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 5d8d2fc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:45:55 2018 -0600

    unicode

commit 2fd3d5d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:45:22 2018 -0600

    unicode

commit ff0c998
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:41:43 2018 -0600

    fixup

commit 5b07906
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:52:16 2018 -0600

    py3 fixup

commit 60e0d02
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:36:39 2018 -0600

    isort

commit 445736d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:30:57 2018 -0600

    unicode, bytes

commit b312fe4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:22:33 2018 -0600

    revert interval

commit d8e7ba4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 21:08:41 2018 -0600

    py2 compat

commit 48e55cc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 20:52:28 2018 -0600

    fixup interval

commit e2b1941
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 16:05:33 2018 -0600

    updates

commit 1635b73
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 15:35:53 2018 -0600

    try this

commit 5a2e1e4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 15:02:39 2018 -0600

    format

commit 193747e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:56:09 2018 -0600

    update docs, type

commit 6e64b7b
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:47:45 2018 -0600

    more cleanup

commit 37638cc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:29:25 2018 -0600

    wip

commit 4e0d91f
Merge: ecfcd72 efd1844
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:36:13 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit ecfcd72
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:33:25 2018 -0600

    clean

commit 1885a97
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:22:29 2018 -0600

    na formatter

commit fef04e6
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:19:58 2018 -0600

    compat

commit 6e76b51
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:16:03 2018 -0600

    test for warning

commit ace62aa
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:07:47 2018 -0600

    Deprecate formatting_values

commit 0fdbfd3
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 09:26:57 2018 -0600

    wip
Squashed commit of the following:
commit f35e8dd
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:12:22 2018 -0600

    extraneous comment

commit ce353af
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:11:30 2018 -0600

    release note

commit 09e07d2
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:00:39 2018 -0600

    PERF: fixup
@jbrockmendel

This comment has been minimized.

Copy link
Member

commented Nov 30, 2018

Thanks for putting this up, I'll take a look. How did you produce those inheritance graphs? Those are neat.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor Author

commented Nov 30, 2018

Thanks for putting this up, I'll take a look. How did you produce those inheritance graphs? Those are neat.

lightly modified http://www.phyast.pitt.edu/~micheles/python/drawMRO.html

Show resolved Hide resolved pandas/core/generic.py Outdated

@TomAugspurger TomAugspurger added this to Design / PRs in DatetimeArray Refactor Dec 1, 2018

TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Dec 1, 2018

REF: Refactor Datetimelike delegation
This is a generalization of PeriodIndex's dispatching to
PeriodArray, without any actual changes yet. This is split
from pandas-dev#24024, where DatetimeIndex and TimedeltaIndex will
implement and inherit from delgates similiar to PeriodDelegateMixin.

jreback added a commit that referenced this pull request Jan 3, 2019

@shashank88 shashank88 referenced this pull request Jan 28, 2019

Closed

Fix broken build #703

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

implement independent parts of pandas-dev#24024 (pandas-dev#24276)
* implement independent parts of pandas-dev#24024

* move monotonic checks up

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

REF: Stop mixing DTA/TDA into DTI/TDI (pandas-dev#24476)
* implement _index_data parts of pandas-dev#24024

* implement _eadata, dispatch arithmetic methods to it

* dont mix DatetimeLikeArrayMixin into DatetimeIndexOpsMixin

* dont inherit TimedeltaIndex from TimedeltaArray

* dont inherit from DatetimeArray

* use ea_passthrough

* remove previously-overriden overridings

* stop double-mixing

* stop over-writing

* handle+test object arrays

* Remove unused import

* flake8 fixup

* edits per comments

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

implement independent parts of pandas-dev#24024 (pandas-dev#24276)
* implement independent parts of pandas-dev#24024

* move monotonic checks up

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

REF: Stop mixing DTA/TDA into DTI/TDI (pandas-dev#24476)
* implement _index_data parts of pandas-dev#24024

* implement _eadata, dispatch arithmetic methods to it

* dont mix DatetimeLikeArrayMixin into DatetimeIndexOpsMixin

* dont inherit TimedeltaIndex from TimedeltaArray

* dont inherit from DatetimeArray

* use ea_passthrough

* remove previously-overriden overridings

* stop double-mixing

* stop over-writing

* handle+test object arrays

* Remove unused import

* flake8 fixup

* edits per comments

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.