BUG: Assorted DatetimeIndex bugfixes #24157

jbrockmendel · 2018-12-08T00:09:34Z

Bit of a hodge-podge.

Deprecate time_rule in offsets.generate_range
Improve error messages if NaT gets passed to date_range, or in validate_frequency (closes ERR: DatetimeIndex creation inconsistencies between list and ndarray #11587)
Preserve timezone in DatetimeIndex->CategoricalIndex->DatetimeIndex (closes Broken roundrip DatetimeIndex -> CategoricalIndex -> DatetimeIndex #18664)
Preserve freq when indexing DTI/TDI with Ellipsis (closes DatetimeIndex loses freq information if mask with ellipses #21282)
closes ERR: DatetimeIndex creation inconsistencies between list and ndarray #11587

@TomAugspurger nothing here is urgent, so if this overlaps with DTA it can stay on the backburner.

pep8speaks · 2018-12-08T00:09:49Z

Hello @jbrockmendel! Thanks for submitting the PR.

There are no PEP8 issues in the file pandas/core/arrays/datetimelike.py !
There are no PEP8 issues in the file pandas/core/arrays/datetimes.py !
There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_construction.py !
There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_date_range.py !
There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_indexing.py !
There are no PEP8 issues in the file pandas/tests/indexes/period/test_indexing.py !
There are no PEP8 issues in the file pandas/tests/indexes/timedeltas/test_indexing.py !
There are no PEP8 issues in the file pandas/tests/tseries/offsets/test_offsets.py !
There are no PEP8 issues in the file pandas/tseries/offsets.py !

codecov · 2018-12-08T01:45:39Z

Codecov Report

Merging #24157 into master will decrease coverage by 49.17%.
The diff coverage is 50%.

@@             Coverage Diff             @@
##           master   #24157       +/-   ##
===========================================
- Coverage    92.2%   43.02%   -49.18%     
===========================================
  Files         162      162               
  Lines       51700    51714       +14     
===========================================
- Hits        47670    22251    -25419     
- Misses       4030    29463    +25433

Flag	Coverage Δ
#multiple	`?`
#single	`43.02% <50%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimes.py	`65.01% <25%> (-33.55%)`	⬇️
pandas/tseries/offsets.py	`46.43% <50%> (-50.41%)`	⬇️
pandas/core/arrays/datetimelike.py	`48.98% <83.33%> (-47.38%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/core/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-98.62%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-95.46%)`	⬇️
... and 122 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c911151...06d0a8e. Read the comment docs.

codecov · 2018-12-08T01:45:43Z

Codecov Report

Merging #24157 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24157      +/-   ##
==========================================
- Coverage   92.22%   92.22%   -0.01%     
==========================================
  Files         162      162              
  Lines       51785    51796      +11     
==========================================
+ Hits        47758    47767       +9     
- Misses       4027     4029       +2

Flag	Coverage Δ
#multiple	`90.62% <100%> (-0.01%)`	⬇️
#single	`43.01% <70.58%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimelike.py	`96.41% <100%> (+0.05%)`	⬆️
pandas/tseries/offsets.py	`96.84% <100%> (-0.01%)`	⬇️
pandas/core/arrays/datetimes.py	`98.22% <100%> (-0.35%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01afc42...b927925. Read the comment docs.

jbrockmendel · 2018-12-08T04:13:37Z

Azure fail is in developer.rst, unrelated.

jbrockmendel · 2018-12-08T22:44:16Z

@datapythonista is the azure fail caused by something in this PR? It looks unrelated but re-pushing didn’t fix it.

datapythonista · 2018-12-08T22:57:09Z

master was broken (I merged an old PR that didn't have some linting), but it's been fixed in #24160

If you merge master and push again, the CI should pass now.

jreback

pls add to #6581 for the deprecation

jreback · 2018-12-09T14:35:08Z

pandas/core/arrays/datetimelike.py

@@ -327,6 +327,10 @@ def __getitem__(self, key):
                             "numpy.newaxis (`None`) and integer or boolean "
                             "arrays are valid indices")

+        if key is Ellipsis:


can you make a more explicit comment. This is likely a more general fix (IOW this is likely broken for all __getitem__ on EA)

why don't this just work as is? why is this not a view (and not a copy)?

why don't this just work as is?

as-is DatetimeIndex[...] loses its freq attribute

why is this not a view (and not a copy)?

Since the copy doesn't have deep=True, self.copy() effectively a view?

can you be more explicit in your comment, this is not obvious

Moved to a hopefully clearer spot a few lines down

pandas/core/arrays/datetimelike.py

pandas/core/arrays/datetimes.py

jreback · 2018-12-09T14:37:56Z

pandas/tests/indexes/datetimes/test_construction.py

+        ci = pd.CategoricalIndex(dti)
+        carr = pd.Categorical(dti)
+        cser = pd.Series(ci)
+


can you parametrize

will take a look

Not without adding casting code that makes the verbosity a wash.

Once Categorical(DatetimeArray) works (currently raises bc DatetimeArray doesn't have _constructor) then this test can be parametrized over DatetimeIndex/DatetimeArray

jbrockmendel · 2018-12-09T19:23:24Z

pls add to #6581 for the deprecation

Will do. Actually, is this private enough that a deprecation cycle is unnecessary? If so we should just remove the arg now and move the whole func to arrays.datetimes, since that is the only place it is used

jbrockmendel · 2018-12-11T21:05:55Z

@TomAugspurger does this step on your toes?

TomAugspurger

LGTM.

I think it'll apply cleanly aside from imports.

jreback · 2018-12-12T13:02:00Z

pandas/core/arrays/datetimelike.py

@@ -327,6 +327,10 @@ def __getitem__(self, key):
                             "numpy.newaxis (`None`) and integer or boolean "
                             "arrays are valid indices")

+        if key is Ellipsis:


why don't this just work as is? why is this not a view (and not a copy)?

jreback · 2018-12-13T01:56:16Z

pandas/core/arrays/datetimelike.py

@@ -327,6 +327,10 @@ def __getitem__(self, key):
                             "numpy.newaxis (`None`) and integer or boolean "
                             "arrays are valid indices")

+        if key is Ellipsis:


can you be more explicit in your comment, this is not obvious

pandas/core/arrays/datetimelike.py

pandas/core/arrays/datetimes.py

jreback · 2018-12-13T01:58:01Z

pandas/tseries/offsets.py

@@ -2487,6 +2488,9 @@ def generate_range(start=None, end=None, periods=None,

    """
    if time_rule is not None:


did you need to change any existing tests to account for this?

yes, tests.indexes.datetimes.test_date_range test_generate and test_generate_cday

is this sufficiently internal that we don't need to do a deprecation cycle?

yeah, let's just drop it entirely (fix the tests). I don't think generate_range is exposed, though maybe add in a note in api breaking changes just to cover.

Will do. OK with moving this to arrays.datetimes while we're at it?

jreback · 2018-12-13T13:09:03Z

doc/source/whatsnew/v0.24.0.rst

@@ -1133,7 +1133,7 @@ Deprecations
 - Passing a string alias like ``'datetime64[ns, UTC]'`` as the `unit` parameter to :class:`DatetimeTZDtype` is deprecated. Use :class:`DatetimeTZDtype.construct_from_string` instead (:issue:`23990`).
 - In :meth:`Series.where` with Categorical data, providing an ``other`` that is not present in the categories is deprecated. Convert the categorical to a different dtype or add the ``other`` to the categories first (:issue:`24077`).
 - :meth:`Series.clip_lower`, :meth:`Series.clip_upper`, :meth:`DataFrame.clip_lower` and :meth:`DataFrame.clip_upper` are deprecated and will be removed in a future version. Use ``Series.clip(lower=threshold)``, ``Series.clip(upper=threshold)`` and the equivalent ``DataFrame`` methods (:issue:`24203`)
-
+- Passing a `time_rule` to `pandas.tseries.offsets.generate_range` is deprecated and will raise a ``TypeError`` in a future version.  Pass an ``offset`` instead (:issue:`24157`)


these don't jive with what's at the top of the PR. missing 2 notes?

what about #11587 ?

Does it merit a note? All that is changed is an error message. But yes, this does close 11587, I'll update this above.

pandas/core/arrays/datetimelike.py

jreback · 2018-12-13T13:09:56Z

pandas/core/arrays/datetimelike.py

+                # non-fixed frequencies are not meaningful for timedelta64;
+                #  we retain that error message
+                raise e
+            # GH#11587 if index[0] is NaT _generate_range will raise


this is too cryptic, pls make more readable if you don't actually know what's happening here

jreback · 2018-12-13T13:10:25Z

pandas/core/arrays/datetimes.py

@@ -1652,6 +1654,13 @@ def maybe_convert_dtype(data, copy):
        raise TypeError("Passing PeriodDtype data is invalid.  "
                        "Use `data.to_timestamp()` instead")

+    elif is_categorical_dtype(data):
+        # GH#18664 preserve tz in going DTI->Categorical->DTI
+        # TODO: cases where we need to do another pass through this func,


why is the TODO case here, what is an example? do you have an xfailed test?

example is Categorical(TimedeltaIndex) where we should do another pass through maybe_convert_dtype in order to issue the FutureWarning. I'm assuming there are other corner cases here that will need to be caught/tested.

jreback

small comments, ping on green.

jreback · 2018-12-13T18:58:16Z

doc/source/whatsnew/v0.24.0.rst

@@ -380,6 +380,7 @@ Backwards incompatible API changes
 - ``max_rows`` and ``max_cols`` parameters removed from :class:`HTMLFormatter` since truncation is handled by :class:`DataFrameFormatter` (:issue:`23818`)
 - :meth:`read_csv` will now raise a ``ValueError`` if a column with missing values is declared as having dtype ``bool`` (:issue:`20591`)
 - The column order of the resultant :class:`DataFrame` from :meth:`MultiIndex.to_frame` is now guaranteed to match the :attr:`MultiIndex.names` order. (:issue:`22420`)
+- :func:`pd.offsets.generate_range` argument ``time_rule`` has been removed; use ``offset`` instead (:issue:`24157`)


this probably won't render as its not in the api.rst but ok

jreback · 2018-12-13T18:58:59Z

doc/source/whatsnew/v0.24.0.rst

@@ -1310,6 +1310,8 @@ Datetimelike
 - Bug in :class:`Index` where calling ``np.array(dtindex, dtype=object)`` on a timezone-naive :class:`DatetimeIndex` would return an array of ``datetime`` objects instead of :class:`Timestamp` objects, potentially losing nanosecond portions of the timestamps (:issue:`23524`)
 - Bug in :class:`Categorical.__setitem__` not allowing setting with another ``Categorical`` when both are undordered and have the same categories, but in a different order (:issue:`24142`)
 - Bug in :func:`date_range` where using dates with millisecond resolution or higher could return incorrect values or the wrong number of values in the index (:issue:`24110`)
+- Bug in :class:`DatetimeIndex` where constructing a :class:`DatetimeIndex` from a :class:`Categorical` or :class:`CategoricalIndex` would incorrectly drop timezone information (:issue:`18664`)


where is the note for #11587?

Just pushed, should be just below this line.

jreback · 2018-12-13T19:40:55Z

pandas/tseries/offsets.py

@@ -2,6 +2,7 @@
 from datetime import date, datetime, timedelta
 import functools
 import operator
+import warnings


prob going to fail the build as this is not needed

jreback · 2018-12-13T19:42:02Z

linting issue. ping on green.

jbrockmendel · 2018-12-13T22:22:30Z

Ping

jreback · 2018-12-14T14:53:45Z

thanks!

jbrockmendel added 11 commits December 7, 2018 14:25

Deprecate time_rule

092b3b9

cleanup

fbcc04b

fix test

ac52857

Fix indexing with ellipsis, GH#21282

121c373

isort tests.arithmetic, tests.dtypes

ef66bb9

raise on date_range with NaT for either start or end

91738d3

revert noncentral changes

bb0d065

more useful error messages on validate_frequency failure

ff3a5c0

Fix DTI-->Categorical -->DTI roundtrip

e54159d

extend test to DTA

44e0126

comment

12e0f4e

jbrockmendel added 2 commits December 7, 2018 16:09

update GH reference

28bf2de

make TDI catch nanos valuerror

06d0a8e

fix error message, hopefully

81c7d0f

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

97f976f

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

9e55d97

jreback added the Datetime Datetime data dtype label Dec 9, 2018

jreback requested changes Dec 9, 2018

View reviewed changes

jreback added the Deprecate Functionality to remove in pandas label Dec 9, 2018

jbrockmendel added 4 commits December 9, 2018 07:57

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

c7f280f

improve comments

af303c7

use take

8ece686

whitespace

30f01a0

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

9617b85

jsexauer mentioned this pull request Dec 11, 2018

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

TomAugspurger approved these changes Dec 11, 2018

View reviewed changes

jreback requested changes Dec 12, 2018

View reviewed changes

jreback requested changes Dec 13, 2018

View reviewed changes

jbrockmendel added 4 commits December 12, 2018 20:35

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

3731098

do ellipsis check later

df05c88

Avoid AssertionError

8f39b23

remove resolved comment

e958ce6

jreback requested changes Dec 13, 2018

View reviewed changes

jbrockmendel added 5 commits December 13, 2018 09:51

Merge branch 'master' of https://github.com/pandas-dev/pandas into deets

97cb6b3

more whatsnew

36d37e4

comment harder

3fc1c19

comment harder

88f7094

remove time_rule entirely

e178bb9

jreback added this to the 0.24.0 milestone Dec 13, 2018

jreback requested changes Dec 13, 2018

View reviewed changes

whatsnew

50f6b7e

jreback reviewed Dec 13, 2018

View reviewed changes

jreback approved these changes Dec 13, 2018

View reviewed changes

flake8 fixup

b927925

jreback merged commit a1cf3f6 into pandas-dev:master Dec 14, 2018

jbrockmendel deleted the deets branch December 14, 2018 16:12

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG: Assorted DatetimeIndex bugfixes (pandas-dev#24157)

948eef6

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG: Assorted DatetimeIndex bugfixes (pandas-dev#24157)

56f15d0

jreback mentioned this pull request Dec 3, 2019

DEPR: deprecations log for removed issues #13777

Closed

		@@ -2487,6 +2488,9 @@ def generate_range(start=None, end=None, periods=None,

		"""
		if time_rule is not None:

BUG: Assorted DatetimeIndex bugfixes #24157

BUG: Assorted DatetimeIndex bugfixes #24157

Conversation

jbrockmendel commented Dec 8, 2018 • edited Loading

pep8speaks commented Dec 8, 2018

codecov bot commented Dec 8, 2018

Codecov Report

codecov bot commented Dec 8, 2018 • edited Loading

Codecov Report

jbrockmendel commented Dec 8, 2018

jbrockmendel commented Dec 8, 2018

datapythonista commented Dec 8, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Dec 9, 2018

jbrockmendel commented Dec 11, 2018

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 13, 2018

jbrockmendel commented Dec 13, 2018

jreback commented Dec 14, 2018

jbrockmendel commented Dec 8, 2018 •

edited

Loading

codecov bot commented Dec 8, 2018 •

edited

Loading