API: 'D' and offset.Day always represents calendar day

[mroeschke]

• closes API: Make 'D' & offsets.Day always operate as calendar day instead of 24 Hour #22864
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[pep8speaks]

Hello @mroeschke! Thanks for updating the PR.

• There are no PEP8 issues in the file pandas/core/arrays/datetimes.py !

• There are no PEP8 issues in the file pandas/core/indexes/datetimes.py !

• There are no PEP8 issues in the file pandas/core/indexes/interval.py !

• There are no PEP8 issues in the file pandas/core/indexes/period.py !

• There are no PEP8 issues in the file pandas/core/resample.py !

• There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_date_range.py !

• There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_indexing.py !

• There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_timezones.py !

• There are no PEP8 issues in the file pandas/tests/indexes/timedeltas/test_timedelta_range.py !

• There are no PEP8 issues in the file pandas/tests/resample/test_datetime_index.py !

• There are no PEP8 issues in the file pandas/tests/resample/test_period_index.py !

• There are no PEP8 issues in the file pandas/tests/scalar/timedelta/test_arithmetic.py !

• There are no PEP8 issues in the file pandas/tests/series/test_timezones.py !

• In the file pandas/tests/tseries/offsets/test_offsets.py, following are the PEP8 issues :

Line 3229:66: W292 no newline at end of file

• There are no PEP8 issues in the file pandas/tests/tseries/offsets/test_ticks.py !

• There are no PEP8 issues in the file pandas/tseries/offsets.py !

Comment last updated on December 13, 2018 at 23:43 Hours UTC

[jreback]

u can add this PR number as well

[jorisvandenbossche]

@mroeschke I will try to give this a look (finally, sorry about that). Can you give a short status update? Are there still problems you are encountering, .. ?

[jorisvandenbossche]

One of the example I used in the discussion is now errorring:

In [1]: idx = pd.date_range("2016-10-30", freq='H', periods=4*24, tz='Europe/Helsinki')                                                                                         

In [2]: pd.Series(index=idx).resample('D').count()                                                                                                                              
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-aaf4c03a8f51> in <module>
----> 1 pd.Series(index=idx).resample('D').count()

~/scipy/pandas/pandas/core/resample.py in f(self, _method)
    864 for method in ['count']:
    865     def f(self, _method=method):
--> 866         return self._downsample(_method)
    867     f.__doc__ = getattr(GroupBy, method).__doc__
    868     setattr(Resampler, method, f)

~/scipy/pandas/pandas/core/resample.py in _downsample(self, how, **kwargs)
    990         **kwargs : kw args passed to how function
    991         """
--> 992         self._set_binner()
    993         how = self._is_cython_func(how) or how
    994         ax = self.ax

~/scipy/pandas/pandas/core/resample.py in _set_binner(self)
    170         """
    171         if self.binner is None:
--> 172             self.binner, self.grouper = self._get_binner()
    173 
    174     def _get_binner(self):

~/scipy/pandas/pandas/core/resample.py in _get_binner(self)
    178         """
    179 
--> 180         binner, bins, binlabels = self._get_binner_for_time()
    181         bin_grouper = BinGrouper(bins, binlabels, indexer=self.groupby.indexer)
    182         return binner, bin_grouper

~/scipy/pandas/pandas/core/resample.py in _get_binner_for_time(self)
    979         if self.kind == 'period':
    980             return self.groupby._get_time_period_bins(self.ax)
--> 981         return self.groupby._get_time_bins(self.ax)
    982 
    983     def _downsample(self, how, **kwargs):

~/scipy/pandas/pandas/core/resample.py in _get_time_bins(self, ax)
   1428         # general version, knowing nothing about relative frequencies
   1429         bins = lib.generate_bins_dt64(
-> 1430             ax_values, bin_edges, self.closed, hasnans=ax.hasnans)
   1431 
   1432         if self.closed == 'right':

~/scipy/pandas/pandas/_libs/lib.pyx in pandas._libs.lib.generate_bins_dt64()

ValueError: Values falls after last bin

[jorisvandenbossche]

How do you see the way forward here with _Day. Is the idea that Day will be replaced by that, after deprecating the behaviours that are different?
(but I don't see any deprecations right now?)

[jorisvandenbossche]

Can you give some examples here?

For example, with the following I don't see a change or warning:

In [11]: pd.Timestamp("2012-10-28", tz='Europe/Brussels') + pd.offsets.Day()                                                                                                    
Out[11]: Timestamp('2012-10-28 23:00:00+0100', tz='Europe/Brussels')

[mroeschke]

Sorry, I have been primarily working on reverting things and getting the test to pass again by rolling back CalendarDay.

TODO
Add all the depreciation warnings (I anticipate there being quite a few of these). This needs to be included in the following:

• Day tick arithmetic with other Ticks, Timedeltas, and DatetimeTZ
• DatetimeIndex.shift (tz-aware only)

Migration
The plan is for _Day (formerly CalendarDay) to just replace Day once the prior Day behavior is replaced.

Concern
During the last dev chat, there was interest for 'D' to be compatible freq argument/offset with both Timedelta and Datetime. I don't see a clear way to make this possible without adding a lot of monkeypatching.

Example: timedelta_range(..., freq='D'); to_offset('D') will return _Day in the future and this offset will need to increment a Timedelta, but _Day + Timedelta is an invalid operation.

[mroeschke]

Closing in favor of #24330 for now (reverts behavior to pre-CalendarDay). We can revisit this after the 0.24.0 release.