Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0

[davidia]

Code Sample

import pandas as pd
import datetime as dt
s = pd.Series([0,1],pd.DatetimeIndex([dt.date(2019,1,1),dt.date(2019,1,2)]))
s[dt.date(2019,1,1):]  # works 
s[dt.date(2018,1,1):]  # error

Problem description

As of 1.0.0 you can no longer slice a DatetimeIndex with dt.date values outside the index.

Error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    714         try:
--> 715             return Index.get_loc(self, key, method, tolerance)
    716         except (KeyError, ValueError, TypeError):

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1514764800000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2018-01-01 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1514764800000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    727                     stamp = stamp.tz_localize(self.tz)
--> 728                 return Index.get_loc(self, stamp, method, tolerance)
    729             except KeyError:

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2018-01-01 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_slice_bound(self, label, side, kind)
   4841         try:
-> 4842             slc = self.get_loc(label)
   4843         except KeyError as err:

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in get_loc(self, key, method, tolerance)
    729             except KeyError:
--> 730                 raise KeyError(key)
    731             except ValueError as e:

KeyError: datetime.date(2018, 1, 1)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-47-a23c04333556> in <module>
      3 s = pd.Series([0,1],pd.DatetimeIndex([dt.date(2019,1,1),dt.date(2019,1,2)]))
      4 s[dt.date(2019,1,1):]  # works
----> 5 s[dt.date(2018,1,1):]  # error

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
    908             key = check_bool_indexer(self.index, key)
    909 
--> 910         return self._get_with(key)
    911 
    912     def _get_with(self, key):

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in _get_with(self, key)
    913         # other: fancy integer or otherwise
    914         if isinstance(key, slice):
--> 915             return self._slice(key)
    916         elif isinstance(key, ABCDataFrame):
    917             raise TypeError(

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\series.py in _slice(self, slobj, axis, kind)
    863 
    864     def _slice(self, slobj: slice, axis: int = 0, kind=None):
--> 865         slobj = self.index._convert_slice_indexer(slobj, kind=kind or "getitem")
    866         return self._get_values(slobj)
    867 

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in _convert_slice_indexer(self, key, kind)
   2961             indexer = key
   2962         else:
-> 2963             indexer = self.slice_indexer(start, stop, step, kind=kind)
   2964 
   2965         return indexer

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in slice_indexer(self, start, end, step, kind)
    806 
    807         try:
--> 808             return Index.slice_indexer(self, start, end, step, kind=kind)
    809         except KeyError:
    810             # For historical reasons DatetimeIndex by default supports

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in slice_indexer(self, start, end, step, kind)
   4711         slice(1, 3)
   4712         """
-> 4713         start_slice, end_slice = self.slice_locs(start, end, step=step, kind=kind)
   4714 
   4715         # return a slice

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in slice_locs(self, start, end, step, kind)
   4924         start_slice = None
   4925         if start is not None:
-> 4926             start_slice = self.get_slice_bound(start, "left", kind)
   4927         if start_slice is None:
   4928             start_slice = 0

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in get_slice_bound(self, label, side, kind)
   4843         except KeyError as err:
   4844             try:
-> 4845                 return self._searchsorted_monotonic(label, side)
   4846             except ValueError:
   4847                 # raise the original KeyError

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\base.py in _searchsorted_monotonic(self, label, side)
   4794     def _searchsorted_monotonic(self, label, side="left"):
   4795         if self.is_monotonic_increasing:
-> 4796             return self.searchsorted(label, side=side)
   4797         elif self.is_monotonic_decreasing:
   4798             # np.searchsorted expects ascending sort order, have to reverse

C:\ProgramData\Miniconda3.7\lib\site-packages\pandas\core\indexes\datetimes.py in searchsorted(self, value, side, sorter)
    851         elif not isinstance(value, DatetimeArray):
    852             raise TypeError(
--> 853                 "searchsorted requires compatible dtype or scalar, "
    854                 f"not {type(value).__name__}"
    855             )

TypeError: searchsorted requires compatible dtype or scalar, not date

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.1.final.0
python-bits : 64
OS : Windows
OS-release : 7
machine : AMD64
processor : Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.0
numpy : 1.17.3
pytz : 2018.9
dateutil : 2.8.1
pip : 20.0.2
setuptools : 41.6.0.post20191030
Cython : 0.29.5
pytest : 5.1.2
hypothesis : None
sphinx : None
blosc : 1.7.0
feather : None
xlsxwriter : None
lxml.etree : 4.3.2
html5lib : None
pymysql : None
psycopg2 : 2.7.7 (dt dec pq3 ext lo64)
jinja2 : 2.10.3
IPython : 7.5.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.3.2
matplotlib : 3.0.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.14.0
pytables : None
pytest : 5.1.2
pyxlsb : None
s3fs : None
scipy : 1.2.1
sqlalchemy : 1.2.17
tables : None
tabulate : None
xarray : 0.12.1
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
numba : 0.45.1

[jorisvandenbossche]

@davidia Thanks for the report! That's indeed a regression, tagged it for the 1.0.1 milestone.

[davidia]

My pleasure, thanks for all the awesome work!

[seth-p]

Note that slicing with datetime objects does work. See pydata/xarray#3736 (comment).

[TomAugspurger]

Looking into this now.