BUG: datetime.date slicing doesn't work with Pandas 1.0.0 #3736

seth-p · 2020-01-31T15:39:44Z

The following code used to work before I upgraded Pandas to 1.0.0. I would expect [5] to produce the same result as [4]. I don't know if the failure of datetime.date slicing is (a) expected behavior; (b) a Pandas bug; or (c) an xarray bug due to not being updated to reflect an intended change in Pandas.

Python 3.7.6 | packaged by conda-forge | (default, Jan  7 2020, 22:33:48) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import xarray as xr, pandas as pd, datetime as dt                                                                                                                                                                                                                         

In [2]: x = xr.DataArray([1., 2., 3.], [('foo', pd.date_range('2010-01-01', periods=3))])                                                                                                                                                                                         

In [3]: x                                                                                                                                                                                                                                                                         
Out[3]: 
<xarray.DataArray (foo: 3)>
array([1., 2., 3.])
Coordinates:
  * foo      (foo) datetime64[ns] 2010-01-01 2010-01-02 2010-01-03

In [4]: x.loc['2010-01-02':'2010-01-04']                                                                                                                                                                                                                                          
Out[4]: 
<xarray.DataArray (foo: 2)>
array([2., 3.])
Coordinates:
  * foo      (foo) datetime64[ns] 2010-01-02 2010-01-03

In [5]: x.loc[dt.date(2010, 1, 2):dt.date(2010, 1, 4)]                                                                                                                                                                                                                           
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2010, 1, 4)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)
    714         try:
--> 715             return Index.get_loc(self, key, method, tolerance)
    716         except (KeyError, ValueError, TypeError):

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()

KeyError: datetime.date(2010, 1, 4)

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1262563200000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2010-01-04 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1262563200000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)
    727                     stamp = stamp.tz_localize(self.tz)
--> 728                 return Index.get_loc(self, stamp, method, tolerance)
    729             except KeyError:

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2010-01-04 00:00:00')

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_slice_bound(self, label, side, kind)
   4841         try:
-> 4842             slc = self.get_loc(label)
   4843         except KeyError as err:

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)
    729             except KeyError:
--> 730                 raise KeyError(key)
    731             except ValueError as e:

KeyError: datetime.date(2010, 1, 4)

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-7-d06631d74971> in <module>
----> 1 x.loc[dt.date(2010, 1, 2): dt.date(2010, 1, 4)]

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/dataarray.py in __getitem__(self, key)
    194             labels = indexing.expanded_indexer(key, self.data_array.ndim)
    195             key = dict(zip(self.data_array.dims, labels))
--> 196         return self.data_array.sel(**key)
    197 
    198     def __setitem__(self, key, value) -> None:

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/dataarray.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs)
   1049             method=method,
   1050             tolerance=tolerance,
-> 1051             **indexers_kwargs,
   1052         )
   1053         return self._from_temp_dataset(ds)

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/dataset.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs)
   2012         indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel")
   2013         pos_indexers, new_indexes = remap_label_indexers(
-> 2014             self, indexers=indexers, method=method, tolerance=tolerance
   2015         )
   2016         result = self.isel(indexers=pos_indexers, drop=drop)

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/coordinates.py in remap_label_indexers(obj, indexers, method, tolerance, **indexers_kwargs)
    390 
    391     pos_indexers, new_indexes = indexing.remap_label_indexers(
--> 392         obj, v_indexers, method=method, tolerance=tolerance
    393     )
    394     # attach indexer's coordinate to pos_indexers

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/indexing.py in remap_label_indexers(data_obj, indexers, method, tolerance)
    258             coords_dtype = data_obj.coords[dim].dtype
    259             label = maybe_cast_to_coords_dtype(label, coords_dtype)
--> 260             idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
    261             pos_indexers[dim] = idxr
    262             if new_idx is not None:

~/.conda/envs/build/lib/python3.7/site-packages/xarray/core/indexing.py in convert_label_indexer(index, label, index_name, method, tolerance)
    122             _sanitize_slice_element(label.start),
    123             _sanitize_slice_element(label.stop),
--> 124             _sanitize_slice_element(label.step),
    125         )
    126         if not isinstance(indexer, slice):

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in slice_indexer(self, start, end, step, kind)
    806 
    807         try:
--> 808             return Index.slice_indexer(self, start, end, step, kind=kind)
    809         except KeyError:
    810             # For historical reasons DatetimeIndex by default supports

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in slice_indexer(self, start, end, step, kind)
   4711         slice(1, 3)
   4712         """
-> 4713         start_slice, end_slice = self.slice_locs(start, end, step=step, kind=kind)
   4714 
   4715         # return a slice

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in slice_locs(self, start, end, step, kind)
   4930         end_slice = None
   4931         if end is not None:
-> 4932             end_slice = self.get_slice_bound(end, "right", kind)
   4933         if end_slice is None:
   4934             end_slice = len(self)

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_slice_bound(self, label, side, kind)
   4843         except KeyError as err:
   4844             try:
-> 4845                 return self._searchsorted_monotonic(label, side)
   4846             except ValueError:
   4847                 # raise the original KeyError

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/base.py in _searchsorted_monotonic(self, label, side)
   4794     def _searchsorted_monotonic(self, label, side="left"):
   4795         if self.is_monotonic_increasing:
-> 4796             return self.searchsorted(label, side=side)
   4797         elif self.is_monotonic_decreasing:
   4798             # np.searchsorted expects ascending sort order, have to reverse

~/.conda/envs/build/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in searchsorted(self, value, side, sorter)
    851         elif not isinstance(value, DatetimeArray):
    852             raise TypeError(
--> 853                 "searchsorted requires compatible dtype or scalar, "
    854                 f"not {type(value).__name__}"
    855             )

TypeError: searchsorted requires compatible dtype or scalar, not date

commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jan  7 2020, 22:33:48) [GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.3

xarray: 0.14.1
pandas: 1.0.0
numpy: 1.17.5
scipy: 1.4.1
netCDF4: 1.5.3
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.0.4.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.1
dask: 2.10.1
distributed: 2.10.0
matplotlib: 3.1.2
cartopy: None
seaborn: 0.9.0
numbagg: installed
setuptools: 45.1.0.post20200119
pip: 20.0.2
conda: 4.8.2
pytest: None
IPython: 7.11.1
sphinx: None

The text was updated successfully, but these errors were encountered:

keewis · 2020-01-31T19:13:45Z

that issue also happens with pure pandas:

In [2]: x = pd.DataFrame({"test": [1, 2, 3]}, index=pd.date_range('2010-01-01', periods=3))
   ...: x.loc[dt.date(2010, 1, 2):dt.date(2010, 1, 4)]
TypeError

note that using dt.datetime instead of dt.date works as expected

seth-p · 2020-01-31T20:24:30Z

Pandas bug: pandas-dev/pandas#31501

seth-p · 2020-02-05T21:09:43Z

This is fixed in Pandas 1.0.1.

seth-p mentioned this issue Jan 31, 2020

Can no longer slice DatetimeIndex with datetime.date values outside the index in 1.0.0 pandas-dev/pandas#31501

Closed

seth-p closed this as completed Feb 5, 2020

seth-p mentioned this issue Aug 21, 2020

Indexing a datetime64[ns] coordinate with a scalar datetime.date produces a KeyError #4363

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: datetime.date slicing doesn't work with Pandas 1.0.0 #3736

BUG: datetime.date slicing doesn't work with Pandas 1.0.0 #3736

seth-p commented Jan 31, 2020

keewis commented Jan 31, 2020

seth-p commented Jan 31, 2020 •

edited

Loading

seth-p commented Feb 5, 2020

BUG: datetime.date slicing doesn't work with Pandas 1.0.0 #3736

BUG: datetime.date slicing doesn't work with Pandas 1.0.0 #3736

Comments

seth-p commented Jan 31, 2020

keewis commented Jan 31, 2020

seth-p commented Jan 31, 2020 • edited Loading

seth-p commented Feb 5, 2020

seth-p commented Jan 31, 2020 •

edited

Loading