Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty Series[datetime64[ns, tz]].reindex fails #20869

Closed
TomAugspurger opened this issue Apr 29, 2018 · 3 comments

Comments

Projects
None yet
5 participants
@TomAugspurger
Copy link
Contributor

commented Apr 29, 2018

TZ-naive is fine.

In [41]: pd.Series(dtype='datetime64[ns]').reindex([0, 1])
Out[41]:
0   NaT
1   NaT
dtype: datetime64[ns]

In [42]: pd.Series(dtype='datetime64[ns, UTC]').reindex([0, 1])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-42-a882503115f7> in <module>()
----> 1 pd.Series(dtype='datetime64[ns, UTC]').reindex([0, 1])

~/sandbox/pandas/pandas/core/series.py in reindex(self, index, **kwargs)
   3309     @Appender(generic._shared_docs['reindex'] % _shared_doc_kwargs)
   3310     def reindex(self, index=None, **kwargs):
-> 3311         return super(Series, self).reindex(index=index, **kwargs)
   3312
   3313     def drop(self, labels=None, axis=0, index=None, columns=None,

~/sandbox/pandas/pandas/core/generic.py in reindex(self, *args, **kwargs)
   3691         # perform the reindex on the axes
   3692         return self._reindex_axes(axes, level, limit, tolerance, method,
-> 3693                                   fill_value, copy).__finalize__(self)
   3694
   3695     def _reindex_axes(self, axes, level, limit, tolerance, method, fill_value,

~/sandbox/pandas/pandas/core/generic.py in _reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
   3709             obj = obj._reindex_with_indexers({axis: [new_index, indexer]},
   3710                                              fill_value=fill_value,
-> 3711                                              copy=copy, allow_dups=False)
   3712
   3713         return obj

~/sandbox/pandas/pandas/core/generic.py in _reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups)
   3812                                                 fill_value=fill_value,
   3813                                                 allow_dups=allow_dups,
-> 3814                                                 copy=copy)
   3815
   3816         if copy and new_data is self._data:

~/sandbox/pandas/pandas/core/internals.py in reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy)
   4419         if axis == 0:
   4420             new_blocks = self._slice_take_blocks_ax0(indexer,
-> 4421                                                      fill_tuple=(fill_value,))
   4422         else:
   4423             new_blocks = [blk.take_nd(indexer, axis=axis, fill_tuple=(

~/sandbox/pandas/pandas/core/internals.py in _slice_take_blocks_ax0(self, slice_or_indexer, fill_tuple)
   4458                 return [blk.take_nd(slobj, axis=0,
   4459                                     new_mgr_locs=slice(0, sllen),
-> 4460                                     fill_tuple=fill_tuple)]
   4461
   4462         if sl_type in ('slice', 'mask'):

~/sandbox/pandas/pandas/core/internals.py in take_nd(self, indexer, axis, new_mgr_locs, fill_tuple)
   1256             fill_value = fill_tuple[0]
   1257             new_values = algos.take_nd(values, indexer, axis=axis,
-> 1258                                        allow_fill=True, fill_value=fill_value)
   1259
   1260         if new_mgr_locs is None:

~/sandbox/pandas/pandas/core/algorithms.py in take_nd(arr, indexer, axis, out, fill_value, mask_info, allow_fill)
   1580         return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
   1581     elif is_datetimetz(arr):
-> 1582         return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
   1583     elif is_interval_dtype(arr):
   1584         return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)

~/sandbox/pandas/pandas/core/indexes/datetimelike.py in take(self, indices, axis, allow_fill, fill_value, **kwargs)
    497                                            allow_fill=allow_fill,
    498                                            fill_value=fill_value,
--> 499                                            na_value=iNaT)
    500
    501         # keep freq in PeriodIndex, reset otherwise

~/sandbox/pandas/pandas/core/indexes/base.py in _assert_take_fillable(self, values, indices, allow_fill, fill_value, na_value)
   2178                        'all indices must be >= -1')
   2179                 raise ValueError(msg)
-> 2180             taken = values.take(indices)
   2181             mask = indices == -1
   2182             if mask.any():

IndexError: cannot do a non-empty take from an empty axes.

@TomAugspurger TomAugspurger changed the title Series[datetime64[ns, tz]].reindex fails Empty Series[datetime64[ns, tz]].reindex fails Apr 29, 2018

@WillAyd

This comment has been minimized.

Copy link
Member

commented Apr 30, 2018

IIUC datetime64 has a customized algos take helper that accounts for NA values. datetime64-tz does not have this and ends up hitting the code linked below in base (bottom of your traceback):

taken = values.take(indices)

I suppose an easy fix is to resize values to match the size of indices if the former is empty but the latter is not. Not sure what edge cases might fall in between.

Happy to submit a PR if you think it's a good approach

@sschuldenzucker

This comment has been minimized.

Copy link

commented Jul 19, 2018

This seems to be fixed in 0.23.1. The above code now doesn't crash, but yields

0   NaT
1   NaT
dtype: datetime64[ns, UTC]

as expected.

Btw, I got here because I saw a crash in 0.20.1 in the same source line mentioned above when trying to print (format) an all-NaT categorical datetime series. This is fixed in 0.23.1, too.

@WillAyd

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

Thanks @sschuldenzucker - this may have been fixed via some of the work @jbrockmendel has been doing. If you are looking to make a contribution then we'll gladly accept a test to ensure this doesn't break in the future

@mroeschke mroeschke referenced this issue Nov 4, 2018

Merged

TST: Empty Series.reindex with tz-aware dtype #23480

4 of 4 tasks complete

@jreback jreback added this to the 0.24.0 milestone Nov 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.