BUG: ValueError raised by cummin/cummax when datetime64 Series contains NaT. #8966
Conversation
| @@ -4109,13 +4109,16 @@ def func(self, axis=None, dtype=None, out=None, skipna=True, | |||
| axis = self._get_axis_number(axis) | |||
|
|
|||
| y = _values_from_object(self).copy() | |||
| if not issubclass(y.dtype.type, (np.integer, np.bool_)): | |||
|
|
|||
| if skipna and com.is_datetime64_dtype(y): | |||
There was a problem hiding this comment.
you can handle timedelta64[ns] here as well; just put pd.tslib.iNaT directly as the value (it will work! for both datetime64[ns] and timedelta64[ns])
There was a problem hiding this comment.
but I think this is a bit more complicated, e.g. for cummax this will work, but not cummin
see nanops/nanmin for what you can do (e.g. replace the values with the largest intmax value before cumming, then replace the result at the end)
jreback
added
Bug
Missing-data
unutbu
force-pushed
the
cummin-NaT
branch
2 times, most recently
from
71cc1b7
to
1394b3a
Compare
|
@jreback: Sorry, I don't understand what part of nanops I should be using, or what makes cummax more complicated. Could you give me some more guidance? |
|
|
| expected = pd.Series(pd.to_datetime( | ||
| ['NaT', '2000-1-2', '2000-1-2', '2000-1-1', '2000-1-1', '2000-1-1'])) | ||
| result = s.cummin(skipna=False) | ||
| self.assert_series_equal(expected, result) |
There was a problem hiding this comment.
@jreback: s.cummin(skipna=False) is passing this test. Can you give an example where it fails?
| else: | ||
| result = accum_func(y, axis) | ||
|
|
||
| d = self._construct_axes_dict() | ||
| d['copy'] = False | ||
| s = self._constructor(result, **d).__finalize__(self) |
There was a problem hiding this comment.
this looks like an extra line here?
There was a problem hiding this comment.
@jreback: Yes, thanks for the correction.
…contains NaT. (:issue:`8965`)
|
|
||
| if skipna and issubclass(y.dtype.type, | ||
| (np.datetime64, np.timedelta64)): | ||
| result = accum_func(y, axis) |
There was a problem hiding this comment.
I think calling accum_func with NaTs present is okay because the NaTs are neither greater than nor less than any other datetime64:
In [6]: pd.NaT < np.datetime64(('2010-01-01'))
Out[6]: False
In [7]: np.datetime64(('2010-01-01')) < pd.NaT
Out[7]: False
|
thanks! |
closes #8965