BUG: nested dict construction timedelta

[chris-b1]

Related to #10160 / PR pydata#10269

In [2]: from datetime import timedelta
...: from pandas import * 
...: td_as_int = [1, 2, 3, 4]
...: 
...: def create_data(constructor):
...:     return dict((i, {constructor(s): 2*i}) for i, s in enumerate(td_as_int))
...: 
...: data_timedelta64 = create_data(lambda x: np.timedelta64(x, 'D'))
...: data_timedelta = create_data(lambda x: timedelta(days=x))
...: 
...: result_timedelta64 = DataFrame(data_timedelta64)
...: result_timedelta = DataFrame(data_timedelta)
...: 

In [3]: result_timedelta64
Out[3]: 
        0   1   2   3
1 days   0 NaN NaN NaN
2 days NaN   2 NaN NaN
3 days NaN NaN   4 NaN
4 days NaN NaN NaN   6

In [4]: result_timedelta
Out[4]: 
        0   1   2   3
1 days NaN NaN NaN NaN
2 days NaN NaN NaN NaN
3 days NaN NaN NaN NaN
4 days NaN NaN NaN NaN

This seems to happen because Timedelta doesn't have hash equality with datetime.timedelta It could easily be worked around here, but I was wondering if there's any reason not to make Timedelta work like Timestamp and match hashes if above a certain resolution?

[lmjohns3]

Are you sure you meant to reference #11115 (Interpolate erroneously fills NaNs at beginning of sequence when limit_direction == 'both') on this issue?

[chris-b1]

Sorry, wrong issue. Edited above, thanks!

[jreback]

@chris-b1 yep this prob need just a change to core.common._maybe_box_datetimelike for datetime.datetime and datetime.timedelta in the isinstance

[chris-b1]

FYI, this actually works for datetime.datetime because the hash matches pd.Timestamp (covered by tests in that PR), but I'll go ahead and change it as suggested.

[jreback]

what exactly is not matching for the hash of a Timedelta (prob don't have any tests for this).

[chris-b1]

right now the Timedelta hash is just based on the nanosecond value, so it won't ever match a datetime.timedelta

In [42]: from datetime import timedelta, datetime

In [43]: d = {timedelta(days=1): 1, datetime(2013,1,1): 2}

In [44]: d[pd.Timestamp('2013-1-1')]
Out[44]: 2

In [45]: d[pd.Timedelta(1, 'D')]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-45-927af64de84f> in <module>()
----> 1 d[pd.Timedelta(1, 'D')]

KeyError: Timedelta('1 days 00:00:00')

I tried changing the hash definition as below, patterned after _Timestamp, which works, but wasn't sure about the perf impact of _ensure_components()

def __hash__(_Timedelta self):
    self._ensure_components()
    if self._ns > 0:
        return hash(self.value)
    else:
        return timedelta.__hash__(self)

[jreback]

hmm, _ensure_components can be cdef inline anyhow. its not called by python I don't think.

try that. (hash change looks ok). But I think we also need the change to _maybe_box_datetimelike

[shoyer]

We should definitely match the timedelta hash on our subclass. I would consider that also a bug.