Series replace values using timestamps in a dict

[MichaelWS]

I was working with financial data and was replacing dates that were not valid trading days. It seems that replace does not work on any dict between two timestamps. Here is an example. This simply throws an exception on 0.12.

import pandas.io.data as web
ibm = web.DataReader("IBM", "yahoo", "1993/01/01").reset_index()
next_date_dict = dict(zip(ibm["Date"], ibm["Date"].shift()))
ibm_next_date = ibm["Date"].replace(next_date_dict)
print (ibm_next_date == ibm["Date"]).all()

in 0.12 there is an exception:
https://gist.github.com/MichaelWS/8186689

[jreback]

this works in 0.13/master. I recall fixing this but not of the exact issue.

[MichaelWS]

Jeff,
i installed from master before posting. Are you getting replacements on your side?
It should print out false.

[jreback]

I take that back, this is partially broken; from a dict does not work (but a single replace does)
(this is only with timestamp)

In [1]: s = pd.Series(pd.date_range('20130101',periods=5))

In [2]: s
Out[2]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
3   2013-01-04 00:00:00
4   2013-01-05 00:00:00
dtype: datetime64[ns]

In [3]: s.replace({ pd.Timestamp('20130103') : pd.Timestamp('20120101') })
Out[3]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
3   2013-01-04 00:00:00
4   2013-01-05 00:00:00
dtype: datetime64[ns]

In [4]: s.replace(pd.Timestamp('20130103'),pd.Timestamp('20120101'))
Out[4]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2012-01-01 00:00:00
3   2013-01-04 00:00:00
4   2013-01-05 00:00:00
dtype: datetime64[ns]

You can 'force' it to work by first casting to 'object' (so this is a bug)

In [5]: s = pd.Series(pd.date_range('20130101',periods=5))

In [6]: s
Out[6]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
3   2013-01-04 00:00:00
4   2013-01-05 00:00:00
dtype: datetime64[ns]

In [7]: s.astype(object).replace({ pd.Timestamp('20130103') : pd.Timestamp('20120101') })
Out[7]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2012-01-01 00:00:00
3   2013-01-04 00:00:00
4   2013-01-05 00:00:00
dtype: datetime64[ns]

[jreback]

@cpcloud if you have a chance

[jreback]

https://github.com/pydata/pandas/blob/master/pandas/core/internals.py#L2325

needs something like:

value = getattr(value,'asm8',value)

before the comparison

[MichaelWS]

Thanks jeback. I will force it for now.

[jreback]

gr8 (you could also patch in your source code)....

can you do a PR to fix (with some tests?)

[MichaelWS]

just did. the getattr line was all that was needed for the other list.

return values == getattr(s, 'asm8', s)

[jreback]

closed by #5806