astype(object) with dtype datetime64 does wacky things when value is NaT

[lexual]

Similar to #3416, failing in 0.11

Calling astype(object) when the dtype is datetime64 and there are NaT values, sets them to 2262-04-10 00:12:43.145224

I would have thought they should be set to None.

See attached test case.

[lexual]

Apparently you can't upload files to github issues?? See below:

# test case for pandas bug with datetime64 conversions.
# NaT is getting converted to
    # 2262-04-10 00:12:43 WTF?!?
# tested against v0.10.1

import pandas as pd
import numpy as np
import datetime


dates = [
    np.datetime64(datetime.date(2013, 1, 1)),
    np.datetime64(datetime.date(2013, 1, 2)),
    np.datetime64(datetime.date(2013, 1, 3)),
]
s = pd.Series(dates, dtype='datetime64')
s.ix[0] = np.nan
print s
print s.dtype
s_o = s.astype(object)
print s_o.dtype
print s_o
print
# 2262-04-10 00:12:43 WTF?!?
print type(s_o.ix[0]), s_o.ix[0]

[jreback]

this was fixed for 0.11

[jreback]

FYI either put code inline or u can use a gist

[jreback]

http://pandas.pydata.org/pandas-docs/dev/whatsnew.html

[jreback]

actually this is a buglet (not the issue you are talking about, that is ok in 0.11, but an odd numpy related one). what version of numpy?

[jreback]

For a workaround - don't wrap in np.datetime64's

In [48]: pd.Series([ datetime.datetime(2013,1,1), datetime.datetime(2013,1,2) ])
Out[48]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
dtype: datetime64[ns]

[lexual]

I see this bug with both numpy 1.6.1 & numpy 1.7.1.

This is a contrived testcase to illustrate my problem, not actually how I came across the bug in my codebase.

Thanks for looking into it @jreback ;)

[jreback]

@lexual will get to this with a slew of other np.datetime64 issues (essentially working around numpy bugs, I thought np 1.7.1 should allow this to work, but ok...will fix for 0.12

[jreback]

@lexual this should be ok in master...

[jreback]

@lexual

Right now we don't convert types like this (due to various numpy bugs)

pd.Series(dates, dtype='datetime64')
TypeError: cannot convert datetimelike to dtype [datetime64[us]]

And the rest works as expected (the buglet was that the np.datetime64 were not dealt with correct)

In [5]: pd.Series(dates)
Out[5]: 
0   2013-01-01 00:00:00
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
dtype: datetime64[ns]

In [6]: s = pd.Series(dates)

In [7]: s.ix[0] = np.nan

In [8]: s
Out[8]: 
0                   NaT
1   2013-01-02 00:00:00
2   2013-01-03 00:00:00
dtype: datetime64[ns]

In [9]: s.dtype
Out[9]: dtype('datetime64[ns]')

In [10]: s.astype(object)
Out[10]: 
0                    NaN
1    2013-01-02 00:00:00
2    2013-01-03 00:00:00
dtype: object

closing unless you have further comments