BUG: to_csv issue when multiple datetime64[ns] columns with some NaT

[jreback]

http://stackoverflow.com/questions/16175761/pandas-to-csv-drops-values/16178032?noredirect=1#comment23125823_16178032

[jreback]

@y-p did we test multiple datetime64[ns] columns with NaT, I think getting confused somewhere....

[jreback]

it works if rows < 25000....bug! darn

Yeah, we really tested the hell out of it.... you can tell him them to use

In [11]: df.to_csv('a.csv',engine='python')

which we prepared with some foresight...

[jreback]

thanks.....see my PR to fix this....but I don't think we have a test case for this (e.g. > 25k rows, multiple datetime columns, some with NaT)

keep this open, I'll add a test when I get a chance.

[jreback]

thnxs

[jreback]

ok...user reports that it works....(the fix)

@jreback , can't repro with:

from pandas import NaT
def make_dtnat_arr(n,nnat=None):
    if nnat is None:
        nnat= int(n*0.1) # 10%
    s=list(date_range('2000',freq='5min',periods=n))
    if nnat:
        for i in np.random.randint(0,len(s),nnat):
            s[i] = NaT
        i = np.random.randint(100)
        s[-i] = NaT
        s[i] = NaT
    return s
N=30000
s1=make_dtnat_arr(N)
s2=make_dtnat_arr(N)
s3=make_dtnat_arr(N,0)
df=DataFrame(dict(a=s1,b=s2,c=s3))
df.to_csv('/tmp/1.csv')
s=DataFrame.from_csv('/tmp/1.csv')
s=s.convert_objects('coerce')
In [54]: df.shape == s.shape
Out[54]: True

have you got a repro snippet?

[jreback]

must e something pathological about 30000, try 35000 or >....fails

nope.

[jreback]

your function was failing, change len(s1) -> len(s) (inside the function)

already did, i'm not missing lines on readback.

[jreback]

are you using the fix?

In [12]: df
Out[12]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 35000 entries, 0 to 34999
Data columns (total 3 columns):
a    31657  non-null values
b    31645  non-null values
c    35000  non-null values
dtypes: datetime64[ns](3)

In [13]: s
Out[13]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 35000 entries, 0 to 34999
Data columns (total 3 columns):
a    31657  non-null values
b    30295  non-null values
c    29999  non-null values
dtypes: datetime64[ns](3)