BUG: to_csv issue when multiple datetime64[ns] columns with some NaT

[jreback]

http://stackoverflow.com/questions/16175761/pandas-to-csv-drops-values/16178032?noredirect=1#comment23125823_16178032

[jreback]

@y-p did we test multiple datetime64[ns] columns with NaT, I think getting confused somewhere....

[jreback]

it works if rows < 25000....bug! darn

[ghost]

Yeah, we really tested the hell out of it.... you can tell him them to use

In [11]: df.to_csv('a.csv',engine='python')

which we prepared with some foresight...

[jreback]

thanks.....see my PR to fix this....but I don't think we have a test case for this (e.g. > 25k rows, multiple datetime columns, some with NaT)

[ghost]

keep this open, I'll add a test when I get a chance.

[jreback]

thnxs

[jreback]

ok...user reports that it works....(the fix)

[ghost]

@jreback , can't repro with:

from pandas import NaT
def make_dtnat_arr(n,nnat=None):
    if nnat is None:
        nnat= int(n*0.1) # 10%
    s=list(date_range('2000',freq='5min',periods=n))
    if nnat:
        for i in np.random.randint(0,len(s),nnat):
            s[i] = NaT
        i = np.random.randint(100)
        s[-i] = NaT
        s[i] = NaT
    return s
N=30000
s1=make_dtnat_arr(N)
s2=make_dtnat_arr(N)
s3=make_dtnat_arr(N,0)
df=DataFrame(dict(a=s1,b=s2,c=s3))
df.to_csv('/tmp/1.csv')
s=DataFrame.from_csv('/tmp/1.csv')
s=s.convert_objects('coerce')
In [54]: df.shape == s.shape
Out[54]: True

have you got a repro snippet?

[jreback]

must e something pathological about 30000, try 35000 or >....fails

[ghost]

nope.

[jreback]

your function was failing, change len(s1) -> len(s) (inside the function)

[ghost]

already did, i'm not missing lines on readback.

[jreback]

are you using the fix?

In [12]: df
Out[12]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 35000 entries, 0 to 34999
Data columns (total 3 columns):
a    31657  non-null values
b    31645  non-null values
c    35000  non-null values
dtypes: datetime64[ns](3)

In [13]: s
Out[13]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 35000 entries, 0 to 34999
Data columns (total 3 columns):
a    31657  non-null values
b    30295  non-null values
c    29999  non-null values
dtypes: datetime64[ns](3)

[ghost]

I don't think so.

λ gl
b544463 - (56 minutes ago) TST: add df.to_csv multiple dt cols with NaT GH3437 — y-p (HEAD, moar_more)
dfaf365 - (22 hours ago) DOC: mention numpydoc docstrings in CONTRIBUTING.md — y-p (upstream/master)

In [4]: pd.__version__
Out[4]: '0.12.0.dev-b544463'

[jreback]

that's just weird then...I just recloned

[ghost]

wait, I misunderstood the reported issue. I thought the
len was supposed to be < 35000, but it's the non-null count
you meant.

got it.

[jreback]

no total count needs > 25000 (not sure why 30k works but 35k fails)....was setting the mask on the original array size and not the sliced version, numpy should really raise on this....(it just wraps the assignment I think), that's why it 'usually' works

[ghost]

I modified the test to generate based on chunksize, and i'm testing with a small chunksize
or the test becomes too long.

It consistentl;y fails before, passes after fix, can I merge both and close?

[jreback]

yes

[jreback]

thxs

[ghost]

5c3ccdb

I guess the RC always reaches less people, so an X.1 bug fix release is a good habit,
with new features PR merged only after that. live and learn.

[jreback]

well the issue is you don't always try the rc on a production system, so don't find things.. (I know I didn't!))
found a bug when I tried it out on windows...

[ghost]

The RC is mostly psychological,basically getting a bunch of users to try a snapshot of master,
enticing them with the new features. But it only works up to a point, which is why we're getting
bug reports a day after final rather then within a week after the RC.