Skip to content
Permalink
Browse files

Merge branch 'more_csv_moar'

* more_csv_moar:
  BUG: GH3437 to_csv issue with large number of rows and some NaT in multiple
  TST: add df.to_csv multiple dt cols with NaT GH3437
  • Loading branch information...
y-p
y-p committed Apr 24, 2013
2 parents 0c784e6 + 13111ad commit 5c3ccdba2b0dcc8963af6de0c7a2ab6716364147
Showing with 38 additions and 4 deletions.
  1. +6 −3 RELEASE.rst
  2. +1 −1 pandas/core/internals.py
  3. +31 −0 pandas/tests/test_frame.py
@@ -44,14 +44,17 @@ pandas 0.12.0

**Bug Fixes**

- Fixed an esoteric excel reading bug, xlrd>= 0.9.0 now required for excel
support. Should provide python3 support (for reading) which has been
lacking. (GH3164_)
- Fixed an esoteric excel reading bug, xlrd>= 0.9.0 now required for excel
support. Should provide python3 support (for reading) which has been
lacking. (GH3164_)
- Fix to_csv issue when having a large number of rows and ``NaT`` in some
columns (GH3437_)

.. _GH3164: https://github.com/pydata/pandas/issues/3164
.. _GH3251: https://github.com/pydata/pandas/issues/3251
.. _GH3379: https://github.com/pydata/pandas/issues/3379
.. _GH3038: https://github.com/pydata/pandas/issues/3038
.. _GH3437: https://github.com/pydata/pandas/issues/3437



@@ -787,7 +787,7 @@ def to_native_types(self, slicer=None, na_rep=None, **kwargs):
values = values[:,slicer]
mask = isnull(values)

rvalues = np.empty(self.shape,dtype=object)
rvalues = np.empty(values.shape,dtype=object)
if na_rep is None:
na_rep = 'NaT'
rvalues[mask] = na_rep
@@ -4706,6 +4706,37 @@ def stuple_to_tuple(x):
N = 100
chunksize=1000

# GH3437
from pandas import NaT
def make_dtnat_arr(n,nnat=None):
if nnat is None:
nnat= int(n*0.1) # 10%
s=list(date_range('2000',freq='5min',periods=n))
if nnat:
for i in np.random.randint(0,len(s),nnat):
s[i] = NaT
i = np.random.randint(100)
s[-i] = NaT
s[i] = NaT
return s
# N=35000
s1=make_dtnat_arr(chunksize+5)
s2=make_dtnat_arr(chunksize+5,0)
# s3=make_dtnat_arr(chunksize+5,0)
df=DataFrame(dict(a=s1,b=s2))
df.to_csv('/tmp/1.csv',chunksize=chunksize)
recons = DataFrame.from_csv('/tmp/1.csv').convert_objects('coerce')
assert_frame_equal(df, recons,check_names=False,check_less_precise=True)

for ncols in [4]:
base = int((chunksize// ncols or 1) or 1)
for nrows in [2,10,N-1,N,N+1,N+2,2*N-2,2*N-1,2*N,2*N+1,2*N+2,
base-1,base,base+1]:
_do_test(mkdf(nrows, ncols,r_idx_type='dt',
c_idx_type='s'),path, 'dt','s')
pass


for ncols in [4]:
base = int((chunksize// ncols or 1) or 1)
for nrows in [2,10,N-1,N,N+1,N+2,2*N-2,2*N-1,2*N,2*N+1,2*N+2,

0 comments on commit 5c3ccdb

Please sign in to comment.
You can’t perform that action at this time.