nrows incompatible with chunksize in read_csv #15755

Closed
toobaz opened this Issue Mar 21, 2017 · 0 comments

Comments

Projects
None yet
2 participants
Contributor

toobaz commented Mar 21, 2017

Code Sample, a copy-pastable example if possible

In [2]: pd.read_csv('/tmp/boh.csv', nrows=10, chunksize=2)
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-2-c652c48282ad> in <module>()
----> 1 pd.read_csv('/tmp/boh.csv', nrows=10, chunksize=2)

/home/nobackup/repo/pandas/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    651                     skip_blank_lines=skip_blank_lines)
    652 
--> 653         return _read(filepath_or_buffer, kwds)
    654 
    655     parser_f.__name__ = name

/home/nobackup/repo/pandas/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    391 
    392     if (nrows is not None) and (chunksize is not None):
--> 393         raise NotImplementedError("'nrows' and 'chunksize' cannot be used"
    394                                   " together yet.")
    395     elif nrows is not None:

NotImplementedError: 'nrows' and 'chunksize' cannot be used together yet.

(as introduced by #6774 )

Problem description

Supporting them together is not complicated - PR on its way.

toobaz referenced this issue Mar 21, 2017

Closed

ENH: support "nrows" and "chunksize" together #15756

4 of 4 tasks complete

jreback added this to the 0.20.0 milestone Mar 21, 2017

jreback closed this in 163d18e Mar 21, 2017

@mattip mattip added a commit to mattip/pandas that referenced this issue Apr 3, 2017

@toobaz @mattip toobaz + mattip ENH: support "nrows" and "chunksize" together
closes #15755

Author: Pietro Battiston <me@pietrobattiston.it>

Closes #15756 from toobaz/nrows_chunksize and squashes the following commits:

d0288e3 [Pietro Battiston] ENH: support "nrows" and "chunksize" together
69c1533

@linebp linebp added a commit to linebp/pandas that referenced this issue Apr 17, 2017

@toobaz @linebp toobaz + linebp ENH: support "nrows" and "chunksize" together
closes #15755

Author: Pietro Battiston <me@pietrobattiston.it>

Closes #15756 from toobaz/nrows_chunksize and squashes the following commits:

d0288e3 [Pietro Battiston] ENH: support "nrows" and "chunksize" together
7d0cddb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment