A 19kb record throws CsvRecordTooLargeException

[ohtwadi]

Hi,

A record in my CSV is failing with CsvRecordTooLargeException

Sylvan.Data.Csv.CsvRecordTooLargeException: Exception of type 'Sylvan.Data.Csv.CsvRecordTooLargeException' was thrown.
   at Sylvan.Data.Csv.CsvDataReader.NextRecordAsync(CancellationToken cancel)
   at Sylvan.Data.Csv.CsvDataReader.InitializeReaderAsync(CancellationToken cancel)
   at Sylvan.Data.Csv.CsvDataReader.InitializeReader()
   at Sylvan.Data.Csv.CsvDataReader.CreateInternal(TextReader reader, Char[] buffer, CsvDataReaderOptions options)

Putting the record in question in its own file tells me the record is only 19kb. If I reduce the column data in this record to under 9kb, the error disappears. My code is straight forward.

using var dr = CsvDataReader.Create(file.FullName);
foreach (var record in dr.GetRecords<MyClass>())

Am I running into a buffer limit per record or a column in a record? I'm not sure how to debug this further and would appreciate any assistance.

[MarkPflug]

The default buffer size is 16k. You can set the MaxBufferSize to allow the buffer to grow to accommodate larger records.

var opts = new CsvDataReaderOptions { MaxBufferSize = int.MaxValue};
using var dr = CsvDataReader.Create(file.FullName, opts);
foreach (var record in dr.GetRecords<MyClass>())

This will allow the reader to potentially process records up to 2GB.

[picasso566]

@MarkPflug Don't know where else to thank you.

Client requested a data export of their database. Certain columns are MB big with pasted html formatted text (hence this Issue). I've been importing/exporting csv files since... SQL Server 7? and I swear all of the usual tools are either getting worse and I could not produce the output I desired.

Your library successfully handles large text fields which may contain commas, line breaks, or quoted text and doesn't unnecessarily quote all of the non-string columns! I tried PS:Export-Csv, bcp, CSVHelper, FileHelpers, SSIS, SQL Exports, and on and on.

From this list: fastest-net-csv-parsers I started going through libraries one by one. I didn't need a parser, just a writer. I spent 2 days trying to get this simple task done and when I switched to your library, it was done in about 2 hours.

I've seen your projects for years, but never tried them. Glad I did! Thank you so much for this library.

[MarkPflug]

@picasso566 Thanks for the kind words! Glad you've found the library useful.