throwing err with large tsv

[nickdestefano]

I have a 2M line tsv, and im getting this

events.js:72
    throw er; // Unhandled 'error' event
          ^
Error: no writecb in Transform class
    at afterTransform (_stream_transform.js:90:33)
    at TransformState.afterTransform (_stream_transform.js:74:12)
    at /Users/nickdestefano/node_modules/fast-csv/lib/parser/parser_stream.js:206:21
    at /Users/nickdestefano/node_modules/fast-csv/lib/parser/parser_stream.js:121:17
    at asyncIterator (/Users/nickdestefano/node_modules/fast-csv/lib/extended.js:29:17)
    at Object._onImmediate (/Users/nickdestefano/node_modules/fast-csv/lib/extended.js:18:37)
    at processImmediate [as _immediateCallback] (timers.js:345:15)

My code is using the writeStream. I am trying to use a readStream in here as well but I'm wondering if this should be good enough, or is there a cap on size?

var formatStream = csv
  .createWriteStream({headers: true})
  .transform(function(data){
      return {
          'Author': data['Author'],
          'GUID': data['GUID'],
          'Contents': data['GUID'].split(',').join(''),
          'Date(GMT)': moment(new Date(data['Date(GMT)'])).format('MM/DD/YYYY HH:mm:ss A')
      };
  });
csv
   .fromPath(file, {delimiter: '\t', quote:'�', objectMode:true, headers: true, escape:'′'})
   .pipe(formatStream)

.pipe(fs.createWriteStream(test, {encoding: "utf8"}));

Also I tried a few different csv parsing libraries and this is best I've seen so nice work.

[benjyblack]

I have a ~7000 line TSV and I'm also getting this error

Edit: When making use of the 'error" event I noticed that the problem was actually with quotes in my TSV file

[doug-martin]

OK got time to investigate this today....it looks like the issue was an error that occured while parsing the tsv however internally an error event was emitted instead of calling the done cb from the transform with the error. I will publish a new version today that should emit the correct error

[doug-martin]

So Im pretty sure I found the bug I was able to reproduce this by throwing an error in the end listener which caused an error to be bubbled back up to transform. I added tests for this in issues.test.js. Closing for now, you can reopen if the issue persists.

[spkism]

I'm also getting this error, using the latest version(0.5.3). it's a ~0.5GB TSV file.

csv
    .fromPath(__dirname+'/../testfiles//allCountries.txt', {delimiter: "\t"})
    .on("data", function(entry){
         var obj = {
                geonameid: entry[0],
                timezone: entry[17]
            };
    })
    .on("end", function(){
         console.log("done");
    });

results in:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: no writecb in Transform class
    at afterTransform (_stream_transform.js:90:33)
    at TransformState.afterTransform (_stream_transform.js:74:12)
    at /home/aks/dev/drok/node_modules/fast-csv/lib/parser/parser_stream.js:190:21
    at /home/aks/dev/drok/node_modules/fast-csv/lib/parser/parser_stream.js:105:17
    at asyncIterator (/home/aks/dev/drok/node_modules/fast-csv/lib/extended.js:29:17)
    at Object._onImmediate (/home/aks/dev/drok/node_modules/fast-csv/lib/extended.js:18:37)
    at processImmediate [as _immediateCallback] (timers.js:330:15)

[coyotte]

Same problem(

[PepijnSenders]

Same problem with 450k line csv, throws error after approximately 200k lines parsed.

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: no writecb in Transform class
    at afterTransform (_stream_transform.js:90:33)
    at TransformState.afterTransform (_stream_transform.js:74:12)
    at /Volumes/Development/static-maps/node_modules/fast-csv/lib/parser/parser_stream.js:190:21
    at /Volumes/Development/static-maps/node_modules/fast-csv/lib/parser/parser_stream.js:105:17
    at asyncIterator (/Volumes/Development/static-maps/node_modules/fast-csv/lib/extended.js:29:17)
    at Object._onImmediate (/Volumes/Development/static-maps/node_modules/fast-csv/lib/extended.js:18:37)
    at processImmediate [as _immediateCallback] (timers.js:345:15)

[doug-martin]

I have been unsuccessful in reproducing this error...so I was wondering if any one can provide me with an example tsv and some code that produces the error?

[PepijnSenders]

Tested the file with a node-csv-parser, and it gave me and unexpected quote error.

After the setting the correct delimiter, quote and escape it worked.

[surajdesai87]

I am also getting a same error with 20k line of CSV(file size 1.3mb)

error: [D:\innovateMR\api_platform\main.js] undefined: "no writecb in Transform
class" "Error: no writecb in Transform class\n at afterTransform (_stream_tra
nsform.js:90:33)\n at TransformState.afterTransform (_stream_transform.js:74:
12)\n at D:\innovateMR\api_platform\node_modules\fast-csv\lib\parser\p
arser_stream.js:190:21\n at D:\innovateMR\api_platform\node_modules\fast-
csv\lib\parser\parser_stream.js:105:17\n at asyncIterator (D:\innovateMR
\api_platform\node_modules\fast-csv\lib\extended.js:29:17)\n at Object._o
nImmediate (D:\innovateMR\api_platform\node_modules\fast-csv\lib\extended.
js:18:37)\n at processImmediate as _immediateCallback

[doug-martin]

I was finally able to reproduce this by creating a large tsv and not registering an error handler. If you can give v0.5.4 and let me know if this fixes your issues that would be much appreciated!

-Doug

[surajdesai87]

v0.5.4 fixes my issue. It works for me. Only thing its reading one extra row at the end of CSV file.

[doug-martin]

Great! Does your file have extra line breaks at the end?

[surajdesai87]

Yes... but same file was working on v0.5.3, anyways i fixed it in my code by putting a check.

[pseudosavant]

I am having the same issue with a csv that has ~2.2M rows. I added an error event listener which allowed it to continue processing the file but the end event is never emitted. I made a csv file with just the first 100k rows and everything works fine (no errors, end is emitted).

I would love to share the file but it contains data that I can't share.

Thoughts?

[pseudosavant]

I found the rows that were causing the error. This is what they look like:

1234,987654321,123456789,1,"2014-01-01",1.0000,"{\"viewPref\":false,\"isOlbUserDescription\":false}"

It is the last column that is causing the problem. Rows that don't have any escaping work fine. After stripping those rows out of my CSV file everything processed fine.

[loretoparisi]

@pseudosavant you are referring to the on('error') callback? If I use this, it will catch the error, but the Parser will break and end the parsing of the file without success.

[doug-martin]

With out some more info this one is going to be hard to reproduce. Closing again until there is good example.