"TypeError: cb is not a function" When Calling unzipper.Parse() From An SSH Stream

[JonnyBoy333]

I'm trying to read multiple streams asynchronously from an SFTP server which are all zip folders. Running into an issue with larger zip files (>64KB) causing errors. With the default stream chunk size I'm getting a cb is not a function error, but if I increase the highWaterMark size I don't get the error, but 'end' never gets called on the stream. Also worth mentioning that if I don't create the stream inside of a Promise I don't seem to get this error, not sure why that is.

Here is what I've got:

sftp.connect(configObj).then(() => {
  var fileNames = ['file1'];
  var streamPromiseArr = [];
  for (fileName of fileNames) {

    // Transformers
    var selectFile = new Transform({
      objectMode: true,
      transform: function (entry, e, cb) {
        var fileName = entry.path;
        if (fileName.indexOf('DailyPunch') >= 0) {
          entry.pipe(concat(res => {
            this.push(res);
            cb();
          }))
        } else {
          entry.autodrain();
          cb();
        }
      }
    });

    // Collect the promises (promises resolve to a buffer)
    var streamPromise = new Promise((resolve, reject) => {
      sftp.get(fileName, true, 'utf8', { highWaterMark: 256 * 1024 }).then((stream) => {
        var text = '';
        stream
          .pipe(unzipper.Parse())
          .pipe(selectFile)
          .pipe(filterArray)
          .on('data', (d) => text += d)
          .on('finish', () => {
            resolve(text);
          });
      });
    });
    streamPromiseArr.push(streamPromise);
  }

  // Consolidate promised streams and write to a file
  Promise.all(streamPromiseArr).then((textArr) => {
    fs.writeFile('consolidate_file.csv', textArr.join(), ((err) => {
      if (err) console.log(err);
      console.log('Finished');
      sftp.end();
    }));
  });
}).catch(err => {
  console.log(err)
  sftp.end();
});```

[ZJONSSON]

Can you see from the stack which file/line is throwing the cb is not a function error?

I think the lack of end might be a problem with using on('data') which is an event emitter and might not actually pull from the pullStream. So if you wanted to collect the data without an emitter you could do something like:

  .pipe(filterArray)
  .pipe(new Transform({
    objectMode: true,
    transform: (d,e,cb) => {
      text += d;
      cb();
     }
  })
  .on('finish', () => resolve(text));

Here are some other hints:

• parseOne(regex) parses the first file that matches a regex.
• .promise() returns a promise on all the data as an array of the packets unzipped.

So you could do this instead of using the SelectFile transform:

var streamPromiseArr = fileNames.map(async fileName => {
  var stream = await sftp.get(fileName, true, 'utf8', { highWaterMark: 256 * 1024 });
  var data =  await stream.pipe(unzipper.ParseOne('DailyPunch')).promise();
  // data = filterarray(data) ?
  return data.join('');
});

[JonnyBoy333]

Thanks for the feedback, that gives me some food for thought. In my case, there are actually multiple files that match 'DailyPunch' inside the zip folders, so I'm guessing that eliminates the ability to use unzipper.ParseOne()? Feature request for a ParseMultiple(regex)?

I'm still experimenting with the event triggers, I did try using 'end' instead of 'finish' but they both have the same result. There is something strange happening that I can't pinpoint when downloading large zips that get broken into multiple chunks. I'll keep investigating.

This is the stack trace.

TypeError: cb is not a function
    at afterWrite (_stream_writable.js:454:3)
    at onwrite (_stream_writable.js:445:7)
    at /Users/jonlamb/projects/node-test/node_modules/unzipper/lib/PullStream.js:59:60
    at afterWrite (_stream_writable.js:454:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[ZJONSSON]

I have never experienced this before so I'm not sure exactly how this happens. But it seems that the _write method in Pullstream ends up being called without a callback (presumably by the pipe from the sftp).

I just published a version where we ignore the missing callback. Can you please check it out and see if you have any issues? You could also try to put a PassThrough stream between the sftp client and the unzipper and see if node internally does some magic to fix the piping. Also you need to ensure that the sftp stream actually ends when the file is done.

Unrelated: You can always do .buffer().then(...) to get the buffered output instead of your SelectedFile custom transform.

[ZJONSSON]

@JonnyBoy333 did you check out the new version ?
npm install unzipper@0.9.4

[ZJONSSON]

Hey @JonnyBoy333 this has probably been fixed here
Latest version is: + unzipper@0.9.9

Can you please verify and let me know? If this issue persist please reopen the ticket