Out of memory when zipping directory.

[rloehr]

Archiver has an upper bound memory limit when zipping a file.

For example, let's say you have a million files in a directory and you attempt to zip that directory.
`util.walkdir(dirpath, function(err, results) {
if (err) {
self.emit('error', err);
} else {
results.forEach(function(file) {
var entryData = _.extend({}, data);
entryData.name = file.relative;
entryData.prefix = destpath;
entryData.stats = file.stats;

    try {
      if (dataFunction) {
        entryData = dataFunction(entryData);

        if (typeof entryData !== 'object') {
          throw new Error('directory: invalid data returned from custom function');
        }
      }
    } catch(e) {
      self.emit('error', e);
      return;
    }

    self._append(file.path, entryData);
  });
}

self._pending--;
self._maybeFinalize();

});

return this;
};`

If there happened to a plethora of files in the directory, (e.g. hundreds of thousands or a million) the results array becomes huge.

Then for each result, an entry data object is created and a task is ultimately created for each entry data. All that has to live in memory at the same time and will have a finite bound.

The task queue is a great idea but it seems like to alleviate memory pressure we should be filling it with a subset of files, burning down the queue, filling with next chunk of files....and then repeat.

[Prinzhorn]

Please create a working demo. Your code is incomplete.

Here's what I suggest you do if you want to zip a million files: add them to the archive in series (not in parallel) and stream away the result immediately.

Untested pseudo code sans error handling using https://www.npmjs.com/package/walk

var fs = require('fs');

var walk = require('walk');
var archiver = require('archiver');

var walker = walk.walk('/tmp', options);
var archive = archiver.create('zip', {});

archive.pipe(fs.createWriteStream('out.zip'));

walker.on('file', function(root, fileStats, next) {
    var stream = fs.createReadStream(fileStats.name);

    archive.append(stream, {
        name: fileStats.name
    });

    stream.on('finish', next);
});

walker.on('end', function () {
    archive.finalize();
});

[ctalkington]

so the concern is with the size of array when walking a large directory structure. it does look like an area that could be improved ive put together some changes 287c722 if someone would like to review

[soyuka]

Nice improvements!

[ctalkington]

usage of events for these methods was quietly released last week in

[ctalkington]

... in v1.3.0