async / stream support

[dduponchel]

This pull request replaces #141.
The current API is synchronous : if JSZip takes too much time to finish its task, the page crashes
(it freezes during the task anyway). This commit does a the following :

• rewrite the code into workers which are asynchronous
• replace the public methods
• add nodejs stream support
• break the compatibility with existing code

Note : I created a branch jszip_v3 which I target : a change this big can't go
to the current v2.

Workers

A worker is like a nodejs stream but with some differences. On the good side :

• it works on IE 6-9 without any issue / polyfill
• it weights less than the full dependencies bundled with browserify
• it forwards errors (no need to declare an error handler EVERYWHERE)

On the bad side :
It adds new code to maintain, fix, optimize.

A chunk is an object with 2 attributes : meta and data. The former is an
object containing anything (percent for example), see each worker for more
details. The latter is the real data (String, Uint8Array, etc).

Public API

I've updated my gh-pages with the latest API documentation.

Getters are replaced by nodeStream(type) and async(type), generate() is replaced by generateNodeStream and generateAsync. The "async" methods return a Promise, the "stream" methods return a Stream3 (from nodejs) if available.
They have an update callback to get access to metadata like the
name of the current file being compressed or the operation progress (with percent).

With this pull request, the API will be :

// generate
zip.generateAsync(...) // return a Promise of the content
zip.generateNodeStream(...) // return the node stream
zip.generateInternalStream(...) // return the internal stream to let the user write glue code

// getters
zip.async("arraybuffer") // return a Promise
zip.nodeStream("arraybuffer") // return a nodejs stream
zip.internalStream(...) // return the internal stream to let the user write glue code

The previous sync methods now throw exceptions.

Nodejs stream support

file(name, data) now accepts a nodejs stream as data. The "nodeStream" methods generates streams.

Breaking changes

The undocumented JSZip.compressions object changed : the object now returns
workers to do the job, the previous methods are not used anymore.

A lot of sync methods has been removed.

[dduponchel]

Keeping the sync methods will be too much troubles, I'll remove them.

That would give

zip.file("content.txt").async("uint8array", function onComplete(){...});
zip.file("content.txt").async({
    type: "uint8array",
    onUpdate: function () {...} // allow progress update notifications
}, function onComplete(){...});

zip.file("content.txt").stream("uint8array").on("data", ...); // our simple stream
zip.file("content.txt").nodeStream("uint8array").on("data", ...); // nodejs stream

zip.generateAsync({...}, function onComplete(){...});
zip.generateStream({...}).on("data", ...);
zip.generateNodeStream({...}).on("data", ...);

The old methods will throw an exception like "This method has been removed in JSZip 3.0, please check the upgrade guide.".

I'll remove the synchronous .load() method too : a sync checkCRC32 option isn't possible anymore and an async method allows more things (loading from a Blob, from a nodejs file descriptor, etc).

For the async operations, ES6 Promises look like the right choice (instead of manually adding a callback) but that means adding more bytes in the build.

@Stuk Does this sound reasonable ?

[andrewvarga]

Is this going to be part of the stable release some time? (Or any other async (webworker) support?

[dduponchel]

I'm still working on it... When I have enough time. I may have some time this weekend to continue working on the async load.

[andrewvarga]

Let me know if I can help, I could actually make it work in a webworker really easily by just adding some wrapper methods on top of jszip.min.js like:

self.onmessage = function(message) {
    var messageData = message.data;
    var command = messageData.command;
    var args = messageData.args;
    var requestId = messageData.requestId;

    switch (command) {
        case "load":
                   jsZip = new JSZip();
            jsZip.load(args[0]);

            self.postMessage({
                requestId: requestId
            });
                       break;
        case "parse":
           // ... etc

This way I didn't need to touch the actual JSZIP code at all, it just allows to call its functions in a webworker.

[dduponchel]

I've updated the pull request with async methods and I've updated the description. I've updated my gh-pages with the latest API documentation.
The next step will be to merge this with master.

@andrewvarga doesn't importScripts solve this issue ? Making JSZip Worker aware may give a strange API.

[balaclark]

Looks great, async is very necessary for large files.

[dduponchel]

Up-to-date with v2.5.0 (I moved the branch jszip_v3 to this tag). @Stuk are you ok with this pull request ?

[dduponchel]

I've prefixed stream methods with "node": browsers are also adding streams, see 1, 2, 3. While the spirit is the same, the API is different from nodejs streams. To avoid future confusion, I prefer renaming generateStream to generateNodeStream and stream to nodeStream (we already have nodebuffer as type).

[Stuk]

Should ZipFileWorker be put in its own file?

[dduponchel]

Yes.

[dduponchel]

Done.

[Stuk]

👍 is this ready to me merged then?

[dduponchel]

@Stuk this pull request is ready, I'll continue on #224.

[0cv]

damned, this is amazing, seriously. v2.5 just failed and crashed after 1 minute or so on few hundreds files weighting between 10KB to 500KB each and this new version just did the whole job in half a second! Unless there are some bugs (which I have not found on my few basic tests), it would make sense it becomes the new official version.

Small info: grunt (which I had to use to produce the dist v3.0) is somehow outdated with its uglify dependency and throws an error. Updating uglify to the latest version simply fixed the problem.