Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Added stream-like read API and write API to node-zipfile #16

Closed
wants to merge 9 commits into from

3 participants

@toots

Hi!

As discussed previously, I have performed some changes in node-zipfile. They are the following:

  • Made it impossible to have multiple concurrent calls to libzip functions by referencing opened objects
    (file, source) and throwing an exception when they are used.

  • Added a stream-like API for file read. I figured this out because

    • I prefer that an asynchronous read does not necessarily read all the file at once.
    • libzip actually has an API that, on purpose, imitates usual file system IO.

    Thus, the code for this API in zipfile.js is actually taken from node's code..

  • Added write API. For now, one can only add/replace local files and local directories. Adding buffers should
    be fairly trivial to add.

    The trick here is that adding/replacing in libzip returns immediately, Actual write is done when closing
    (saving in the current API). Thus, it is not possible to stream-line write operations in a stream-oriented
    fashion...

I am aware that those changes represent a pretty huge modification of you codebase. I am submitting a pull-request in the hope that you could accept them. However, feel free to refuse them as well, I would have no problem maintaing it myself -- it is just that I believe that there are already way to many node-zipfile bindings floating around to add yet another one..

Cheers,
Romain

@springmeyer
Owner

great, thanks very much for the contribution. Looks good. I am just headed offline for 2 weeks however, so I'll review when I'm back and look into applying the merge then. thanks for the patience.

@springmeyer
Owner

hey there. now that node v6 is out, and things have changed a bit around async stuff. This ideally could be revised so that it automatically merges and works with node v6. If you are on to new things, no worries I will try to take a look once node v6 settles out a bit.

@chrishamant

+1 to this..

I tried to merge myself with current head but it appears there is some reworking going on with 'node_zipfile.cpp' for the 0.6.x update - and i'm not a C++/node extension expert :)
I'll look at it again later to see if I can get it working (am working on something that could using this streaming interface right now) but might be better if someone how know wtf they were doing did it :) (doesn't otherwise seem to be a huge merge (assuming it works))

@springmeyer
Owner

ya, wish I had of merged it long ago, apologies. It will require more wrangling time than I have to update to node v6. Currently I'm pushing on trying to get node-zipfile compiled on windows.

@toots

Hi,

I've had a try at this. A lot of C headers have been changed since 0.4.x, which I use here. Thus, I cannot compile the new code.

It would need to work around the new uv_work_t type, using the NODE_VERSION_AT_LEAST macro..

@springmeyer
Owner

this patch should ideally be revised to work with node v0.6.x, which, as you note uses libuv now and uv_work_t. Current node-zipfile master code is already using these libuv types. Thoughts?

@toots

It's feasible but I want to use node 0.4.x here for production reasons. I'll see if I find time to have another 0.6.x install and update the patch to make it compile with both versions.

@springmeyer
Owner

curious, what is holding you back from 0.6.x?

@toots

I'm planning on using on heroku at some point...

@springmeyer
Owner

closing, this is stale. sorry we missed getting it in. would review an updated pull.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 9, 2011
  1. @toots

    Added stream-like API for asynchronous zipfile read..

    toots authored
    Next step: stream-like API for file write :-)
Commits on Sep 13, 2011
  1. @toots
  2. @toots

    Minor stuff..

    toots authored
Commits on Sep 16, 2011
  1. @toots
  2. @toots
  3. @toots

    Added addDirectory.

    toots authored
Commits on Sep 17, 2011
  1. @toots
  2. @toots
  3. @toots

    More README.md..

    toots authored
This page is out of date. Refresh to see the latest.
View
56 README.md
@@ -1,4 +1,3 @@
-
# Node-Zipfile
Bindings to [libzip](http://nih.at/libzip/libzip.html) for handling zipfiles in [node](http://nodejs.org).
@@ -6,15 +5,54 @@
# Example
- > var zipfile = require('zipfile')
- > var zf = new zipfile.ZipFile('./data/world_merc.zip')
- > zf
- { names: [ 'world_merc.dbf', 'world_merc.prj', 'world_merc.shp', 'world_merc.shx' ],
+ var zipfile = require('zipfile');
+ var fs = require('fs');
+
+ // Open:
+ var zf = new zipfile.ZipFile('./data/world_merc.zip');
+ // { names: [ 'world_merc.dbf', 'world_merc.prj', 'world_merc.shp', 'world_merc.shx' ],
count: 4 }
- > var buffer = zf.readFileSync('world_merc.prj')
- > buffer.toString()
- 'PROJCS["Google Maps Global Mercator",GEOGCS .... '
+
+ // Synchronous read:
+ var buffer = zf.readFileSync('world_merc.prj');
+ // 'PROJCS["Google Maps Global Mercator",GEOGCS .... '
+
+ // Asynchronous read:
+ var stream = zf.createReadStream('world_merc.prj');
+ var fstream = fs.createWriteStream('/tmp/world_merc.prj');
+ stream.pipe(fstream);
+
+ // Write:
+ // Add:
+ zf.addDirectory('foo');
+ zf.addFile('foo/world_merc.prj', '/tmp/world_merc.prj');
+ zf.save(function (err) {
+ if (err)
+ console.log("Error :" + err)
+ else
+ console.log("Done")
+ });
+ // Done
+
+ // Replace:
+ zf.replaceFile('foo/world_merc.prj', '/tmp/world_merc.prj');
+ zf.save(...);
+
+
+# Note:
+
+* libzip is not thread-safe. Thus, there can only be one of the following operation at a given time:
+ * `zf.createReadStream`
+ * `zf.readFileSync`
+ * `zf.addFile`
+ * `zf.replaceFile`
+ * `zf.save`
+ Call to `zf.readFileSync`, `zf.addFile` and `zf.replaceFile` are synchronous. Upon opening a read stream,
+ the zip archive can be used again with another of the operations above once the `'close'` event has been emitted
+ by the stream.
+* Adding and replacing files in the archive returns immediatly. Files are actually written
+ when calling `zf.save`.
## Depends
@@ -84,4 +122,4 @@
## License
- BSD, see LICENSE.txt
+ BSD, see LICENSE.txt
View
140 lib/zipfile.js
@@ -1,8 +1,138 @@
-var assert = require('assert');
+var Stream = require('stream').Stream;
+var util = require('util')
var zipfile = require('./_zipfile');
-/* assert ABI compatibility */
-assert.ok(zipfile.versions.node === process.versions.node, 'The running node version "' + process.versions.node + '" does not match the node version that node-zipfile was compiled against: "' + zipfile.versions.node + '"');
+var pool;
+var kMinPoolSpace = 128;
+var kPoolSize = 40 * 1024;
-// push all C++ symbols into js module
-for (var k in zipfile) { exports[k] = zipfile[k]; }
+function allocNewPool() {
+ pool = new Buffer(kPoolSize);
+ pool.used = 0;
+}
+
+zipfile.ZipFile.prototype.createReadStream = function (file) {
+ return new ReadStream(this, file);
+}
+
+var ReadStream = function(zf, file) {
+ if (!(this instanceof ReadStream)) return new ReadStream(zip, file);
+
+ Stream.call(this);
+
+ this.readable = true;
+ this.paused = false;
+ this.bufferSize = 64 * 1024;
+
+ if (this.encoding) this.setEncoding(this.encoding);
+
+ this.zf = zf;
+
+ zf.open(file);
+ this._read();
+}
+util.inherits(ReadStream, Stream);
+
+ReadStream.prototype.setEncoding = function(encoding) {
+ var StringDecoder = require('string_decoder').StringDecoder; // lazy load
+ this._decoder = new StringDecoder(encoding);
+};
+
+ReadStream.prototype._read = function() {
+ var self = this;
+ if (!self.readable || self.paused || self.reading) return;
+
+ self.reading = true;
+
+ if (!pool || pool.length - pool.used < kMinPoolSpace) {
+ // discard the old pool. Can't add to the free list because
+ // users might have refernces to slices on it.
+ pool = null;
+ allocNewPool();
+ }
+
+ // Grab another reference to the pool in the case that while we're in the
+ // thread pool another read() finishes up the pool, and allocates a new
+ // one.
+ var thisPool = pool;
+ var toRead = Math.min(pool.length - pool.used, this.bufferSize);
+ var start = pool.used;
+
+ if (this.pos !== undefined) {
+ toRead = Math.min(this.end - this.pos + 1, toRead);
+ }
+
+ function afterRead(err, bytesRead) {
+ self.reading = false;
+ if (err) {
+ self.emit('error', err);
+ self.readable = false;
+ return;
+ }
+
+ if (bytesRead === 0) {
+ self.emit('end');
+ self.destroy();
+ return;
+ }
+
+ var b = thisPool.slice(start, start + bytesRead);
+
+ // Possible optimizition here?
+ // Reclaim some bytes if bytesRead < toRead?
+ // Would need to ensure that pool === thisPool.
+
+ // do not emit events if the stream is paused
+ if (self.paused) {
+ self.buffer = b;
+ return;
+ }
+
+ // do not emit events anymore after we declared the stream unreadable
+ if (!self.readable) return;
+
+ self._emitData(b);
+ self._read();
+ }
+
+ self.zf.read(pool, pool.used, toRead, afterRead);
+
+ if (self.pos !== undefined) {
+ self.pos += toRead;
+ }
+ pool.used += toRead;
+};
+
+ReadStream.prototype._emitData = function(d) {
+ if (this._decoder) {
+ var string = this._decoder.write(d);
+ if (string.length) this.emit('data', string);
+ } else {
+ this.emit('data', d);
+ }
+};
+
+
+ReadStream.prototype.destroy = function() {
+ this.readable = false;
+ this.zf.close();
+ this.emit('close');
+};
+
+
+ReadStream.prototype.pause = function() {
+ this.paused = true;
+};
+
+ReadStream.prototype.resume = function() {
+ this.paused = false;
+
+ if (this.buffer) {
+ this._emitData(this.buffer);
+ this.buffer = null;
+ }
+
+ this._read();
+};
+
+exports.ZipFile = zipfile.ZipFile;
View
2  package.json
@@ -1,6 +1,6 @@
{
"name" : "zipfile",
- "version" : "0.2.3",
+ "version" : "0.3.0",
"main" : "./lib/index.js",
"description" : "C++ library for handling zipfiles in node",
"keywords" : ["zipfile", "uncompress", "unzip", "zlib"],
View
546 src/node_zipfile.cpp
@@ -5,49 +5,68 @@
#include <vector>
#include <cstring>
#include <algorithm>
-//#include <iostream>
+#include <cassert>
+#include <set>
#include <node_buffer.h>
#include <node_version.h>
-
#define TOSTR(obj) (*String::Utf8Value((obj)->ToString()))
Persistent<FunctionTemplate> ZipFile::constructor;
+static ev_async notifier;
+static std::set<save_closure_t *> saving_closures;
+
void ZipFile::Initialize(Handle<Object> target) {
- HandleScope scope;
-
constructor = Persistent<FunctionTemplate>::New(FunctionTemplate::New(ZipFile::New));
constructor->InstanceTemplate()->SetInternalFieldCount(1);
constructor->SetClassName(String::NewSymbol("ZipFile"));
// functions
+ NODE_SET_PROTOTYPE_METHOD(constructor, "open", Open);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "read", Read);
NODE_SET_PROTOTYPE_METHOD(constructor, "readFileSync", readFileSync);
- NODE_SET_PROTOTYPE_METHOD(constructor, "readFile", readFile);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "close", Close);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "addFile", Add_File);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "replaceFile", Replace_File);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "addDirectory", Add_Directory);
+ NODE_SET_PROTOTYPE_METHOD(constructor, "save", Save);
// properties
constructor->InstanceTemplate()->SetAccessor(String::NewSymbol("count"), get_prop);
constructor->InstanceTemplate()->SetAccessor(String::NewSymbol("names"), get_prop);
+ // Initiate async handler
+ ev_async_init(&notifier, Save_Callback);
+ ev_async_start(EV_DEFAULT_UC_ &notifier);
+
target->Set(String::NewSymbol("ZipFile"),constructor->GetFunction());
}
ZipFile::ZipFile(std::string const& file_name) :
ObjectWrap(),
- file_name_(file_name),
- archive_(),
- names_() {}
+ file_name(file_name),
+ archive(),
+ file(NULL),
+ file_index(-1),
+ saving(false),
+ names() {}
ZipFile::~ZipFile() {
- zip_close(archive_);
+ assert(file == NULL);
+ ev_async_stop(EV_DEFAULT_UC_ &notifier);
+ zip_close(archive);
+}
+
+bool ZipFile::Busy() {
+ if (this->file != NULL || this->saving) return true;
+ return false;
}
Handle<Value> ZipFile::New(const Arguments& args)
{
- HandleScope scope;
-
if (!args.IsConstructCall())
return ThrowException(String::New("Cannot call constructor as function, you need to use 'new' keyword"));
@@ -59,7 +78,7 @@ Handle<Value> ZipFile::New(const Arguments& args)
struct zip *za;
int err;
char errstr[1024];
- if ((za=zip_open(input_file.c_str(), 0, &err)) == NULL) {
+ if ((za=zip_open(input_file.c_str(), ZIP_CREATE, &err)) == NULL) {
zip_error_to_str(errstr, sizeof(errstr), err, errno);
std::stringstream s;
s << "cannot open file: " << input_file << " error: " << errstr << "\n";
@@ -69,19 +88,24 @@ Handle<Value> ZipFile::New(const Arguments& args)
ZipFile* zf = new ZipFile(input_file);
- int num = zip_get_num_files(za);
- zf->names_.reserve(num);
+ zf->archive = za;
+ zf->GetNames();
+ zf->Wrap(args.This());
+ return args.This();
+
+}
+
+void ZipFile::GetNames()
+{
+ int num = zip_get_num_files(this->archive);
+ this->names.clear();
+ this->names.reserve(num);
int i = 0;
for (i=0; i<num; i++) {
struct zip_stat st;
- zip_stat_index(za, i, 0, &st);
- zf->names_.push_back(st.name);
+ zip_stat_index(this->archive, i, 0, &st);
+ this->names.push_back(st.name);
}
-
- zf->archive_ = za;
- zf->Wrap(args.This());
- return args.This();
-
}
Handle<Value> ZipFile::get_prop(Local<String> property,
@@ -91,14 +115,14 @@ Handle<Value> ZipFile::get_prop(Local<String> property,
ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(info.This());
std::string a = TOSTR(property);
if (a == "count") {
- return scope.Close(Integer::New(zf->names_.size()));
+ return scope.Close(Integer::New(zf->names.size()));
}
if (a == "names") {
- unsigned num = zf->names_.size();
+ unsigned num = zf->names.size();
Local<Array> a = Array::New(num);
for (unsigned i = 0; i < num; ++i )
{
- a->Set(i,String::New(zf->names_[i].c_str()));
+ a->Set(i,String::New(zf->names[i].c_str()));
}
return scope.Close(a);
@@ -106,6 +130,62 @@ Handle<Value> ZipFile::get_prop(Local<String> property,
return Undefined();
}
+Handle<Value> ZipFile::Open(const Arguments& args)
+{
+ if (args.Length() != 1 || !args[0]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a file name inside the zip")));
+
+ std::string name = TOSTR(args[0]);
+
+ // TODO - enforce valid index
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
+
+ zf->file_index = -1;
+
+ std::vector<std::string>::iterator it = std::find(zf->names.begin(), zf->names.end(), name);
+ if (it!=zf->names.end()) {
+ zf->file_index = distance(zf->names.begin(), it);
+ }
+
+ if (zf->file_index == -1) {
+ std::stringstream s;
+ s << "No file found by the name of: '" << name << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
+
+ if ((zf->file=zip_fopen_index(zf->archive, zf->file_index, 0)) == NULL) {
+ zip_fclose(zf->file);
+ zf->file = NULL;
+ std::stringstream s;
+ s << "cannot open file #" << zf->file_index << " in " << name << ": archive error: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
+
+ return Undefined();
+}
+
+Handle<Value> ZipFile::Close(const Arguments& args)
+{
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+
+ if (zf->file == NULL)
+ return ThrowException(Exception::Error(String::New("No file opened!")));
+
+ if (zip_fclose(zf->file) != 0) {
+ std::stringstream s;
+ s << "Error while closing zip file: " << zip_file_strerror(zf->file) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
+
+ zf->file = NULL;
+
+ return Undefined();
+}
+
Handle<Value> ZipFile::readFileSync(const Arguments& args)
{
HandleScope scope;
@@ -119,13 +199,16 @@ Handle<Value> ZipFile::readFileSync(const Arguments& args)
// TODO - enforce valid index
ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
+
struct zip_file *zf_ptr;
int idx = -1;
- std::vector<std::string>::iterator it = std::find(zf->names_.begin(), zf->names_.end(), name);
- if (it!=zf->names_.end()) {
- idx = distance(zf->names_.begin(), it);
+ std::vector<std::string>::iterator it = std::find(zf->names.begin(), zf->names.end(), name);
+ if (it!=zf->names.end()) {
+ idx = distance(zf->names.begin(), it);
}
if (idx == -1) {
@@ -134,15 +217,15 @@ Handle<Value> ZipFile::readFileSync(const Arguments& args)
return ThrowException(Exception::Error(String::New(s.str().c_str())));
}
- if ((zf_ptr=zip_fopen_index(zf->archive_, idx, 0)) == NULL) {
+ if ((zf_ptr=zip_fopen_index(zf->archive, idx, 0)) == NULL) {
zip_fclose(zf_ptr);
std::stringstream s;
- s << "cannot open file #" << idx << " in " << name << ": archive error: " << zip_strerror(zf->archive_) << "\n";
+ s << "cannot open file #" << idx << " in " << name << ": archive error: " << zip_strerror(zf->archive) << "\n";
return ThrowException(Exception::Error(String::New(s.str().c_str())));
}
struct zip_stat st;
- zip_stat_index(zf->archive_, idx, 0, &st);
+ zip_stat_index(zf->archive, idx, 0, &st);
std::vector<unsigned char> data;
data.clear();
@@ -169,152 +252,331 @@ Handle<Value> ZipFile::readFileSync(const Arguments& args)
return scope.Close(retbuf->handle_);
}
-typedef struct {
- ZipFile* zf;
- struct zip *za;
- std::string name;
- bool error;
- std::string error_name;
- std::vector<unsigned char> data;
- Persistent<Function> cb;
-} closure_t;
+/* zipfile.read(buffer, pos, len, cb) -> cb(bytesRead, error) */
+Handle<Value> ZipFile::Read(const Arguments& args)
+{
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+ if (!Buffer::HasInstance(args[0])) {
+ return ThrowException(Exception::Error(
+ String::New("First argument needs to be a buffer")));
+ }
+ Local<Object> buffer_obj = args[0]->ToObject();
+ char *buffer_data = Buffer::Data(buffer_obj);
+ size_t buffer_length = Buffer::Length(buffer_obj);
+
+ zip_uint64_t off = args[1]->Int32Value();
+ if (off >= buffer_length) {
+ return ThrowException(Exception::Error(
+ String::New("Offset is out of bounds")));
+ }
+
+ zip_uint64_t len = args[2]->Int32Value();
+ if (off + len > buffer_length) {
+ return ThrowException(Exception::Error(
+ String::New("Length is extends beyond buffer")));
+ }
+
+ Local<Value> cb = args[3];
+ if (!cb->IsFunction())
+ return ThrowException(Exception::Error(
+ String::New("Fourth argument should be a callback function.")));
+
+ read_closure_t *closure = new read_closure_t();
+ closure->zf = zf;
+ closure->read = 0;
+ closure->data = buffer_data+off;
+ closure->len = len;
+ closure->cb = Persistent<Function>::New(Handle<Function>::Cast(args[3]));
-Handle<Value> ZipFile::readFile(const Arguments& args)
+ eio_custom(EIO_Read, EIO_PRI_DEFAULT, EIO_AfterRead, closure);
+ zf->Ref();
+ ev_ref(EV_DEFAULT_UC);
+
+ return Undefined();
+}
+
+int ZipFile::EIO_Read(eio_req *req)
{
- HandleScope scope;
+ read_closure_t *closure = static_cast<read_closure_t *>(req->data);
- if (args.Length() < 2)
- return ThrowException(Exception::TypeError(
- String::New("requires two arguments, the name of a file and a callback")));
-
- // first arg must be name
- if(!args[0]->IsString())
- return ThrowException(Exception::TypeError(
- String::New("first argument must be a file name inside the zip")));
+ closure->read = zip_fread(closure->zf->file, closure->data, closure->len);
+
+ return 0;
+}
+
+int ZipFile::EIO_AfterRead(eio_req *req)
+{
+ read_closure_t *closure = static_cast<read_closure_t *>(req->data);
+
+ Handle<Value> argv[2];
+
+ if (closure->read < 0) {
+ std::stringstream s;
+ s << "Error while reading zip file: " << zip_file_strerror(closure->zf->file) << "\n";
+ argv[0] = Exception::Error(String::New(s.str().c_str()));
+ argv[1] = Undefined();
+ } else {
+ argv[0] = Undefined();
+ argv[1] = Integer::New(closure->read);
+ }
+
+ closure->cb->Call(Context::GetCurrent()->Global(), 2, argv);
+
+ closure->zf->Unref();
+ closure->cb.Dispose();
+ delete closure;
- // last arg must be function callback
- if (!args[args.Length()-1]->IsFunction())
- return ThrowException(Exception::TypeError(
- String::New("last argument must be a callback function")));
-
- std::string name = TOSTR(args[0]);
-
+ ev_unref(EV_DEFAULT_UC);
+
+ return 0;
+}
+
+/* zipfile.addFile(nameInArchive, name, offset, len) */
+Handle<Value> ZipFile::Add_File(const Arguments& args)
+{
ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
- closure_t *closure = new closure_t();
+ struct zip_source *source;
- // libzip is not threadsafe so we cannot use the zf->archive_
- // instead we open a new zip archive for each thread
- struct zip *za;
- int err;
- char errstr[1024];
- if ((za=zip_open(zf->file_name_.c_str() , 0, &err)) == NULL) {
- zip_error_to_str(errstr, sizeof(errstr), err, errno);
- std::stringstream s;
- s << "cannot open file: " << zf->file_name_ << " error: " << errstr << "\n";
- zip_close(za);
- return ThrowException(Exception::Error(
- String::New(s.str().c_str())));
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
+
+ if (!args[0]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a file name.")));
+ std::string archive_file = TOSTR(args[0]);
+
+ std::string name;
+ if (args[1]->IsUndefined())
+ name = archive_file;
+ else
+ if (!args[1]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a file name.")));
+ name = TOSTR(args[1]);
+
+ zip_int64_t off;
+ if (args[2]->IsUndefined())
+ off = 0;
+ else
+ off = args[2]->Int32Value();
+
+ zip_int64_t len;
+ if (args[3]->IsUndefined())
+ len = -1;
+ else
+ len = args[3]->Int32Value();
+
+ source = zip_source_file(zf->archive, name.c_str(), off, len);
+ if (source == NULL) {
+ std::stringstream s;
+ s << "Error while adding file " << name << " to zip archive: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
+
+ int ret = zip_add(zf->archive, archive_file.c_str(), source);
+ if (ret < 0) {
+ zip_source_free(source);
+ std::stringstream s;
+ s << "Error while adding file " << name << " to zip archive: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
}
- closure->zf = zf;
- closure->za = za;
- closure->error = false;
- closure->name = name;
- closure->cb = Persistent<Function>::New(Handle<Function>::Cast(args[args.Length()-1]));
- eio_custom(EIO_ReadFile, EIO_PRI_DEFAULT, EIO_AfterReadFile, closure);
- ev_ref(EV_DEFAULT_UC);
- zf->Ref();
+ zf->GetNames();
+
return Undefined();
}
-
-int ZipFile::EIO_ReadFile(eio_req *req)
+/* zipfile.replaceFile(nameInArchive, name, offset, len) */
+Handle<Value> ZipFile::Replace_File(const Arguments& args)
{
- closure_t *closure = static_cast<closure_t *>(req->data);
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+
+ struct zip_source *source;
- struct zip_file *zf_ptr=NULL;
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
+
+ if (!args[0]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a file name.")));
+ std::string archive_file = TOSTR(args[0]);
+
+ std::string name;
+ if (args[1]->IsUndefined())
+ name = archive_file;
+ else
+ if (!args[1]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a file name.")));
+ name = TOSTR(args[1]);
+
+ zip_int64_t off;
+ if (args[2]->IsUndefined())
+ off = 0;
+ else
+ off = args[2]->Int32Value();
+
+ zip_int64_t len;
+ if (args[3]->IsUndefined())
+ len = -1;
+ else
+ len = args[3]->Int32Value();
int idx = -1;
- std::vector<std::string>::iterator it = std::find(closure->zf->names_.begin(),
- closure->zf->names_.end(),
- closure->name);
- if (it!=closure->zf->names_.end()) {
- idx = distance(closure->zf->names_.begin(), it);
+ std::vector<std::string>::iterator it = std::find(zf->names.begin(), zf->names.end(), archive_file);
+ if (it!=zf->names.end()) {
+ idx = distance(zf->names.begin(), it);
}
if (idx == -1) {
std::stringstream s;
- s << "No file found by the name of: '" << closure->name << "\n";
- closure->error = true;
- closure->error_name = s.str();
-
- } else {
+ s << "No file found by the name of: '" << archive_file << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
- if ((zf_ptr = zip_fopen_index(closure->za, idx, 0)) == NULL) {
- std::stringstream s;
- s << "cannot open file #" << idx << " in "
- << closure->name << ": archive error: " << zip_strerror(closure->za) << "\n";
- closure->error = true;
- closure->error_name = s.str();
+ source = zip_source_file(zf->archive, name.c_str(), off, len);
+ if (source == NULL) {
+ std::stringstream s;
+ s << "Error while replacing file " << name << " in zip archive: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
+ }
- } else {
-
- struct zip_stat st;
- zip_stat_index(closure->za, idx, 0, &st);
- closure->data.clear();
- closure->data.resize( st.size );
-
- int result = 0;
- result = (int)zip_fread( zf_ptr, reinterpret_cast<void*> (&closure->data[0]), closure->data.size() );
-
- if (result < 0) {
- std::stringstream s;
- s << "error reading file #" << idx << " in "
- << closure->name << ": archive error: " << zip_file_strerror(zf_ptr) << "\n";
- closure->error = true;
- closure->error_name = s.str();
- }
- }
+ int ret = zip_replace(zf->archive, idx, source);
+ if (ret < 0) {
+ zip_source_free(source);
+ std::stringstream s;
+ s << "Error while replacing file " << name << " in zip archive: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
}
- zip_fclose(zf_ptr);
- return 0;
+ return Undefined();
}
-int ZipFile::EIO_AfterReadFile(eio_req *req)
+/* zipfile.addDirectory(name) */
+Handle<Value> ZipFile::Add_Directory(const Arguments& args)
{
- HandleScope scope;
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
- closure_t *closure = static_cast<closure_t *>(req->data);
- ev_unref(EV_DEFAULT_UC);
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
- TryCatch try_catch;
-
- if (closure->error) {
- Local<Value> argv[1] = { Exception::Error(String::New(closure->error_name.c_str())) };
- closure->cb->Call(Context::GetCurrent()->Global(), 1, argv);
- } else {
- #if NODE_VERSION_AT_LEAST(0,3,0)
- node::Buffer *retbuf = Buffer::New((char *)&closure->data[0],closure->data.size());
- #else
- node::Buffer *retbuf = Buffer::New(closure->data.size());
- std::memcpy(retbuf->data(), (char *)&closure->data[0], closure->data.size());
- #endif
- Local<Value> argv[2] = { Local<Value>::New(Null()), Local<Value>::New(retbuf->handle_) };
- closure->cb->Call(Context::GetCurrent()->Global(), 2, argv);
+ if (!args[0]->IsString())
+ return ThrowException(Exception::TypeError(
+ String::New("Argument must be a directory name.")));
+ std::string directory = TOSTR(args[0]);
+
+ int ret = zip_add_dir(zf->archive, directory.c_str());
+ if (ret < 0) {
+ std::stringstream s;
+ s << "Error while adding directory " << directory << " to zip archive: " << zip_strerror(zf->archive) << "\n";
+ return ThrowException(Exception::Error(String::New(s.str().c_str())));
}
- if (try_catch.HasCaught()) {
- FatalException(try_catch);
- //try_catch.ReThrow();
+ zf->GetNames();
+
+ return Undefined();
+}
+
+void *ZipFile::Save_Thread(void *data) {
+ save_closure_t *save = (save_closure_t *)data;
+
+ int ret = zip_close(save->zf->archive);
+
+ pthread_mutex_lock(&save->mutex);
+
+ if (ret < 0) {
+ std::stringstream s;
+ s << "Error while saving in zip archive: " << zip_strerror(save->zf->archive) << "\n";
+ save->error = new std::string(s.str());
+ } else {
+ // We assume that reopening is ok here..
+ int err;
+ save->zf->archive = zip_open(save->zf->file_name.c_str(), ZIP_CREATE, &err);
}
- closure->zf->Unref();
- closure->cb.Dispose();
- delete closure;
- return 0;
+ save->done = true;
+
+ pthread_mutex_unlock(&save->mutex);
+
+ ev_async_send(EV_DEFAULT_UC_ &notifier);
+
+ return NULL;
}
+/* zipfile.save(callback) */
+Handle<Value> ZipFile::Save(const Arguments& args)
+{
+ ZipFile* zf = ObjectWrap::Unwrap<ZipFile>(args.This());
+
+ if (zf->Busy())
+ return ThrowException(Exception::Error(String::New("Zipfile already in use..")));
+
+ Local<Value> cb = args[0];
+ if (!cb->IsFunction())
+ return ThrowException(Exception::Error(
+ String::New("Second argument should be a callback function.")));
+
+ save_closure_t *save = new save_closure_t;
+
+ save->done = false;
+ save->zf = zf;
+ save->error = NULL;
+ save->save_cb = Persistent<Function>::New(Handle<Function>::Cast(cb));
+ pthread_mutex_init(&save->mutex, NULL);
+
+ saving_closures.insert(save);
+
+ zf->saving = true;
+ zf->Ref();
+
+ ev_ref(EV_DEFAULT_UC);
+ pthread_create(&save->thread, NULL, Save_Thread, save);
+
+ return Undefined();
+}
+
+// callback that runs javascript in main thread
+void ZipFile::Save_Callback(EV_P_ ev_async *watcher, int revents)
+{
+ std::set<save_closure_t *>::iterator it;
+ save_closure_t *save;
+ Handle<Value> argv[1];
+
+ assert(watcher == &notifier);
+ assert(revents == EV_ASYNC);
+
+ std::set<save_closure_t *> remove;
+
+ for(it = saving_closures.begin(); it != saving_closures.end(); it++) {
+ save = *it;
+ pthread_mutex_lock(&save->mutex);
+ if (save->done) {
+ save->zf->saving = false;
+
+ if (save->error != NULL)
+ argv[0] = Exception::Error(String::New(save->error->c_str()));
+ else
+ argv[0] = Undefined();
+
+ save->save_cb->Call(Context::GetCurrent()->Global(), 1, argv);
+ remove.insert(save);
+ }
+ pthread_mutex_unlock(&save->mutex);
+ }
+
+ // Now, let's do some janitoring !
+ for(it = remove.begin(); it != remove.end(); it++) {
+ save = *it;
+ // Now, let's do some janitoring !
+ saving_closures.erase(save);
+ if (save->error != NULL) delete save->error;
+ ev_unref(EV_DEFAULT_UC);
+ save->zf->Unref();
+ save->save_cb.Dispose();
+ pthread_mutex_destroy(&save->mutex);
+ delete save;
+ }
+}
View
51 src/node_zipfile.hpp
@@ -9,6 +9,9 @@
#include <string>
#include <vector>
+// Threads
+#include <pthread.h>
+
// libzip
#include <zlib.h>
#include <errno.h>
@@ -23,25 +26,55 @@ class ZipFile: public node::ObjectWrap {
static Persistent<FunctionTemplate> constructor;
static void Initialize(Handle<Object> target);
static Handle<Value> New(const Arguments &args);
-
static Handle<Value> get_prop(Local<String> property,
const AccessorInfo& info);
// Sync
+ static Handle<Value> Open(const Arguments& args);
static Handle<Value> readFileSync(const Arguments& args);
-
+ static Handle<Value> Close(const Arguments& args);
+ static Handle<Value> Add_File(const Arguments& args);
+ static Handle<Value> Replace_File(const Arguments& args);
+ static Handle<Value> Add_Directory(const Arguments& args);
+
// Async
- static Handle<Value> readFile(const Arguments& args);
- static int EIO_ReadFile(eio_req *req);
- static int EIO_AfterReadFile(eio_req *req);
-
+ static Handle<Value> Read(const Arguments& args);
+ static int EIO_Read(eio_req *req);
+ static int EIO_AfterRead(eio_req *req);
+ static Handle<Value> Save(const Arguments& args);
+ static void *Save_Thread(void *data);
+ static void Save_Callback(EV_P_ ev_async *watcher, int revents);
ZipFile(std::string const& file_name);
+ protected:
+ void GetNames();
+ std::string const file_name;
+ struct zip *archive;
+ struct zip_file *file;
+ int file_index;
+ bool saving;
+
private:
~ZipFile();
- std::string const file_name_;
- struct zip *archive_;
- std::vector<std::string> names_;
+ bool Busy();
+ std::vector<std::string> names;
};
+typedef struct {
+ ZipFile *zf;
+ zip_uint64_t read;
+ zip_uint64_t len;
+ char *data;
+ Persistent<Function> cb;
+} read_closure_t;
+
+typedef struct {
+ ZipFile *zf;
+ pthread_mutex_t mutex;
+ pthread_t thread;
+ bool done;
+ Persistent<Function> save_cb;
+ std::string *error;
+} save_closure_t;
+
#endif
View
80 test.js
@@ -27,11 +27,11 @@ var zf = new zipfile.ZipFile('./data/folder.zip');
assert.equal(zf.count, 3); // one folder, two files
assert.deepEqual(zf.names, ['folder/', 'folder/one.txt', 'folder/two.txt']);
+var data = zf.readFileSync('folder/one.txt')
assert.throws(function() { zf.readFileSync('foo')});
var zf = new zipfile.ZipFile('./data/world_merc.zip');
-
function mkdirP(p, mode, f) {
var cb = f || function() {};
// if relative
@@ -49,47 +49,57 @@ function mkdirP(p, mode, f) {
});
}
-// test writing with Sync reading method
-// and sync node writing functions
-zf.names.forEach(function(name) {
- var uncompressed = path.join('/tmp/sync', name);
- var dirname = path.dirname(uncompressed);
- mkdirP(dirname, 0755 , function(err) {
- if (err && err.errno != constants.EEXIST) throw err;
- if (path.extname(name)) {
- var buffer = zf.readFileSync(name);
- fd = fs.openSync(uncompressed, 'w');
- fs.writeSync(fd, buffer, 0, buffer.length, null);
- fs.closeSync(fd);
- }
- });
-});
-
-
// test writing with Async reading method
// and async node writing functions
-zf.names.forEach(function(name) {
+names = zf.names.slice(0);
+function asyncDecompress(cb) {
+ if (names.length == 0) {
+ cb();
+ return;
+ }
+ var name = names.shift();
+ console.log("Decompressing " + name);
var uncompressed = path.join('/tmp/async', name);
var dirname = path.dirname(uncompressed);
mkdirP(dirname, 0755 , function(err) {
if (err && err.errno != constants.EEXIST) throw err;
if (path.extname(name)) {
- zf.readFile(name, function(err, buffer) {
- if (err) throw err;
- fs.open(uncompressed, 'w', 0755, function(err,fd) {
- if (err) throw err;
- fs.write(fd, buffer, 0, buffer.length, null, function(err,written) {
- if (err) throw err;
- // written is number of bytes written
- assert.ok(written > 0);
- fs.close(fd, function(err) {
- if (err) throw err;
- });
- });
- });
- });
+ zip_stream = zf.createReadStream(name);
+ file_stream = fs.createWriteStream(uncompressed);
+ zip_stream.pipe(file_stream);
+ zip_stream.on('close', function () { asyncDecompress(cb)});
}
});
-});
+};
+
+try { fs.unlinkSync('/tmp/compress.zip'); }
+catch (err) {}
+var new_zf = new zipfile.ZipFile('/tmp/compress.zip');
+
+function asyncCompress() {
+ zf.names.forEach(function (name) {
+ console.log("Adding " + name);
+ var uncompressed = path.join('/tmp/async', name);
+ var compressed = path.join('tmp/async', name);
+ new_zf.addFile(compressed, uncompressed);
+ });
+ new_zf.save(function (err) {
+ if (err)
+ console.log("Error while saving file: " + err);
+ else {
+ new_zf.names.forEach(function (name) {
+ console.log("Replacing " + name);
+ var uncompressed = '/' + name;
+ new_zf.replaceFile(name, uncompressed);
+ });
+ new_zf.addDirectory("tmp");
+ new_zf.addDirectory("tmp/async");
+ new_zf.save(function (err) {
+ if (err) console.log("Error while saving file: " + err);
+ else console.log("All test passed!");
+ });
+ }
+ });
+}
-console.log('All tests pass...');
+asyncDecompress(asyncCompress);
Something went wrong with that request. Please try again.