Skip to content
Browse files

Zlib bindings ftw.

This is a squashed commit.  For posterity, the original commit messages
are below.

* zlib initial stub
* Doing deflate synchronously and slowly
* Do deflate on the uv_work thread pool
* Print the strm.msg if set
* Assert that the deflateInit leaves things Z_OK
* Use a template class for Deflate/Inflate
* Add Inflate
* Implement gzip, gunzip, deflateraw, and inflateraw
* Configurable chunk size.  Almost working.  Still some garbage, though.
* Remove node_zlib.h. Use enum for modes. Null out memory blocks
* Macro out the copy-pasta
* Keep a ref to the buffer, so that it doesn't get corrupted
* checkpoint - lib/zlib.js stub
* Working entirely
* s/Flate/Zlib/g Makes more sense this way
* Add Z_FIXED as an acceptable strategy
* Test for zlib compression and decompression
* Correction s/opts/options/ typeo, and add constants
* Doc
* Do more in js, use _ on members, no throwing in C++
* Add Unzip, which can handle Gzip or Deflate by detecting headers
* Move a bunch more logic into JS
* Remove err_ member
* Clarify which options are not relevant for inflate
* Style issues
* Use node::MakeCallback
* .h style issue
* Pass namespace lint complaint
* Correct docs about defaults
* Reduce default chunk to 16K
* Namespaces, gah
* Build against staticly linked zlib
  • Loading branch information...
1 parent 335f07c commit ffce52b2bc91ed260153dc70715b50d38aec25e9 @isaacs committed
Showing with 1,035 additions and 7 deletions.
  1. +95 −0 doc/api/zlib.markdown
  2. +325 −0 lib/zlib.js
  3. +3 −0 node.gyp
  4. +1 −0 src/node_extensions.h
  5. +359 −0 src/node_zlib.cc
  6. +221 −0 test/simple/test-zlib.js
  7. +31 −7 wscript
View
95 doc/api/zlib.markdown
@@ -0,0 +1,95 @@
+## Zlib
+
+You can access this module with:
+
+ var zlib = require('zlib');
+ var gzip = zlib.createGzip();
+ var fs = require('fs');
+ var inp = fs.createReadStream('input.txt');
+ var out = fs.createWriteStream('input.txt.gz');
+
+ inp.pipe(gzip).pipe(out);
+
+This provides bindings to Gzip/Gunzip, Deflate/Inflate, and
+DeflateRaw/InflateRaw classes. Each class takes the same options, and
+is a readable/writable Stream.
+
+### Constants
+
+All of the constants defined in zlib.h are also defined on
+`require('zlib')`. They are described in more detail in the zlib
+documentation. See <http://zlib.net/manual.html#Constants>
+for more details.
+
+### zlib.Gzip
+
+Compress data using gzip.
+
+### zlib.Gunzip
+
+Decompress a gzip stream.
+
+### zlib.Deflate
+
+Compress data using deflate.
+
+### zlib.Inflate
+
+Decompress a deflate stream.
+
+### zlib.DeflateRaw
+
+Compress data using deflate, and do not append a zlib header.
+
+### zlib.InflateRaw
+
+Decompress a raw deflate stream.
+
+### zlib.Unzip
+
+Decompress either a Gzip- or Deflate-compressed stream by auto-detecting
+the header.
+
+### Options
+
+Each class takes an options object. All options are optional.
+
+Note that some options are only
+relevant when compressing, and are ignored by the decompression classes.
+
+* chunkSize (default: 16*1024)
+* windowBits
+* level (compression only)
+* memLevel (compression only)
+* strategy (compression only)
+
+See the description of `deflateInit2` and `inflateInit2` at
+<http://zlib.net/manual.html#Advanced> for more information on these.
+
+### Memory Usage Tuning
+
+From `zlib/zconf.h`, modified to node's usage:
+
+The memory requirements for deflate are (in bytes):
+
+ (1 << (windowBits+2)) + (1 << (memLevel+9))
+
+that is: 128K for windowBits=15 + 128K for memLevel = 8
+(default values) plus a few kilobytes for small objects.
+
+For example, if you want to reduce
+the default memory requirements from 256K to 128K, set the options to:
+
+ { windowBits: 14, memLevel: 7 }
+
+Of course this will generally degrade compression (there's no free lunch).
+
+The memory requirements for inflate are (in bytes)
+
+ 1 << windowBits
+
+that is, 32K for windowBits=15 (default value) plus a few kilobytes
+for small objects.
+
+This is in addition to a single internal output slab buffer of size
+`chunkSize`, which defaults to 16K.
View
325 lib/zlib.js
@@ -0,0 +1,325 @@
+// Copyright Joyent, Inc. and other Node contributors.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a
+// copy of this software and associated documentation files (the
+// "Software"), to deal in the Software without restriction, including
+// without limitation the rights to use, copy, modify, merge, publish,
+// distribute, sublicense, and/or sell copies of the Software, and to permit
+// persons to whom the Software is furnished to do so, subject to the
+// following conditions:
+//
+// The above copyright notice and this permission notice shall be included
+// in all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN
+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+var binding = process.binding('zlib');
+var util = require('util');
+var stream = require('stream');
+
+// zlib doesn't provide these, so kludge them in following the same
+// const naming scheme zlib uses.
+binding.Z_MIN_WINDOWBITS = 8;
+binding.Z_MAX_WINDOWBITS = 15;
+binding.Z_DEFAULT_WINDOWBITS = 15;
+
+// fewer than 64 bytes per chunk is stupid.
+// technically it could work with as few as 8, but even 64 bytes
+// is absurdly low. Usually a MB or more is best.
+binding.Z_MIN_CHUNK = 64;
+binding.Z_MAX_CHUNK = Infinity;
+binding.Z_DEFAULT_CHUNK = (16 * 1024);
+
+binding.Z_MIN_MEMLEVEL = 1;
+binding.Z_MAX_MEMLEVEL = 9;
+binding.Z_DEFAULT_MEMLEVEL = 8;
+
+binding.Z_MIN_LEVEL = -1;
+binding.Z_MAX_LEVEL = 9;
+binding.Z_DEFAULT_LEVEL = binding.Z_DEFAULT_COMPRESSION;
+
+// expose all the zlib constants
+Object.keys(binding).forEach(function(k) {
+ if (k.match(/^Z/)) exports[k] = binding[k];
+});
+
+
+exports.Deflate = Deflate;
+exports.Inflate = Inflate;
+exports.Gzip = Gzip;
+exports.Gunzip = Gunzip;
+exports.DeflateRaw = DeflateRaw;
+exports.InflateRaw = InflateRaw;
+exports.Unzip = Unzip;
+
+exports.createDeflate = function (o) { return new Deflate(o); };
+exports.createInflate = function (o) { return new Inflate(o); };
+exports.createDeflateRaw = function (o) { return new DeflateRaw(o); };
+exports.createInflateRaw = function (o) { return new InflateRaw(o); };
+exports.createGzip = function (o) { return new Gzip(o); };
+exports.createGunzip = function (o) { return new Gunzip(o); };
+exports.createUnzip = function (o) { return new Unzip(o); };
+
+
+
+// generic zlib
+// minimal 2-byte header
+function Deflate(opts) {
+ if (!(this instanceof Deflate)) return new Deflate(opt);
+ Zlib.call(this, opts, binding.Deflate);
+}
+
+function Inflate(opts) {
+ if (!(this instanceof Inflate)) return new Inflate(opts);
+ Zlib.call(this, opts, binding.Inflate);
+}
+
+
+
+// gzip - bigger header, same deflate compression
+function Gzip(opts) {
+ if (!(this instanceof Gzip)) return new Gzip(opts);
+ Zlib.call(this, opts, binding.Gzip);
+}
+
+function Gunzip(opts) {
+ if (!(this instanceof Gunzip)) return new Gunzip(opts);
+ Zlib.call(this, opts, binding.Gunzip);
+}
+
+
+
+// raw - no header
+function DeflateRaw(opts) {
+ if (!(this instanceof DeflateRaw)) return new DeflateRaw(opts);
+ Zlib.call(this, opts, binding.DeflateRaw);
+}
+
+function InflateRaw(opts) {
+ if (!(this instanceof InflateRaw)) return new InflateRaw(opts);
+ Zlib.call(this, opts, binding.InflateRaw);
+}
+
+
+// auto-detect header.
+function Unzip(opts) {
+ if (!(this instanceof Unzip)) return new Unzip(opts);
+ Zlib.call(this, opts, binding.Unzip);
+}
+
+
+// the Zlib class they all inherit from
+// This thing manages the queue of requests, and returns
+// true or false if there is anything in the queue when
+// you call the .write() method.
+
+function Zlib(opts, Binding) {
+ this._opts = opts = opts || {};
+ this._queue = [];
+ this._processing = false;
+ this._ended = false;
+ this.readable = true;
+ this.writable = true;
+ this._flush = binding.Z_NO_FLUSH;
+
+ if (opts.chunkSize) {
+ if (opts.chunkSize < exports.Z_MIN_CHUNK ||
+ opts.chunkSize > exports.Z_MAX_CHUNK) {
+ throw new Error("Invalid chunk size: "+opts.chunkSize);
+ }
+ }
+
+ if (opts.windowBits) {
+ if (opts.windowBits < exports.Z_MIN_WINDOWBITS ||
+ opts.windowBits > exports.Z_MAX_WINDOWBITS) {
+ throw new Error("Invalid windowBits: "+opts.windowBits);
+ }
+ }
+
+ if (opts.level) {
+ if (opts.level < exports.Z_MIN_LEVEL ||
+ opts.level > exports.Z_MAX_LEVEL) {
+ throw new Error("Invalid compression level: "+opts.level);
+ }
+ }
+
+ if (opts.memLevel) {
+ if (opts.memLevel < exports.Z_MIN_MEMLEVEL ||
+ opts.memLevel > exports.Z_MAX_MEMLEVEL) {
+ throw new Error("Invalid memLevel: "+opts.memLevel);
+ }
+ }
+
+ if (opts.strategy) {
+ if (opts.strategy != exports.Z_FILTERED &&
+ opts.strategy != exports.Z_HUFFMAN_ONLY &&
+ opts.strategy != exports.Z_RLE &&
+ opts.strategy != exports.Z_FIXED &&
+ opts.strategy != exports.Z_DEFAULT_STRATEGY) {
+ throw new Error("Invalid strategy: "+opts.strategy);
+ }
+ }
+
+ this._binding = new Binding();
+ this._binding.init(opts.windowBits || exports.Z_DEFAULT_WINDOWBITS,
+ opts.level || exports.Z_DEFAULT_COMPRESSION,
+ opts.memLevel || exports.Z_DEFAULT_MEMLEVEL,
+ opts.strategy || exports.Z_DEFAULT_STRATEGY);
+
+ this._chunkSize = opts.chunkSize || exports.Z_DEFAULT_CHUNK;
+ this._buffer = new Buffer(this._chunkSize);
+ this._offset = 0;
+ var self = this;
+
+ this._binding.onData = function(c) {
+ self.emit('data', c);
+ };
+
+ this._binding.onEnd = function() {
+ self.emit('end');
+ };
+}
+
+util.inherits(Zlib, stream.Stream);
+
+Zlib.prototype.write = function write(chunk, cb) {
+ if (this._ended) {
+ return this.emit('error', new Error('Cannot write after end'));
+ }
+
+ if (arguments.length === 1 && typeof chunk === 'function') {
+ cb = chunk;
+ chunk = null;
+ }
+
+ if (!chunk) {
+ chunk = null;
+ } else if (typeof chunk === 'string') {
+ chunk = new Buffer(chunk);
+ } else if (!Buffer.isBuffer(chunk)) {
+ return this.emit('error', new Error('Invalid argument'));
+ }
+
+
+ var empty = this._queue.length === 0;
+
+ this._queue.push([chunk, cb]);
+ this._process();
+ if (!empty) {
+ this._needDrain = true;
+ }
+ return empty;
+};
+
+Zlib.prototype.flush = function flush(cb) {
+ this._flush = binding.Z_SYNC_FLUSH;
+ return this.write(cb);
+};
+
+Zlib.prototype.end = function end(chunk, cb) {
+ var self = this;
+ this._ending = true;
+ var ret = this.write(chunk, function() {
+ self.emit('end');
+ if (cb) cb();
+ });
+ this._ended = true;
+ return ret;
+};
+
+Zlib.prototype._process = function() {
+ if (this._processing || this._paused) return;
+
+ if (this._queue.length === 0) {
+ if (this._needDrain) {
+ this._needDrain = false;
+ this.emit('drain');
+ }
+ return;
+ }
+
+ var req = this._queue.shift();
+ var cb = req.pop();
+ var chunk = req.pop();
+
+ if (this._ending && this._queue.length === 0) {
+ this._flush = binding.Z_FINISH;
+ }
+
+ var self = this;
+ var availInBefore = chunk && chunk.length;
+ var availOutBefore = this._chunkSize - this._offset;
+
+ var inOff = 0;
+ var req = this._binding.write(this._flush,
+ chunk, // in
+ inOff, // in_off
+ availInBefore, // in_len
+ this._buffer, // out
+ this._offset, //out_off
+ availOutBefore); // out_len
+
+ req.buffer = chunk;
+ req.callback = callback;
+ this._processing = req;
+
+ function callback(availInAfter, availOutAfter) {
+ var have = self.chunkSize - availOutAfter;
+ if (have > 0) {
+ var out = self._buffer.slice(self._offset, self._offset + have);
+ self._offset += have;
+ }
+
+ // XXX Maybe have a 'min buffer' size so we don't dip into the
+ // thread pool with only 1 byte available or something?
+ if (availOutAfter === 0 || self._offset >= self._chunkSize) {
+ self._offset = 0;
+ self._buffer = new Buffer(self._chunkSize);
+ }
+
+ if (availOutAfter === 0) {
+ // Not actually done. Need to reprocess.
+ inOff += (availInBefore - availInAfter);
+ var newReq = self._binding.write(self._flush,
+ chunk,
+ inOff,
+ availInAfter,
+ self._buffer,
+ self._offset,
+ self._chunkSize);
+ newReq.callback = callback; // this same function
+ newReq.buffer = chunk;
+ self._processing = newReq;
+ return;
+ }
+
+ // finished with the chunk.
+ self._processing = false;
+ if (cb) cb();
+ self._process();
+ }
+};
+
+Zlib.prototype.pause = function() {
+ this._paused = true;
+ this.emit('pause');
+};
+
+Zlib.prototype.resume = function() {
+ this._paused = false;
+ this.emit('resume');
+};
+
+util.inherits(Deflate, Zlib);
+util.inherits(Inflate, Zlib);
+util.inherits(Gzip, Zlib);
+util.inherits(Gunzip, Zlib);
+util.inherits(DeflateRaw, Zlib);
+util.inherits(InflateRaw, Zlib);
+util.inherits(Unzip, Zlib);
View
3 node.gyp
@@ -49,6 +49,7 @@
'lib/url.js',
'lib/util.js',
'lib/vm.js',
+ 'lib/zlib.js',
],
},
@@ -86,6 +87,7 @@
'src/node_os.cc',
'src/node_script.cc',
'src/node_string.cc',
+ 'src/node_zlib.cc',
'src/pipe_wrap.cc',
'src/stdio_wrap.cc',
'src/stream_wrap.cc',
@@ -114,6 +116,7 @@
'src/node_stdio.h',
'src/node_string.h',
'src/node_version.h',
+ 'src/node_zlib.h',
'src/pipe_wrap.h',
'src/platform.h',
'src/req_wrap.h',
View
1 src/node_extensions.h
@@ -40,6 +40,7 @@ NODE_EXT_LIST_ITEM(node_signal_watcher)
#endif
NODE_EXT_LIST_ITEM(node_stdio)
NODE_EXT_LIST_ITEM(node_os)
+NODE_EXT_LIST_ITEM(node_zlib)
// libuv rewrite
NODE_EXT_LIST_ITEM(node_timer_wrap)
View
359 src/node_zlib.cc
@@ -0,0 +1,359 @@
+// Copyright Joyent, Inc. and other Node contributors.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a
+// copy of this software and associated documentation files (the
+// "Software"), to deal in the Software without restriction, including
+// without limitation the rights to use, copy, modify, merge, publish,
+// distribute, sublicense, and/or sell copies of the Software, and to permit
+// persons to whom the Software is furnished to do so, subject to the
+// following conditions:
+//
+// The above copyright notice and this permission notice shall be included
+// in all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN
+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+#include <v8.h>
+#include <errno.h>
+#include <string.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <zlib.h>
+
+#include <node.h>
+#include <node_buffer.h>
+#include <req_wrap.h>
+
+
+
+namespace node {
+using namespace v8;
+
+// write() returns one of these, and then calls the cb() when it's done.
+typedef ReqWrap<uv_work_t> WorkReqWrap;
+
+static Persistent<String> ondata_sym;
+static Persistent<String> callback_sym;
+
+enum node_zlib_mode {
+ DEFLATE = 1,
+ INFLATE,
+ GZIP,
+ GUNZIP,
+ DEFLATERAW,
+ INFLATERAW,
+ UNZIP
+};
+
+template <node_zlib_mode mode> class ZCtx;
+
+
+void InitZlib(v8::Handle<v8::Object> target);
+
+
+/**
+ * Deflate/Inflate
+ */
+template <node_zlib_mode mode> class ZCtx : public ObjectWrap {
+ public:
+
+ ZCtx() : ObjectWrap() {
+ }
+
+ ~ZCtx() {
+ if (mode == DEFLATE || mode == GZIP || mode == DEFLATERAW) {
+ (void)deflateEnd(&strm_);
+ } else if (mode == INFLATE || mode == GUNZIP || mode == INFLATERAW) {
+ (void)inflateEnd(&strm_);
+ }
+ }
+
+ // write(flush, in, in_off, in_len, out, out_off, out_len)
+ static Handle<Value>
+ Write(const Arguments& args) {
+ assert(args.Length() == 7);
+
+ ZCtx<mode> *ctx = ObjectWrap::Unwrap< ZCtx<mode> >(args.This());
+ assert(ctx->init_done_ && "write before init");
+
+ unsigned int flush = args[0]->Uint32Value();
+ Bytef *in;
+ Bytef *out;
+ size_t in_off, in_len, out_off, out_len;
+
+ if (args[1]->IsNull()) {
+ // just a flush
+ Bytef nada[1] = { 0 };
+ in = nada;
+ in_len = 0;
+ in_off = 0;
+ } else {
+ assert(Buffer::HasInstance(args[1]));
+ Local<Object> in_buf;
+ in_buf = args[1]->ToObject();
+ in_off = (size_t)args[2]->Uint32Value();
+ in_len = (size_t)args[3]->Uint32Value();
+
+ assert(in_off + in_len <= Buffer::Length(in_buf));
+ in = reinterpret_cast<Bytef *>(Buffer::Data(in_buf) + in_off);
+ }
+
+ assert(Buffer::HasInstance(args[4]));
+ Local<Object> out_buf = args[4]->ToObject();
+ out_off = (size_t)args[5]->Uint32Value();
+ out_len = (size_t)args[6]->Uint32Value();
+ assert(out_off + out_len <= Buffer::Length(out_buf));
+ out = reinterpret_cast<Bytef *>(Buffer::Data(out_buf) + out_off);
+
+ WorkReqWrap *req_wrap = new WorkReqWrap();
+
+ req_wrap->data_ = ctx;
+ ctx->strm_.avail_in = in_len;
+ ctx->strm_.next_in = &(*in);
+ ctx->strm_.avail_out = out_len;
+ ctx->strm_.next_out = out;
+ ctx->flush_ = flush;
+
+ // set this so that later on, I can easily tell how much was written.
+ ctx->chunk_size_ = out_len;
+
+ // build up the work request
+ uv_work_t* work_req = new uv_work_t();
+ work_req->data = req_wrap;
+
+ uv_queue_work(uv_default_loop(),
+ work_req,
+ ZCtx<mode>::Process,
+ ZCtx<mode>::After);
+
+ req_wrap->Dispatched();
+
+ return req_wrap->object_;
+ }
+
+
+ // thread pool!
+ // This function may be called multiple times on the uv_work pool
+ // for a single write() call, until all of the input bytes have
+ // been consumed.
+ static void
+ Process(uv_work_t* work_req) {
+ WorkReqWrap *req_wrap = reinterpret_cast<WorkReqWrap *>(work_req->data);
+ ZCtx<mode> *ctx = (ZCtx<mode> *)req_wrap->data_;
+
+ // If the avail_out is left at 0, then it means that it ran out
+ // of room. If there was avail_out left over, then it means
+ // that all of the input was consumed.
+ int err;
+ switch (mode) {
+ case DEFLATE:
+ case GZIP:
+ case DEFLATERAW:
+ err = deflate(&(ctx->strm_), ctx->flush_);
+ break;
+ case UNZIP:
+ case INFLATE:
+ case GUNZIP:
+ case INFLATERAW:
+ err = inflate(&(ctx->strm_), ctx->flush_);
+ break;
+ default:
+ assert(0 && "wtf?");
+ }
+ assert(err != Z_STREAM_ERROR);
+
+ // now After will emit the output, and
+ // either schedule another call to Process,
+ // or shift the queue and call Process.
+ }
+
+ // v8 land!
+ static void
+ After(uv_work_t* work_req) {
+ WorkReqWrap *req_wrap = reinterpret_cast<WorkReqWrap *>(work_req->data);
+ ZCtx<mode> *ctx = (ZCtx<mode> *)req_wrap->data_;
+ Local<Integer> avail_out = Integer::New(ctx->strm_.avail_out);
+ Local<Integer> avail_in = Integer::New(ctx->strm_.avail_in);
+
+ // call the write() cb
+ assert(req_wrap->object_->Get(callback_sym)->IsFunction() &&
+ "Invalid callback");
+ Local<Value> args[2] = { avail_in, avail_out };
+ MakeCallback(req_wrap->object_, "callback", 2, args);
+
+ // delete the ReqWrap
+ delete req_wrap;
+ }
+
+ static Handle<Value>
+ New(const Arguments& args) {
+ HandleScope scope;
+ ZCtx<mode> *ctx = new ZCtx<mode>();
+ ctx->Wrap(args.This());
+ return args.This();
+ }
+
+ // just pull the ints out of the args and call the other Init
+ static Handle<Value>
+ Init(const Arguments& args) {
+ HandleScope scope;
+
+ assert(args.Length() == 4 &&
+ "init(level, windowBits, memLevel, strategy)");
+
+ ZCtx<mode> *ctx = ObjectWrap::Unwrap< ZCtx<mode> >(args.This());
+
+ int windowBits = args[0]->Uint32Value();
+ assert((windowBits >= 8 && windowBits <= 15) && "invalid windowBits");
+
+ int level = args[1]->Uint32Value();
+ assert((level >= -1 && level <= 9) && "invalid compression level");
+
+ int memLevel = args[2]->Uint32Value();
+ assert((memLevel >= 1 && memLevel <= 9) && "invalid memlevel");
+
+ int strategy = args[3]->Uint32Value();
+ assert((strategy == Z_FILTERED ||
+ strategy == Z_HUFFMAN_ONLY ||
+ strategy == Z_RLE ||
+ strategy == Z_FIXED ||
+ strategy == Z_DEFAULT_STRATEGY) && "invalid strategy");
+
+ Init(ctx, level, windowBits, memLevel, strategy);
+ return Undefined();
+ }
+
+ static void
+ Init(ZCtx *ctx,
+ int level,
+ int windowBits,
+ int memLevel,
+ int strategy) {
+ ctx->level_ = level;
+ ctx->windowBits_ = windowBits;
+ ctx->memLevel_ = memLevel;
+ ctx->strategy_ = strategy;
+
+ ctx->strm_.zalloc = Z_NULL;
+ ctx->strm_.zfree = Z_NULL;
+ ctx->strm_.opaque = Z_NULL;
+
+ ctx->flush_ = Z_NO_FLUSH;
+
+ if (mode == GZIP || mode == GUNZIP) {
+ ctx->windowBits_ += 16;
+ }
+
+ if (mode == UNZIP) {
+ ctx->windowBits_ += 32;
+ }
+
+ if (mode == DEFLATERAW || mode == INFLATERAW) {
+ ctx->windowBits_ *= -1;
+ }
+
+ int err;
+ switch (mode) {
+ case DEFLATE:
+ case GZIP:
+ case DEFLATERAW:
+ err = deflateInit2(&(ctx->strm_),
+ ctx->level_,
+ Z_DEFLATED,
+ ctx->windowBits_,
+ ctx->memLevel_,
+ ctx->strategy_);
+ break;
+ case INFLATE:
+ case GUNZIP:
+ case INFLATERAW:
+ case UNZIP:
+ err = inflateInit2(&(ctx->strm_), ctx->windowBits_);
+ break;
+ default:
+ assert(0 && "wtf?");
+ }
+
+ ctx->init_done_ = true;
+ assert(err == Z_OK);
+ }
+
+ private:
+
+ bool init_done_;
+
+ z_stream strm_;
+ int level_;
+ int windowBits_;
+ int memLevel_;
+ int strategy_;
+
+ int flush_;
+
+ int chunk_size_;
+};
+
+
+#define NODE_ZLIB_CLASS(mode, name) \
+ { \
+ Local<FunctionTemplate> z = FunctionTemplate::New(ZCtx<mode>::New); \
+ z->InstanceTemplate()->SetInternalFieldCount(1); \
+ NODE_SET_PROTOTYPE_METHOD(z, "write", ZCtx<mode>::Write); \
+ NODE_SET_PROTOTYPE_METHOD(z, "init", ZCtx<mode>::Init); \
+ z->SetClassName(String::NewSymbol(name)); \
+ target->Set(String::NewSymbol(name), z->GetFunction()); \
+ }
+
+void InitZlib(Handle<Object> target) {
+ HandleScope scope;
+
+ NODE_ZLIB_CLASS(INFLATE, "Inflate")
+ NODE_ZLIB_CLASS(DEFLATE, "Deflate")
+ NODE_ZLIB_CLASS(INFLATERAW, "InflateRaw")
+ NODE_ZLIB_CLASS(DEFLATERAW, "DeflateRaw")
+ NODE_ZLIB_CLASS(GZIP, "Gzip")
+ NODE_ZLIB_CLASS(GUNZIP, "Gunzip")
+ NODE_ZLIB_CLASS(UNZIP, "Unzip")
+
+ ondata_sym = NODE_PSYMBOL("onData");
+ callback_sym = NODE_PSYMBOL("callback");
+
+ NODE_DEFINE_CONSTANT(target, Z_NO_FLUSH);
+ NODE_DEFINE_CONSTANT(target, Z_PARTIAL_FLUSH);
+ NODE_DEFINE_CONSTANT(target, Z_SYNC_FLUSH);
+ NODE_DEFINE_CONSTANT(target, Z_FULL_FLUSH);
+ NODE_DEFINE_CONSTANT(target, Z_FINISH);
+ NODE_DEFINE_CONSTANT(target, Z_BLOCK);
+ NODE_DEFINE_CONSTANT(target, Z_OK);
+ NODE_DEFINE_CONSTANT(target, Z_STREAM_END);
+ NODE_DEFINE_CONSTANT(target, Z_NEED_DICT);
+ NODE_DEFINE_CONSTANT(target, Z_ERRNO);
+ NODE_DEFINE_CONSTANT(target, Z_STREAM_ERROR);
+ NODE_DEFINE_CONSTANT(target, Z_DATA_ERROR);
+ NODE_DEFINE_CONSTANT(target, Z_MEM_ERROR);
+ NODE_DEFINE_CONSTANT(target, Z_BUF_ERROR);
+ NODE_DEFINE_CONSTANT(target, Z_VERSION_ERROR);
+ NODE_DEFINE_CONSTANT(target, Z_NO_COMPRESSION);
+ NODE_DEFINE_CONSTANT(target, Z_BEST_SPEED);
+ NODE_DEFINE_CONSTANT(target, Z_BEST_COMPRESSION);
+ NODE_DEFINE_CONSTANT(target, Z_DEFAULT_COMPRESSION);
+ NODE_DEFINE_CONSTANT(target, Z_FILTERED);
+ NODE_DEFINE_CONSTANT(target, Z_HUFFMAN_ONLY);
+ NODE_DEFINE_CONSTANT(target, Z_RLE);
+ NODE_DEFINE_CONSTANT(target, Z_FIXED);
+ NODE_DEFINE_CONSTANT(target, Z_DEFAULT_STRATEGY);
+ NODE_DEFINE_CONSTANT(target, ZLIB_VERNUM);
+
+ target->Set(String::NewSymbol("ZLIB_VERSION"), String::New(ZLIB_VERSION));
+}
+
+} // namespace node
+
+NODE_MODULE(node_zlib, node::InitZlib);
View
221 test/simple/test-zlib.js
@@ -0,0 +1,221 @@
+// Copyright Joyent, Inc. and other Node contributors.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a
+// copy of this software and associated documentation files (the
+// "Software"), to deal in the Software without restriction, including
+// without limitation the rights to use, copy, modify, merge, publish,
+// distribute, sublicense, and/or sell copies of the Software, and to permit
+// persons to whom the Software is furnished to do so, subject to the
+// following conditions:
+//
+// The above copyright notice and this permission notice shall be included
+// in all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN
+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+var common = require('../common.js');
+var assert = require('assert');
+var zlib = require('zlib');
+var path = require('path');
+
+var zlibPairs =
+ [ [zlib.Deflate, zlib.Inflate],
+ [zlib.Gzip, zlib.Gunzip],
+ [zlib.Deflate, zlib.Unzip],
+ [zlib.Gzip, zlib.Unzip],
+ [zlib.DeflateRaw, zlib.InflateRaw ] ];
+
+// how fast to trickle through the slowstream
+var trickle = [128, 1024, 1024 * 1024];
+
+// tunable options for zlib classes.
+
+// several different chunk sizes
+var chunkSize = [128, 1024, 1024 * 16, 1024 * 1024];
+
+// this is every possible value.
+var level = [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
+var windowBits = [8, 9, 10, 11, 12, 13, 14, 15];
+var memLevel = [1, 2, 3, 4, 5, 6, 7, 8, 9];
+var strategy = [0, 1, 2, 3, 4];
+
+// it's nice in theory to test every combination, but it
+// takes WAY too long. Maybe a pummel test could do this?
+if (!process.env.PUMMEL) {
+ trickle = [1024];
+ chunkSize = [1024 * 16];
+ level = [6];
+ memLevel = [8];
+ windowBits = [15];
+ strategy = [0];
+}
+
+var fs = require('fs');
+
+var testFiles = [ 'person.jpg', 'elipses.txt', 'empty.txt' ];
+var tests = {}
+testFiles.forEach(function(file) {
+ tests[file] = fs.readFileSync(path.resolve(common.fixturesDir, file));
+});
+
+var util = require('util');
+var stream = require('stream');
+
+
+// stream that saves everything
+function BufferStream() {
+ this.chunks = [];
+ this.length = 0;
+ this.writable = true;
+ this.readable = true;
+}
+
+util.inherits(BufferStream, stream.Stream);
+
+BufferStream.prototype.write = function(c) {
+ this.chunks.push(c);
+ this.length += c.length;
+};
+
+BufferStream.prototype.end = function(c) {
+ if (c) this.write(c);
+ // flatten
+ var buf = new Buffer(this.length);
+ var i = 0;
+ this.chunks.forEach(function(c) {
+ c.copy(buf, i);
+ i += c.length;
+ });
+ this.emit('data', buf);
+ this.emit('end');
+};
+
+
+function SlowStream(trickle) {
+ this.trickle = trickle;
+ this.offset = 0;
+ this.readable = this.writable = true;
+}
+
+util.inherits(SlowStream, stream.Stream);
+
+SlowStream.prototype.write = function() {
+ throw new Error('not implemented, just call ss.end(chunk)');
+};
+
+SlowStream.prototype.pause = function() {
+ this.paused = true;
+ this.emit('pause');
+};
+
+SlowStream.prototype.resume = function() {
+ var self = this;
+ if (self.ended) return;
+ self.emit('resume');
+ if (!self.chunk) return;
+ self.paused = false;
+ emit();
+ function emit() {
+ if (self.paused) return;
+ if (self.offset >= self.length) {
+ self.ended = true;
+ return self.emit('end');
+ }
+ var end = Math.min(self.offset + self.trickle, self.length);
+ var c = self.chunk.slice(self.offset, end);
+ self.offset += c.length;
+ self.emit('data', c);
+ process.nextTick(emit);
+ }
+}
+
+SlowStream.prototype.end = function(chunk) {
+ // walk over the chunk in blocks.
+ var self = this;
+ self.chunk = chunk;
+ self.length = chunk.length;
+ self.resume();
+ return self.ended;
+};
+
+
+
+// for each of the files, make sure that compressing and
+// decompressing results in the same data, for every combination
+// of the options set above.
+var failures = 0;
+var total = 0;
+var done = 0;
+
+Object.keys(tests).forEach(function(file) {
+ var test = tests[file];
+ chunkSize.forEach(function(chunkSize) {
+ trickle.forEach(function(trickle) {
+ windowBits.forEach(function(windowBits) {
+ level.forEach(function(level) {
+ memLevel.forEach(function(memLevel) {
+ strategy.forEach(function(strategy) {
+ zlibPairs.forEach(function(pair) {
+ var Def = pair[0];
+ var Inf = pair[1];
+ var opts = { level: level,
+ windowBits: windowBits,
+ memLevel: memLevel,
+ strategy: strategy };
+
+ total ++;
+
+ var def = new Def(opts);
+ var inf = new Inf(opts);
+ var ss = new SlowStream(trickle);
+ var buf = new BufferStream();
+
+ // verify that the same exact buffer comes out the other end.
+ buf.on('data', function(c) {
+ var msg = file + ' ' +
+ chunkSize + ' ' +
+ JSON.stringify(opts) + ' ' +
+ Def.name + ' -> ' + Inf.name;
+ var ok = true;
+ var testNum = ++done;
+ for (var i = 0; i < c.length; i++) {
+ if (c[i] !== test[i]) {
+ ok = false;
+ failures ++;
+ break;
+ }
+ }
+ if (ok) {
+ console.log('ok ' + (testNum) + ' ' + msg);
+ } else {
+ console.log('not ok ' + (testNum) + ' ' + msg);
+ console.log(' ...');
+ console.log(' testfile: ' + file);
+ console.log(' type: ' + Def.name + ' -> ' + Inf.name);
+ console.log(' position: ' + i);
+ console.log(' options: ' + JSON.stringify(opts));
+ console.log(' expect: ' + test[i]);
+ console.log(' actual: ' + c[i]);
+ console.log(' chunkSize: ' + chunkSize);
+ console.log(' ---');
+ }
+ });
+
+ // the magic happens here.
+ ss.pipe(def).pipe(inf).pipe(buf);
+ ss.end(test);
+ });
+ }); }); }); }); }); }); // sad stallman is sad.
+});
+
+process.on('exit', function(code) {
+ console.log('1..' + done);
+ assert.equal(done, total, (total - done) + ' tests left unfinished');
+ assert.ok(!failures, 'some test failures');
+});
View
38 wscript
@@ -62,11 +62,11 @@ def set_options(opt):
opt.tool_options('compiler_cc')
opt.tool_options('misc')
opt.add_option( '--libdir'
- , action='store'
- , type='string'
- , default=False
- , help='Install into this libdir [Release: ${PREFIX}/lib]'
- )
+ , action='store'
+ , type='string'
+ , default=False
+ , help='Install into this libdir [Release: ${PREFIX}/lib]'
+ )
opt.add_option( '--debug'
, action='store_true'
, default=False
@@ -158,7 +158,29 @@ def set_options(opt):
)
- opt.add_option('--shared-cares'
+ opt.add_option( '--shared-zlib'
+ , action='store_true'
+ , default=False
+ , help='Link to a shared zlib DLL instead of static linking'
+ , dest='shared_zlib'
+ )
+
+ opt.add_option( '--shared-zlib-includes'
+ , action='store'
+ , default=False
+ , help='Directory containing zlib header files'
+ , dest='shared_zlib_includes'
+ )
+
+ opt.add_option( '--shared-zlib-libpath'
+ , action='store'
+ , default=False
+ , help='A directory to search for the shared zlib DLL'
+ , dest='shared_zlib_libpath'
+ )
+
+
+ opt.add_option( '--shared-cares'
, action='store_true'
, default=False
, help='Link to a shared C-Ares DLL instead of static linking'
@@ -246,6 +268,7 @@ def configure(conf):
conf.env["USE_SHARED_V8"] = o.shared_v8 or o.shared_v8_includes or o.shared_v8_libpath or o.shared_v8_libname
conf.env["USE_SHARED_CARES"] = o.shared_cares or o.shared_cares_includes or o.shared_cares_libpath
+ conf.env["USE_SHARED_ZLIB"] = o.shared_zlib or o.shared_zlib_includes or o.shared_zlib_libpath
conf.env["USE_GDBJIT"] = o.use_gdbjit
@@ -838,7 +861,7 @@ def build(bld):
node = bld.new_task_gen("cxx", product_type)
node.name = "node"
node.target = "node"
- node.uselib = 'RT OPENSSL CARES EXECINFO DL KVM SOCKET NSL KSTAT UTIL OPROFILE'
+ node.uselib = 'RT OPENSSL ZLIB CARES EXECINFO DL KVM SOCKET NSL KSTAT UTIL OPROFILE'
node.add_objects = 'http_parser'
if product_type_is_lib:
node.install_path = '${LIBDIR}'
@@ -857,6 +880,7 @@ def build(bld):
src/node_os.cc
src/node_dtrace.cc
src/node_string.cc
+ src/node_zlib.cc
src/timer_wrap.cc
src/handle_wrap.cc
src/stream_wrap.cc

0 comments on commit ffce52b

Please sign in to comment.
Something went wrong with that request. Please try again.