Skip to content

Commit

Permalink
add start and end options to openReadStream. closes #38
Browse files Browse the repository at this point in the history 
change decrypt and decompress parameter validation slightly
  • Loading branch information
@thejoshwolfe
thejoshwolfe committed Apr 22, 2017
1 parent 08c81b2 commit 04e77fe
Show file tree
Hide file tree
Showing 4 changed files with 199 additions and 10 deletions.
22 changes: 17 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ Calling this method after calling `close()` will cause undefined behavior.
#### openReadStream(entry, [options], callback)

`entry` must be an `Entry` object from this `ZipFile`.
`callback` gets `(err, readStream)`, where `readStream` is a `Readable Stream`.
`callback` gets `(err, readStream)`, where `readStream` is a `Readable Stream` that provides the file data for this entry.
If this zipfile is already closed (see `close()`), the `callback` will receive an `err`.

`options` may be omitted or `null`, and has the following defaults:
Expand All @@ -223,6 +223,8 @@ If this zipfile is already closed (see `close()`), the `callback` will receive a
{
decompress: entry.isCompressed() ? true : null,
decrypt: null,
start: 0, // actually the default is null, see below
end: entry.compressedSize, // actually the default is null, see below
}
```

Expand All @@ -234,20 +236,29 @@ Omitting the `decompress` option is what most clients should do.
The `decompress` option must be `null` (or omitted) when the entry is not compressed (see `isCompressed()`),
and either `true` (or omitted) or `false` when the entry is compressed.
Specifying `decompress: false` for a compressed entry causes the read stream
to provide the raw compressed file data without going through zlib.
to provide the raw compressed file data without going through a zlib inflate transform.

If the entry is encrypted (see `isEncrypted()`), clients may want to avoid calling `openReadStream()` on the entry entirely.
Alternatively, clients may call `openReadStream()` for encrypted entries and specify `decrypt: false`.
If the entry is also compressed, clients must *also* specify `decompress: false`, or else the `callback` will receive an `err`.
Specifying `decrypt: false` for an encrypted entry causes the read stream to provide the raw encrypted file data.
If the entry is also compressed, clients must *also* specify `decompress: false`.
Specifying `decrypt: false` for an encrypted entry causes the read stream to provide the raw, still-encrypted file data.
(This data includes the 12-byte header described in the spec.)

The `decrypt` option must be `null` (or omitted) for non-encrypted entries, and `false` for encrypted entries.
Omitting the `decrypt` option (or specifying it as `null`) for an encrypted entry
will result in the `callback` receiving an `err`.
This default behavior is so that clients not accounting for encrypted files aren't surprised by bogus file data.

It's also possible for the `readStream` to emit errors for several reasons.
The `start` (inclusive) and `end` (exclusive) options are byte offsets into this entry's file data,
and can be used to obtain part of an entry's file data rather than the whole thing.
If either of these options are specified and non-`null`,
then the above options must be used to obain the file's raw data.
Speficying `{start: 0, end: entry.compressedSize}` will result in the complete file,
which is effectively the default values for these options,
but note that unlike omitting the options, when you specify `start` or `end` as any non-`null` value,
the above requirement is still enforced that you must also pass the appropriate options to get the file's raw data.

It's possible for the `readStream` provided to the `callback` to emit errors for several reasons.
For example, if zlib cannot decompress the data, the zlib error will be emitted from the `readStream`.
Two more error cases (when `validateEntrySizes` is `true`) are if the decompressed data has too many
or too few actual bytes compared to the reported byte count from the entry's `uncompressedSize` field.
Expand Down Expand Up @@ -556,6 +567,7 @@ This library makes no attempt to interpret the Language Encoding Flag.
* Added option `validateEntrySizes`. [issue #53](https://github.com/thejoshwolfe/yauzl/issues/53)
* Added `examples/promises.js`
* Added ability to read raw file data via `decompress` and `decrypt` options. [issue #11](https://github.com/thejoshwolfe/yauzl/issues/11), [issue #38](https://github.com/thejoshwolfe/yauzl/issues/38), [pull #39](https://github.com/thejoshwolfe/yauzl/pull/39)
* Added `start` and `end` options to `openReadStream()`. [issue #38](https://github.com/thejoshwolfe/yauzl/issues/38)
* 2.7.0
* Added option `decodeStrings`. [issue #42](https://github.com/thejoshwolfe/yauzl/issues/42)
* Fixed documentation for `entry.fileComment` and added compatibility alias. [issue #47](https://github.com/thejoshwolfe/yauzl/issues/47)
Expand Down
37 changes: 32 additions & 5 deletions index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -422,15 +422,21 @@ ZipFile.prototype.readEntry = function() {
ZipFile.prototype.openReadStream = function(entry, options, callback) {
var self = this;
// parameter validation
var relativeStart = 0;
var relativeEnd = entry.compressedSize;
if (callback == null) {
callback = options;
options = {};
} else {
// validate options that the caller has no excuse to get wrong
if (options.decrypt != null) {
if (!entry.isEncrypted()) {
throw new Error("options.decrypt can only be specified for encrypted entries");
}
if (options.decrypt !== false) throw new Error("invalid options.decrypt value: " + options.decrypt);
if (entry.isCompressed()) {
if (options.decompress !== false) throw new Error("entry is encrypted and compressed, and options.decompress !== false");
}
}
if (options.decompress != null) {
if (!entry.isCompressed()) {
Expand All @@ -440,14 +446,32 @@ ZipFile.prototype.openReadStream = function(entry, options, callback) {
throw new Error("invalid options.decompress value: " + options.decompress);
}
}
if (options.start != null || options.end != null) {
if (entry.isCompressed() && options.decompress !== false) {
throw new Error("start/end range not allowed for compressed entry without options.decompress === false");
}
if (entry.isEncrypted() && options.decrypt !== false) {
throw new Error("start/end range not allowed for encrypted entry without options.decrypt === false");
}
}
if (options.start != null) {
relativeStart = options.start;
if (relativeStart < 0) throw new Error("options.start < 0");
if (relativeStart > entry.compressedSize) throw new Error("options.start > entry.compressedSize");
}
if (options.end != null) {
relativeEnd = options.end;
if (relativeEnd < 0) throw new Error("options.end < 0");
if (relativeEnd > entry.compressedSize) throw new Error("options.end > entry.compressedSize");
if (relativeEnd < relativeStart) throw new Error("options.end < options.start");
}
}
// any further errors can be caused by the zipfile, so should be passed to the client rather than thrown
// any further errors can either be caused by the zipfile,
// or were introduced in a minor version of yauzl,
// so should be passed to the client rather than thrown.
if (!self.isOpen) return callback(new Error("closed"));
if (entry.isEncrypted()) {
if (options.decrypt !== false) return callback(new Error("entry is encrypted, and options.decrypt !== false"));
if (entry.isCompressed()) {
if (options.decompress !== false) return callback(new Error("entry is encrypted and compressed, and options.decompress !== false"));
}
}
// make sure we don't lose the fd before we open the actual read stream
self.reader.ref();
Expand Down Expand Up @@ -497,7 +521,10 @@ ZipFile.prototype.openReadStream = function(entry, options, callback) {
fileDataStart + " + " + entry.compressedSize + " > " + self.fileSize));
}
}
var readStream = self.reader.createReadStream({start: fileDataStart, end: fileDataEnd});
var readStream = self.reader.createReadStream({
start: fileDataStart + relativeStart,
end: fileDataStart + relativeEnd,
});
var endpointStream = readStream;
if (decompress) {
var destroyed = false;
Expand Down
146 changes: 146 additions & 0 deletions test/range-test.js
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
var yauzl = require("../");
var PassThrough = require("stream").PassThrough;
var util = require("util");
var Pend = require("pend");
var BufferList = require("bl");

exports.runTest = runTest;

// zipfile obtained via:
// $ echo -n 'aaabaaabaaabaaab' > stored.txt
// $ cp stored.txt compressed.txt
// $ cp stored.txt encrypted.txt
// $ cp stored.txt encrypted-and-compressed.txt
// $ rm -f out.zip
// $ zip out.zip -0 stored.txt
// $ zip out.zip compressed.txt
// $ zip out.zip -e0 encrypted.txt
// $ zip out.zip -e encrypted-and-compressed.txt
var zipfileBuffer = hexToBuffer("" +
"504b03040a00000000006a54954ab413389510000000100000000a001c007374" +
"6f7265642e7478745554090003d842fa5842c5f75875780b000104e803000004" +
"e803000061616162616161626161616261616162504b03041400000008007554" +
"954ab413389508000000100000000e001c00636f6d707265737365642e747874" +
"5554090003ed42fa58ed42fa5875780b000104e803000004e80300004b4c4c4c" +
"4a44c200504b03040a00090000008454954ab41338951c000000100000000d00" +
"1c00656e637279707465642e74787455540900030743fa580743fa5875780b00" +
"0104e803000004e8030000f72e7bb915142131c934f01b163fcadb2a8db7cdaf" +
"d0a6f4dd1694c0504b0708b41338951c00000010000000504b03041400090008" +
"008a54954ab413389514000000100000001c001c00656e637279707465642d61" +
"6e642d636f6d707265737365642e74787455540900031343fa581343fa587578" +
"0b000104e803000004e80300007c4d3ea0d9754b470d3eb32ada5741bfc848f4" +
"19504b0708b41338951400000010000000504b01021e030a00000000006a5495" +
"4ab413389510000000100000000a0018000000000000000000b4810000000073" +
"746f7265642e7478745554050003d842fa5875780b000104e803000004e80300" +
"00504b01021e031400000008007554954ab413389508000000100000000e0018" +
"000000000001000000b48154000000636f6d707265737365642e747874555405" +
"0003ed42fa5875780b000104e803000004e8030000504b01021e030a00090000" +
"008454954ab41338951c000000100000000d0018000000000000000000b481a4" +
"000000656e637279707465642e74787455540500030743fa5875780b000104e8" +
"03000004e8030000504b01021e031400090008008a54954ab413389514000000" +
"100000001c0018000000000001000000b48117010000656e637279707465642d" +
"616e642d636f6d707265737365642e74787455540500031343fa5875780b0001" +
"04e803000004e8030000504b0506000000000400040059010000910100000000" +
"");
// the same file in all 4 supported forms:
// [0b00]: stored
// [0b01]: compressed
// [0b10]: encrypted
// [0b11]: encrypted and compressed
function shouldBeCompressed(index) { return (index & 1) !== 0; }
function shouldBeEncrypted (index) { return (index & 2) !== 0; }
var expectedFileDatas = [
hexToBuffer("61616162616161626161616261616162"),
hexToBuffer("4b4c4c4c4a44c200"),
hexToBuffer("f72e7bb915142131c934f01b163fcadb2a8db7cdafd0a6f4dd1694c0"),
hexToBuffer("7c4d3ea0d9754b470d3eb32ada5741bfc848f419"),
];

function runTest(cb) {
function StingyRandomAccessReader(buffer) {
yauzl.RandomAccessReader.call(this);
this.buffer = buffer;
this.upcomingByteCounts = [];
}
StingyRandomAccessReader.prototype._readStreamForRange = function(start, end) {
if (this.upcomingByteCounts.length > 0) {
var expectedByteCount = this.upcomingByteCounts.shift();
if (expectedByteCount != null) {
if (expectedByteCount !== end - start) {
throw new Error("expected " + expectedByteCount + " got " + (end - start) + " bytes");
}
}
}
var result = new PassThrough();
result.write(this.buffer.slice(start, end));
result.end();
return result;
};
util.inherits(StingyRandomAccessReader, yauzl.RandomAccessReader);

var zipfileReader = new StingyRandomAccessReader(zipfileBuffer);
var options = {lazyEntries: true, autoClose: false};
yauzl.fromRandomAccessReader(zipfileReader, zipfileBuffer.length, options, function(err, zipfile) {
var entries = [];
zipfile.readEntry();
zipfile.on("entry", function(entry) {
var index = entries.length;
// asser the structure of the zipfile is what we expect
if (entry.isCompressed() !== shouldBeCompressed(index)) throw new Error("assertion failure");
if (entry.isEncrypted() !== shouldBeEncrypted(index)) throw new Error("assertion failure");
entries.push(entry);
zipfile.readEntry();
});
zipfile.on("end", function() {
// now we get to the testing

var pend = new Pend();
// 1 thing at a time for better determinism/reproducibility
pend.max = 1;

[null, 0, 2].forEach(function(start) {
[null, 3, 5].forEach(function(end) {
entries.forEach(function(entry, index) {
var expectedFileData = expectedFileDatas[index];
pend.go(function(cb) {
var effectiveStart = start != null ? start : 0;
var effectiveEnd = end != null ? end : expectedFileData.length;
var expectedSlice = expectedFileData.slice(effectiveStart, effectiveEnd);
// the next read will be to check the local file header.
// then we assert that yauzl is asking for just the bytes we asked for.
zipfileReader.upcomingByteCounts = [null, expectedSlice.length];

var options = {};
if (start != null) options.start = start;
if (end != null) options.end = end;
if (entry.isCompressed()) options.decompress = false;
if (entry.isEncrypted()) options.decrypt = false;
zipfile.openReadStream(entry, options, function(err, readStream) {
if (err) throw err;
readStream.pipe(BufferList(function(err, data) {
var prefix = "openReadStream with range(" + start + "," + end + "," + index + "): ";
if (!data.equals(expectedSlice)) {
throw new Error(prefix + "contents mismatch");
}
console.log(prefix + "pass");
cb();
}));
});
});
});
});
});
pend.wait(cb);
});
});
}

function hexToBuffer(hexString) {
var buffer = new Buffer(hexString.length / 2);
for (var i = 0; i < buffer.length; i++) {
buffer[i] = parseInt(hexString.substr(i * 2, 2), 16);
}
return buffer;
}

if (require.main === module) runTest(function() {});
4 changes: 4 additions & 0 deletions test/test.js
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
var yauzl = require("../");
var zip64 = require("./zip64");
var rangeTest = require("./range-test");
var fs = require("fs");
var path = require("path");
var Pend = require("pend");
Expand Down Expand Up @@ -326,6 +327,9 @@ pend.go(function(cb) {
// zip64
pend.go(zip64.runTest);

// openReadStream with range
pend.go(rangeTest.runTest);

pend.wait(function() {
// if you don't see this, something never happened.
console.log("done");
Expand Down

0 comments on commit 04e77fe

Please sign in to comment.