/node

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zlib: decompression throw on truncated input #2595

Closed
wants to merge 1 commit into
base: master
from

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
semver-major zlib
Projects
None yet
Milestone
  5.0.0
6 participants
@jhamhader @Fishrock123 @indutny @trevnorris @thefourtheye @targos
@jhamhader
Contributor

jhamhader commented Aug 28, 2015

Check for unexpected end-of-stream error when decompressing.
If the output buffer still has space after decompressing and deflate
returned Z_OK or Z_BUF_ERROR - that means unexpected end-of-stream.
Added test-zlib-truncated.js for the case of truncated input.
Fixed the zlib dictionary test to not end the inflate stream on a
truncated output (no crc) of deflate
See issue #2043

@targos targos added the zlib label Aug 28, 2015

@Fishrock123

This comment has been minimized.

Sign in to view
Member

Fishrock123 commented Aug 28, 2015

CI run: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/200/
@Fishrock123

Fishrock123 reviewed Aug 28, 2015
View changes

test/parallel/test-zlib-truncated.js Outdated
var common = require('../common');
var assert = require('assert');
var zlib = require ('zlib');

This comment has been minimized.

Sign in to view
Copy link
@Fishrock123

Fishrock123 Aug 28, 2015

Member

Please use const for values that won't be changed that you either added or modified in this commit. :)

This comment has been minimized.

Sign in to view
Copy link
@jhamhader

jhamhader Aug 28, 2015

Contributor

Thanks for the input :)
Should I use 'let' for the scope-specific variables or is 'var' ok?
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Aug 28, 2015

Committed 'let' and 'const' for 'test-zlib-dictionary.js' and 'test-zlib-truncated.js'

@rvagg rvagg force-pushed the nodejs:master branch from 11c25c2 to ba02bd0 Sep 6, 2015

@jhamhader jhamhader force-pushed the jhamhader:zlib-truncated branch 2 times, most recently Sep 19, 2015

@indutny

This comment has been minimized.

Sign in to view
Member

indutny commented Sep 24, 2015

cc @nodejs/collaborators
@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Sep 24, 2015

Is this done elsewhere (as in, in other programs), and/or is it what the user would expect to happen?
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Sep 27, 2015

  • gnu gunzip: fails with unexpected end of file
  • python2 gzip: fails with IOError: CRC check failed 0xca600aac != 0x6fe58c67L
  • python3 gzip: fails with EOFError: Compressed file ended before the end-of-stream marker was reached
  • ruby gzip: fails with Zlib::GzipFile::Error: unexpected end of file
  • perl gzip: fails with unexpected end of file at (eval 15)[/usr/share/perl5/core_perl/perl5db.pl:737] line 2.

@jhamhader jhamhader force-pushed the jhamhader:zlib-truncated branch Sep 27, 2015

@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Sep 27, 2015

Seems at the minimum we should be propagation an error. So will it throw for sync and emit error on async? (not familiar w/ much of the zlib api)
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Sep 27, 2015

Throw for sync.
What do you mean by emit? The callback of the async function will be called with err on the first argument.
@jhamhader
zlib: decompression throw on truncated input 
Check for unexpected end-of-file error when decompressing.
If the output buffer still has space after decompressing and deflate
returned Z_OK or Z_BUF_ERROR - that means unexpected end-of-file.
Added test-zlib-truncated.js for the case of truncated input.
Fixed the zlib dictionary test to not end the inflate stream on a
truncated output (no crc) of deflate
aae310e

@jhamhader jhamhader force-pushed the jhamhader:zlib-truncated branch to aae310e Sep 28, 2015

@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Sep 28, 2015

The callback of the async function will be called with err on the first argument.

Nm me. Was thinking about using zlib w/ the streams API and how a pipe would propagate the error.
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Sep 28, 2015

Sure :)
Anything else missing for the PR?
@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Sep 28, 2015

Not from me. Would like @indutny to sign off on this as well.
@thefourtheye

This comment has been minimized.

Sign in to view
Contributor

thefourtheye commented Sep 29, 2015

I would prefer @chrisdickinson to sign off as well.
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Oct 8, 2015

@indutny @chrisdickinson could you please take a look and confirm?
Thanks :)
@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Oct 8, 2015

@jhamhader ping me tomorrow if @indutny hasn't responded.
@jhamhader

This comment has been minimized.

Sign in to view
Contributor

jhamhader commented Oct 10, 2015

Pinging
@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Oct 10, 2015

Thanks. Let's go ahead and land this. I'll be back at my computer on Monday.

Also, I'm initially inclined to say this is semver-major. Even if it's not, don't know if this is something LTS would accept. We can check after it's in master. :-)

@Fishrock123 Fishrock123 added the semver-major label Oct 10, 2015

@Fishrock123 Fishrock123 added this to the 5.0.0 milestone Oct 10, 2015

@rvagg rvagg referenced this pull request Oct 16, 2015

Closed

v5.0.0 Planning & Checklist #3397

13 of 13 tasks complete
@Fishrock123

This comment has been minimized.

Sign in to view
Member

Fishrock123 commented Oct 19, 2015

ping @trevnorris
@trevnorris

This comment has been minimized.

Sign in to view
Contributor

trevnorris commented Oct 19, 2015

Landed on 0fc0902. Thanks much!

@trevnorris trevnorris closed this Oct 19, 2015

trevnorris added a commit that referenced this pull request Oct 19, 2015

@jhamhader @trevnorris
zlib: decompression throw on truncated input 
Check for unexpected end-of-file error when decompressing. If the output
buffer still has space after decompressing and deflate returned Z_OK or
Z_BUF_ERROR - that means unexpected end-of-file. Added
test-zlib-truncated.js for the case of truncated input. Fixed the zlib
dictionary test to not end the inflate stream on a truncated output (no
crc) of deflate

Fixes: #2043
PR-URL: #2595
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
0fc0902

rvagg added a commit that referenced this pull request Oct 21, 2015

@jhamhader @rvagg
zlib: decompression throw on truncated input 
Check for unexpected end-of-file error when decompressing. If the output
buffer still has space after decompressing and deflate returned Z_OK or
Z_BUF_ERROR - that means unexpected end-of-file. Added
test-zlib-truncated.js for the case of truncated input. Fixed the zlib
dictionary test to not end the inflate stream on a truncated output (no
crc) of deflate

Fixes: #2043
PR-URL: #2595
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
80169b1

@rvagg rvagg referenced this pull request Oct 21, 2015

Merged

Release proposal: v5.0.0 #3466

rvagg added a commit to rvagg/io.js that referenced this pull request Oct 29, 2015

@rvagg
2015-10-29, Version 5.0.0 (Stable) 
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) nodejs#3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) nodejs#3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) nodejs#3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) nodejs#3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) nodejs#3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) nodejs#3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) nodejs#2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) nodejs#2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) nodejs#3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) nodejs#3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) nodejs#3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) nodejs#2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) nodejs#2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) nodejs#1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) nodejs#3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) nodejs#3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) nodejs#2595.

PR-URL: nodejs#3466
384e6c2

rvagg added a commit that referenced this pull request Oct 29, 2015

@rvagg
2015-10-29, Version 5.0.0 (Stable) 
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: #3466
837b174

This was referenced Nov 25, 2015

Closed
Closed

@jhamhader jhamhader referenced this pull request Mar 17, 2016

Closed

Gzip decoder too strict #5761

@tomhughes tomhughes referenced this pull request Mar 31, 2016

Merged

Fix "incomplete" test fixture to be valid gzip of incomplete data #167

@conradoplg conradoplg referenced this pull request Oct 25, 2016

Open

Zlib._handle.onerror unexpected end of file #300

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment