Memory leak in unzip() #2595

Closed
vflash opened this Issue Jan 23, 2012 · 11 comments

Projects

None yet

8 participants

@vflash
vflash commented Jan 23, 2012

когда съедает всю память процесс умирает. тест делал на виртуальной машине с 1гб памятью.
eats all memory when the process dies. test done on a virtual machine with 1GB of memory.

latest (0.7.0-pre) node
node --expose-gc

var zlib = require('zlib');
var buffer = new Buffer('eJzT0yMAAGTvBe8=', 'base64');
function nullfun(){};

function test() {
    console.log(process.memoryUsage());
    for (var i = 0; i < 100; i++) {
        zlib.unzip(buffer, nullfun);
    };
    process.nextTick(test);
};
process.nextTick(test);
....
{ rss: 870871040, heapTotal: 42841856, heapUsed: 8936624 }
{ rss: 871223296, heapTotal: 42841856, heapUsed: 9354648 }
{ rss: 871858176, heapTotal: 42841856, heapUsed: 9630272 }
{ rss: 872452096, heapTotal: 42841856, heapUsed: 10048328 }
{ rss: 872853504, heapTotal: 42841856, heapUsed: 10331552 }
Убито
vflash@vu:~/work$

@isaacs
isaacs commented Jan 23, 2012

Fixed in 0.6.8. Not yet ported back to master.

@vflash
vflash commented Jan 23, 2012

в 0.6.8 повторяется
in 0.6.8 is repeated

@bnoordhuis
Member

@isaacs: Can this issue be closed?

@plievone

I just tried this on 0.6.15 (ubuntu) and there seems to be a memory leak, even if I add gc() in there. Does this comment mean that this is fixed only in 0.7.x: #2504 (comment)

Also see #2933.

@vvo
vvo commented Jun 6, 2012

I too have seen this issue on zlib.unzip (that was causing our production servers to crash)

Using only zlib.gzunip convenience method on the same data does not trigger any memory leak (!?).

Memory drop on our servers

https://twitter.com/zeroload/status/210678245576818688
memory drop

This graph shows the memory consumption when switching from zlib.unzip to zlib.gunzip on gzipped content.
Our product features are still ok. I know it doesn't prove anything for node and I'll try to make a good test file for this.

Old zlib version? (=> no)

UPDATE: I tested with latest zlib source code, recompiling nodejs with it but that did not change the behavior.

Leaving rest of the story for archive.
While I'm clearly not an expert, I did some searching about the zlib binding in node and found this:

Reading https://github.com/joyent/node/blob/v0.6.19-release/deps/zlib/README.chromium

Name: zlib
URL: http://zlib.net/
Version: 1.2.3

Then in the licence file we see zlib version 1.2.4, March 14th, 2010 and then in zlib.h we see version 1.2.3, July 18th, 2005.

Hard to tell which version is actually used.

So node use perhaps zlib 1.2.3. which seems old.

But old does not mean crappy, if it's used on the whole chromium project it must be customized and perhaps better than original?

Reading http://zlib.net/ChangeLog.txt and searching for leak :

Changes in 1.2.4.2 (9 Apr 2010)
- Fix memory leak on error in gz_open()
Changes in 1.2.4 (14 Mar 2010)
- Fix memory leaks in gzclose_r() and gzclose_w(), file leak in gz_open()

Also seems like theses leaks were related to using zlib with files not in memory data so I just lost myself here :)

Hope it helps.

@rrusso2007

Just made the same change from unzip to gunzip and my prod memory munin graph now looks exactly like @vvo . this issue report was exactly what I was looking for. What exactly is the implementation difference between unzip and gunzip anyway?

@vvo
vvo commented Jul 8, 2012

AH! Great your adding some support to this i-m-going-completely-nuts issue. I do not have any more clue sorry.

@isaacs isaacs added a commit to isaacs/node-v0.x-archive that referenced this issue Jul 9, 2012
@isaacs isaacs zlib: Call inflateEnd for Unzip objects
Re: #2595
e3aafac
@isaacs isaacs closed this in d2cd456 Jul 9, 2012
@isaacs
isaacs commented Jul 9, 2012

Fix will be included in 0.8.3.

@vflash
vflash commented Jul 9, 2012

Thank you very much

@richardlau richardlau pushed a commit to ibmruntimes/node that referenced this issue Nov 5, 2015
@rvagg rvagg 2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: nodejs/node#3466
837b174
@richardlau richardlau pushed a commit to ibmruntimes/node that referenced this issue Nov 5, 2015
@rvagg rvagg 2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: nodejs/node#3466
384e6c2
@OpakAlex

Same issue on 6.2.1

@addaleax
Member

@OpakAlex This is an old issue that has been fixed for a long, long time. If you have trouble with current Node versions, please create an issue at https://github.com/nodejs/node and include as much detail as possible (esp. code to reproduce).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment