Maximum call stack size exceeded #16

brianhiss · 2013-03-07T19:06:20Z

When I try to unzip a file on Heroku, I am getting a Maximum call stack size exceeded if the zip file contains a file of over ~110kb. If all files are less than that, it doesn't seem to matter how big the zip file itself is. Just seems to choke on files inside the zip if larger than ~110kb.

I am using pipe:
fs.createReadStream('test.zip').pipe(unzip.Extract({ path: 'test/' }))

Here is the Heroku crash log:

domain_thrown: true,
POST /config { [RangeError: Maximum call stack size exceeded]
domain:
{ domain: null,
_events: { error: [Function] },
_maxListeners: 10,
members: [ [Object], [Object] ] } }

timsweb · 2013-03-07T19:27:00Z

I'm also getting a similar (or the same) error:

node_modules/over/overload.js:18
    var args = Array.prototype.slice.call(arguments);
                                     ^
RangeError: Maximum call stack size exceeded

My zip files are about 5Mb, most work fine (and some 300Mb haven't had any issues), but some don't. I might not be able to link to a failing zip file though as it's contains sensitive information :-(

I did an npm update before posting here so should be on the latest version.

brianhiss · 2013-03-07T19:33:23Z

Yeah, I saw your issue. It was closed and a patch issued, but I have cloned the latest code and still no luck.

Not quite sure why it chokes, and it seems to be platform specific issue. On Heroku (less resources) it fails. On my local Mac, it works fine for the big files.

timsweb · 2013-03-07T20:07:39Z

I think I know the other issue your talking about, although it's not mine. I too cloned didn't have any joy.

ralphholzmann · 2013-03-10T03:07:35Z

I'm also experiencing this issue sporadically :/

timsweb · 2013-03-11T11:45:23Z

I've had a chance to look into this a little closer and have submitted a pull request (that's opened issue #17). Feel free give it a test.

brianhiss · 2013-03-11T15:18:57Z

So far it's working on the use cases it was failing on for me. I will get back if I see anything different. Thank you!

EvanOxfeld · 2013-03-12T00:04:46Z

@timsweb @ralphholzmann @brianhiss Thanks for your help troubleshooting this issue. I wanted to get the fix in #17 up on NPM as soon as possible, so I published v0.1.5.

That said, there's a lot of logic not really about parsing zips in parse.js / untilstream.js that would be more reliable as a module based on the Node 0.10 stream.Transform with its own tests. I'll leave this issue open until the logic's impending move barring problems with the match-stream module. Expect an update to this issue soon. I know some zips that have failed to extract in the past contain sensitive information and would appreciate your help again to test.

satazor · 2013-03-17T10:37:43Z

@EvanOxfeld An user complained about this very same issue here: bower/bower#312
I think that's a good test case.

EvanOxfeld · 2013-03-17T20:21:26Z

Really appreciate the test case, @satazor. I'll get it integrated into the unzip tests.

nprbst · 2013-03-27T01:22:25Z

I'm not sure if this is a useful clue or not, but for the same file I can repeatably cause #16 to break or not break by...wait for it...changing from 1 CPU to 2 CPUs!

Configuration

Node v0.8.22
unzip v0.1.4 (v0.1.5 "hangs" on me)

1 CPU: Breaks every time

AWS EC2 Micro Instance, Ubuntu 12.04.1 LTS (~640 MB, 1 CPU)
VirtualBox, Ubuntu 12.04.1 LTS (640 MB, 1 CPU)

2 CPUs: Works every time

Mac mini Server, OS X Lion (16 GB, 4 cores)
VirtualBox, Ubuntu 12.04.1 LTS (640 MB, 2 CPUs)

satazor · 2013-04-03T22:21:01Z

@EvanOxfeld Some more details on unzip 0.1.6

node 0.10.3

bootstrap.zip

var unzip = require('unzip');
var fs = require('fs');

fs.createReadStream('bootstrap.zip')
    .pipe(unzip.Extract({ path: 'test' }))
    .on('error', function (err) { console.log('error', err); })
    .on('close', function () { console.log('closed'); });

Works!

jquery-ui-1.10.0.custom.zip

var unzip = require('unzip');
var fs = require('fs');

fs.createReadStream('jquery-ui-1.10.0.custom.zip')
    .pipe(unzip.Extract({ path: 'test' }))
    .on('error', function (err) { console.log('error', err); })
    .on('close', function () { console.log('closed'); });

Fails.. it previously worked. Here's the stack trace:

events.js:72
        throw er; // Unhandled 'error' event
              ^
TypeError: Invalid non-string/buffer chunk
    at chunkInvalid (_stream_readable.js:339:10)
    at readableAddChunk (_stream_readable.js:126:12)
    at Match.Readable.push (_stream_readable.js:113:10)
    at Match.Transform.push (_stream_transform.js:142:32)
    at Match._matchFn (/Users/satazor/Work/twitter/bower/node_modules/unzip/lib/parse.js:161:31)
    at Match._transform (/Users/satazor/Work/twitter/bower/node_modules/unzip/node_modules/match-stream/match.js:39:10)
    at Match.Transform._read (_stream_transform.js:181:10)
    at Match.Transform._write (_stream_transform.js:169:12)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)

node 0.8.22

bootstrap.zip

It just hangs while extracting, high CPU usage

jquery-ui-1.10.0.custom.zip

Fails with the same error as the node 0.10.3

EvanOxfeld · 2013-04-04T06:06:27Z

Thanks for the info @satazor - I'm close to a fix, at least for node 0.10.3.

Recursively calling Parse._flush caused recursive process.nextTick calls. Fixes #16.

satazor · 2013-04-07T14:14:56Z

@EvanOxfeld after upgrading to 0.1.7, all issues present in my previous comment still exist.
Shall I create a separate issue?

EvanOxfeld · 2013-04-07T16:43:24Z

@satazor If you have a chance, check out #23 - fixes for node 0.10.x, close for node 0.8.x. I'll have more time to devote to fixing this issue later today.

sindresorhus · 2013-04-22T21:51:15Z

Is there any updates on this? It's currently a pressing issue for Bower.

euskadi31 · 2013-07-04T21:48:51Z

I have the same problem

Node: 0.10.12
Bower: 0.10.0
unzip: 0.1.8

alFReD-NSH · 2013-07-22T15:53:32Z

Just another sample:

bower downloading http://www.playcraftlabs.com/downloads/playcraft-v0.5.6a.zip
bower extracting /tmp/bower-playcraft-v0.5.6a-2119dtushkw/index.zip

/home/alfred/nvm/v0.10.12/lib/node_modules/bower/node_modules/tmp/lib/tmp.js:260
  throw err;
        ^
TypeError: Invalid non-string/buffer chunk
    at chunkInvalid (_stream_readable.js:365:10)
    at readableAddChunk (_stream_readable.js:137:12)
    at Match.Readable.push (_stream_readable.js:127:10)
    at Match.Transform.push (_stream_transform.js:140:32)
    at Match._matchFn (/home/alfred/nvm/v0.10.12/lib/node_modules/bower/node_modules/unzip/lib/parse.js:161:31)
    at Match._transform (/home/alfred/nvm/v0.10.12/lib/node_modules/bower/node_modules/unzip/node_modules/match-stream/match.js:39:10)
    at Match.Transform._read (_stream_transform.js:179:10)
    at Match.Transform._write (_stream_transform.js:167:12)
    at doWrite (_stream_writable.js:219:10)
    at writeOrBuffer (_stream_writable.js:209:5)
    at Match.Writable.write (_stream_writable.js:180:11)
    at write (_stream_readable.js:573:24)

With the following modules:

bower@0.10.0 /home/alfred/nvm/v0.8.25/lib/node_modules/bower
├── abbrev@1.0.4
├── archy@0.0.2
├── colors@0.6.0-1
├── tmp@0.0.20
├── async@0.2.9
├── mkdirp@0.3.5
├── stable@0.1.3
├── nopt@2.0.0
├── semver@2.0.10
├── which@1.0.5
├── request@2.11.4
├── hogan.js@2.0.0
├── lodash@1.0.1
├── fstream@0.1.23 (inherits@1.0.0, graceful-fs@2.0.0)
├── rimraf@2.0.3 (graceful-fs@1.1.14)
├── tar@0.1.17 (inherits@1.0.0, block-stream@0.0.6)
├── promptly@0.1.0 (read@1.0.5)
├── glob@3.1.21 (inherits@1.0.0, graceful-fs@1.2.3, minimatch@0.2.12)
├── read-package-json@0.1.13 (lru-cache@2.0.4, slide@1.1.4, graceful-fs@1.2.3, semver@1.1.4, npmlog@0.0.4)
├── rc@0.3.0 (deep-extend@0.2.5, ini@1.1.0, optimist@0.3.7)
├── unzip@0.1.7 (setimmediate@1.0.1, readable-stream@1.0.2, pullstream@0.4.0, match-stream@0.0.1, binary@0.3.0)
└── update-notifier@0.1.4 (configstore@0.1.2, request@2.22.0)

EvanOxfeld mentioned this issue Mar 12, 2013

Extract match-stream out of parse.js / untilstream.js #18

Closed

timsweb mentioned this issue Mar 13, 2013

Unzip stops prematurely in Node 0.10.0 #20

Open

satazor mentioned this issue Mar 17, 2013

Installing angular.js causes stack overflow on node v0.10.0 bower/bower#312

Closed

EvanOxfeld pushed a commit that referenced this issue Apr 4, 2013

Fix maximum call stack size exceeded errors

d758ebb

Recursively calling Parse._flush caused recursive process.nextTick calls. Fixes #16.

EvanOxfeld mentioned this issue Apr 4, 2013

Fix maximum call stack size exceeded #23

Open

roytruelove mentioned this issue Jul 31, 2013

Invalid non-string/buffer chunk #25

Open

satazor mentioned this issue Sep 1, 2013

limitation related to node-tmp dependency bower/bower#840

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maximum call stack size exceeded #16

Maximum call stack size exceeded #16

brianhiss commented Mar 7, 2013

timsweb commented Mar 7, 2013

brianhiss commented Mar 7, 2013

timsweb commented Mar 7, 2013

ralphholzmann commented Mar 10, 2013

timsweb commented Mar 11, 2013

brianhiss commented Mar 11, 2013

EvanOxfeld commented Mar 12, 2013

satazor commented Mar 17, 2013

EvanOxfeld commented Mar 17, 2013

nprbst commented Mar 27, 2013

satazor commented Apr 3, 2013

EvanOxfeld commented Apr 4, 2013

satazor commented Apr 7, 2013

EvanOxfeld commented Apr 7, 2013

sindresorhus commented Apr 22, 2013

euskadi31 commented Jul 4, 2013

alFReD-NSH commented Jul 22, 2013

Maximum call stack size exceeded #16

Maximum call stack size exceeded #16

Comments

brianhiss commented Mar 7, 2013

timsweb commented Mar 7, 2013

brianhiss commented Mar 7, 2013

timsweb commented Mar 7, 2013

ralphholzmann commented Mar 10, 2013

timsweb commented Mar 11, 2013

brianhiss commented Mar 11, 2013

EvanOxfeld commented Mar 12, 2013

satazor commented Mar 17, 2013

EvanOxfeld commented Mar 17, 2013

nprbst commented Mar 27, 2013

Configuration

1 CPU: Breaks every time

2 CPUs: Works every time

satazor commented Apr 3, 2013

node 0.10.3

bootstrap.zip

jquery-ui-1.10.0.custom.zip

node 0.8.22

bootstrap.zip

jquery-ui-1.10.0.custom.zip

EvanOxfeld commented Apr 4, 2013

satazor commented Apr 7, 2013

EvanOxfeld commented Apr 7, 2013

sindresorhus commented Apr 22, 2013

euskadi31 commented Jul 4, 2013

alFReD-NSH commented Jul 22, 2013