RangeError in querystream #929

Closed
kof opened this Issue May 23, 2012 · 3 comments

Comments

Projects
None yet
3 participants

kof commented May 23, 2012

Hi,

I have a worker, which is doing periodically a lot of calculations, therefore it streams a lot of data to node. The issue I have is "RangeError: Maximum call stack size exceeded". I read the code and saw that there is already an attempt to fix it in querystream.js

// nextTick is necessary to avoid stack overflows when
// dealing with large result sets. yield occasionally.
if (!(++this._ticks % 20)) {
  process.nextTick(function () {
    self._cursor.nextObject(function (err, doc) {
      self._onNextObject(err, doc);
    });
  });
} else {
  self._cursor.nextObject(function (err, doc) {
    self._onNextObject(err, doc);
  });
}

I have fixed my issue by just using every time process.nextTick. I suppose it depends on amount of data and the way I am processing it, so not everyone will get this issue.

Possible solutions:

  1. check if the stack is really going to overflow and then use nextTick
  2. let user setup the frequence of nextTick usage

My env:
nodel v0.6.17
npm 1.1.21
mongoose 2.6.4

Here is some stack trace for you:

{ first_line: 'RangeError: Maximum call stack size exceeded',
frames: 
[ { named_location: 'log',
 filename: '/xxx/node_modules_local/log/index.js',
 line: 75,
 character: 27,
 _filedata: null },
{ named_location: 'QueryStream.<anonymous>',
 filename: '/xxx/bin/downgrade.js',
 line: 9,
 character: 9,
 _filedata: null },
{ named_location: 'QueryStream.emit',
 filename: 'events.js',
 line: 67,
 character: 17,
 _filedata: null },
{ named_location: 'QueryStream.destroy',
 filename: '/xxx/node_modules/mongoose/lib/querystream.js',
 line: 169,
 character: 10,
 _filedata: null },
{ named_location: 'QueryStream._onNextObject',
 filename: '/xxx/node_modules/mongoose/lib/querystream.js',
 line: 117,
 character: 24,
 _filedata: null },
{ named_location: '<anonymous>',
 filename: '/xxx/node_modules/mongoose/lib/querystream.js',
 line: 103,
 character: 12,
 _filedata: null },
{ named_location: '[object Object].<anonymous>',
 filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/cursor.js',
 line: 529,
 character: 9,
 _filedata: null },
{ named_location: '[object Object].g',
 filename: 'events.js',
 line: 156,
 character: 14,
 _filedata: null },
{ named_location: '[object Object].emit',
 filename: 'events.js',
 line: 88,
 character: 20,
 _filedata: null },
{ named_location: 'Db._callHandler',
 filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js',
 line: 1290,
 character: 25,
 _filedata: null },
{ named_location: '<anonymous>',
 filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js',
 line: 329,
 character: 30,
 _filedata: null },
{ named_location: 'Array.0',
 filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js',
 line: 94,
 character: 11,
 _filedata: null },
{ named_location: 'EventEmitter._tickCallback',
 filename: 'node.js',
 line: 192,
 character: 40,
 _filedata: null } ],
original_error: [RangeError: Maximum call stack size exceeded] }
Collaborator

aheckmann commented May 23, 2012

I like the idea of it being configurable

On May 23, 2012, at 7:41 AM, Oleg Slobodskoireply@reply.github.com wrote:

Hi,

I have a worker, which is doing periodically a lot of calculations, therefore it streams a lot of data to node. The issue I have is "RangeError: Maximum call stack size exceeded". I read the code and saw that there is already an attempt to fix it in querystream.js

// nextTick is necessary to avoid stack overflows when
// dealing with large result sets. yield occasionally.
if (!(++this._ticks % 20)) {
process.nextTick(function () {
self._cursor.nextObject(function (err, doc) {
self._onNextObject(err, doc);
});
});
} else {
self._cursor.nextObject(function (err, doc) {
self._onNextObject(err, doc);
});
}

I have fixed my issue by just using every time process.nextTick. I suppose it is related with an amount of data and the way I am processing it, so not everyone will get this issue.

Possible solutions:

  1. check if the stack is really going to overflow and then use nextTick
  2. let user to setup the frequence of nextTick usage

My env:
nodel v0.6.17
npm 1.1.21
mongoose 2.6.4

Here is some stack trace for you:

{ first_line: 'RangeError: Maximum call stack size exceeded',
frames:
[ { named_location: 'log',
filename: '/xxx/node_modules_local/log/index.js',
line: 75,
character: 27,
_filedata: null },
{ named_location: 'QueryStream.',
filename: '/xxx/bin/downgrade.js',
line: 9,
character: 9,
_filedata: null },
{ named_location: 'QueryStream.emit',
filename: 'events.js',
line: 67,
character: 17,
_filedata: null },
{ named_location: 'QueryStream.destroy',
filename: '/xxx/node_modules/mongoose/lib/querystream.js',
line: 169,
character: 10,
_filedata: null },
{ named_location: 'QueryStream._onNextObject',
filename: '/xxx/node_modules/mongoose/lib/querystream.js',
line: 117,
character: 24,
_filedata: null },
{ named_location: '',
filename: '/xxx/node_modules/mongoose/lib/querystream.js',
line: 103,
character: 12,
_filedata: null },
{ named_location: '[object Object].',
filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/cursor.js',
line: 529,
character: 9,
_filedata: null },
{ named_location: '[object Object].g',
filename: 'events.js',
line: 156,
character: 14,
_filedata: null },
{ named_location: '[object Object].emit',
filename: 'events.js',
line: 88,
character: 20,
_filedata: null },
{ named_location: 'Db._callHandler',
filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js',
line: 1290,
character: 25,
_filedata: null },
{ named_location: '',
filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js',
line: 329,
character: 30,
_filedata: null },
{ named_location: 'Array.0',
filename: '/xxx/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js',
line: 94,
character: 11,
_filedata: null },
{ named_location: 'EventEmitter._tickCallback',
filename: 'node.js',
line: 192,
character: 40,
_filedata: null } ],
original_error: [RangeError: Maximum call stack size exceeded] }


Reply to this email directly or view it on GitHub:
LearnBoost#929

Collaborator

aheckmann commented Jun 6, 2012

In the next release I'm gonna try a trampoline here which avoids the recursion all together. configuring the batchSize will control how many docs you wish to process in each chunk.

T.find().batchSize(1000 /*default*/).stream();

aheckmann closed this in 6f5feff Jun 6, 2012

@aheckmann aheckmann added a commit to aheckmann/mongoose that referenced this issue Jun 7, 2012

@aheckmann aheckmann CursorStream: No stack overflow on any size result
Consider configuring the number of docs to process at a time
through the query#batchSize option / method
which defaults to 1000.

    T.find().batchSize(2000).stream();

closes #929
e37ab4e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment