JS driver - Maximum call stack size exceeded on big arrays

[neumino]

While doing some random tests, I found that this fails with a RangeError: Maximum call stack size exceeded

var a = eval('['+new Array(100002).join('{}, ')+'{}]');
r.expr(a)

It's also broken in 1.13.x
And it works fine with rethinkdbdash. Maybe it's a coffee-script issue?

[srh]

Unlikely. It's probably bad recursive code.

[neumino]

The error is triggered by

/home/ssd3/michel/projects/rethinkdb-all/rethinkdb/build/packages/js/ast.js:1407
    return MakeArray.__super__.constructor.apply(this, arguments);

It looks like this is an expected behavior: https://code.google.com/p/v8/issues/detail?id=2896

Should we make the MakeArray class not extend RDBOp? And just manually copy what we need?
Refactoring everything is definitively not worth the effort since making rethinkdbdash official-compliant is probably less work.

Moving to subsequent for now.

[nathanboktae]

I've been playing around with rethinkdb for an hour and ran into this 2 year old issue. rethinkdbdash just works.... 😒

[danielmewes]

@deontologician can you tell how much work it would be to fix this?
I don't really understand what's going on here.

[deontologician]

The issue is that the Array.apply(func, values) takes the entries in the array and turns them into arguments to func. This all goes onto the stack and there's a relatively small stack limit to avoid accidental infinite recursion. Fortunately, what the function itself is doing is just taking all of its arguments and converting them back into an array, so there's no real reason to do this spread operation. Especially since the Ast constructors are internal, we don't need them to have a convenient way to call them, so we can dispense with the spread and .apply stuff.

It's not hard to fix, but a bit sloggish, since we have to go to every use of super(args...) in RdbOp subclasses and convert them to passing the array directly (as well as changing the RdbOp constructor to accept an array instead of arguments).

[danielmewes]

Ok, thanks for the explanation @deontologician .

@nathanboktae Do you have a work-around for this or is this blocking you?

[nathanboktae]

Blocked if I was still using this package, but now I'm streaming results with rethinkdbdash

  client // pg client
    .query(new QueryStream('SELECT * FROM users'))
    .pipe(r.db('hackathon').table('users').toStream({ writable: true }))

It just works and it's fast. Thanks @neumino !

[deontologician]

The pipes in rethinkdbdash are really nice. We are still planning on making it the official driver

[Iiridayn]

Just ran into this while developing a data migration script - currently the largest insert is 117,094 records. Workaround was to chunk them at 60k (70k didn't work) and insert in a loop.