On a large file (556MB), "RangeError: Invalid string length" in Node 8.7.0, "JavaScript heap out of memory" in Node 10.12.0

[sompylasar]

Version

fx@3.1.0

Input

A one-JSON-per-line file of about 556MB size.

Node 8.7.0

/Users/ivanbabak/.nvm/versions/node/v8.7.0/lib/node_modules/fx/index.js:111
      buff += chunk
              ^

RangeError: Invalid string length
    at Socket.stdin.on (/Users/ivanbabak/.nvm/versions/node/v8.7.0/lib/node_modules/fx/index.js:111:15)
    at emitNone (events.js:105:13)
    at Socket.emit (events.js:207:7)
    at emitReadable_ (_stream_readable.js:514:10)
    at emitReadable (_stream_readable.js:508:7)
    at addChunk (_stream_readable.js:275:7)
    at readableAddChunk (_stream_readable.js:247:13)
    at Socket.Readable.push (_stream_readable.js:209:10)
    at Pipe.onread (net.js:587:20)

Node 10.12.0

<--- Last few GCs --->

[30850:0x102802200]     3051 ms: Mark-sweep 847.8 (1080.7) -> 847.8 (1049.2) MB, 27.3 / 0.0 ms  (average mu = 0.854, current mu = 0.000) last resort GC in old space requested
[30850:0x102802200]     3076 ms: Mark-sweep 847.8 (1049.2) -> 847.8 (1049.2) MB, 25.3 / 0.0 ms  (average mu = 0.727, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x38ad3f32bf29 <JSObject>
    0: builtin exit frame: parse(this=0x38ad3f3204a1 <Object map = 0x38ad3df842a9>,0x38ad76ba6299 <Very long string[582662191]>,0x38ad3f3204a1 <Object map = 0x38ad3df842a9>)

    1: main(aka main) [0x38adde819559] [/Users/ivanbabak/.nvm/versions/node/v10.12.0/lib/node_modules/fx/index.js:47] [bytecode=0x38ad925ff261 offset=62](this=0x38ad428826f1 <undefined>,input=0x38ad76ba6299 <Very long strin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x10003ae75 node::Abort() [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 2: 0x10003b07f node::OnFatalError(char const*, char const*) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 3: 0x1001a6a85 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 4: 0x100574422 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 5: 0x10057d894 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 6: 0x10054f6d4 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 7: 0x1006818a9 v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 8: 0x100266edb v8::internal::Builtin_Impl_JsonParse(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/ivanbabak/.nvm/versions/node/v10.12.0/bin/node]
 9: 0x26807c45bf9d 
10: 0x26807c4118d5 
11: 0x26807c4118d5 
12: 0x26807c4118d5 
Abort trap: 6

Needs some streaming processing for large files, jq has one.

Also looks like it doesn't support one-JSON-per-line like jq (but that's a separate issue).

[antonmedv]

Now fx doesn’t support streaming but defenitly will do.