JSON.stringify error and writeFileSync error

[ralic]

[Issue]

Here is the use case :
There are ~ 95000 txt files stored in the same folder
Each txt files contains array of objects, size=50.
I tried to aggregate all these txt's files's objects into an array.
There are 2037356 obj(s) in the Array, and each object stringified length reported to be no more than "2237" after checked each element's length by JSON.stringify(obj).length.

When I try to write them into a large json file as in code attached below. There are some errors returned. I wonder whether I shall avoid using writeFileSync and go for fs.append to prevent "RangeError: Invalid string length" in such case.

Please advise... thanks !
Note. I've increased heap to 8192 to prevent process out of memory error.

app.js:

var shell=require('shelljs/global')
var lineReader = require('line-reader');
var fs  = require("fs");
var jsonfile = require('jsonfile')
var write2file = 'data.json'

var strings=[];
var objs=[];
// line ~=int range of 200~99000;
lineReader.eachLine('zips',function(line,last){
    var zip='./info/'+line+'.txt';
    console.log(zip);
    var string= fs.readFileSync(zip).toString()
    var arrOfObj = eval(string);
    delete string;
    for ( var i in arrOfObj) {
        console.log(JSON.stringify(arrOfObj[i]).length)
        objs.push(arrOfObj[i])
    }
      if(last){
        console.log(objs.length)
        jsonfile.writeFileSync(write2file, objs, function (err) {
        console.error(err)
        });
        console.log("Done");
      }
});

shjs  app.js

/Users/raliclo/kroger/node_modules/jsonfile/index.js:108
  let str = JSON.stringify(obj, options.replacer, spaces) + '\n'
                 ^

RangeError: Invalid string length
    at Object.stringify (native)
    at Object.writeFileSync (/Users/raliclo/kroger/node_modules/jsonfile/index.js:108:18)
    at /Users/raliclo/kroger/get.js:27:14
    at /Users/raliclo/kroger/node_modules/line-reader/lib/line_reader.js:279:15
    at getLine (/Users/raliclo/kroger/node_modules/line-reader/lib/line_reader.js:166:7)
    at Object.nextLine (/Users/raliclo/kroger/node_modules/line-reader/lib/line_reader.js:175:11)
    at Immediate.readNext [as _onImmediate] (/Users/raliclo/kroger/node_modules/line-reader/lib/line_reader.js:269:14)
    at tryOnImmediate (timers.js:543:15)
    at processImmediate [as _immediateCallback] (timers.js:523:5)

[ecdeveloper]

I'm not sure why you're getting this particular issue, but I'd suggest using streams. Your approach is potentially risky, because at some point you may run out of memory.

[mscdex]

You're exceeding the max string length when converting the (final) value to a string. This is a V8-imposed limit. As @ecdeveloper suggested, if you're sticking with JSON you're better off splitting your data up in some way so that V8 doesn't have to generate one large string.

[ralic]

Thanks! I will make another approach to the code to make it work.

Just want to share some opinion, referred from ECMA2106 String spec : http://www.ecma-international.org/ecma-262/7.0/#sec-ecmascript-language-types-string-type

The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 2^53-1 elements.

In this case, log2(2037356*2372) is about 2^32 for the string length.
So I assumed that it may work...

Or maybe I have some misunderstanding in the spec.. ?

[mscdex]

V8 sets the string length limit to (1 << 28) - 16 which is 268,435,440. According to your original statistics, you easily surpassed this value.

Also, the note about the maximum number of elements in a string is a relatively new addition to the spec (it was only introduced in ES6/ES2015). I would not be surprised if other browser vendors have not yet increased their limits to this new value.

[vsemozhetbyt]

MDN:

In Firefox (SpiderMonkey) the maximum string size is 2²⁸ -1 (0xFFFFFFF)

In the last Firefox Nightly

'*'.repeat(Math.pow(2, 28)).length

still throws, while

'*'.repeat(Math.pow(2, 28) - 1).length

is OK.

[ecdeveloper]

In any event, sticking such a big amount of data in a variable is a bad practice. Streams ftw :)

[mscdex]

Closing this for now as this is expected behavior.