How to deal with strings longer than String::kMaxValue

[alexdima]

• Node.js Version: 7.4.0
• v8 Version: 5.6.326.50
• OS: macOS Sierra
• Scope (install, code, runtime, meta, other?): basic string manipulation
• Module (and version) (if relevant): Buffer

Hi, I'm working on VS Code (based on Electron), and I'm looking into improving our memory usage when dealing with large files in microsoft/vscode#30180.

As part of testing opening large files in VS Code, we have some code paths where we end up creating a large string. e.g. select all + copy results in a large string being created that needs to be placed in the clipboard, etc.

v8's String::kMaxValue is defined as (1 << 28) - 16, which is 268,435,440 or about 256 MB and trying to create a string longer than that gives a "Invalid string length" exception straight from v8. I've seen nodejs/node#1374 touches on the topic and mentions improvements to ExternString.

Is ExternString exposed in any way to nodejs applications, or what is your guidance, when one needs to work with strings above 256MB?

Thank you.

[bnoordhuis]

You can't use strings > 256 MB at the moment; external strings are still subject to the hard limit.

There are plans to raise or remove the limit. I don't have the issue number at hand right now but it was discussed on V8's bug tracker recently.

[alexdima]

Thanks!

That's a bummer, but I guess we could workaround it by writing our own C++ node module where we pass the text to be placed in the clipboard via an array of strings and do the concatenation in C++, I suppose.

Was https://monorail-prod.appspot.com/p/v8/issues/detail?id=6148 the one you meant ?

[I'm not sure how you like to manage the issues here, please feel free to close this one]

[bnoordhuis]

Yep, that's the one. I'll close out the issue.

Aside: a useless trick for stretching the limit: it's 256 MB characters, not bytes, so if you pack a one-byte string into a two-byte string (twobyte[i] = 256 * onebyte[2*i] + onebyte[2*i+1]), you can create a string that is 512 MB bytes big with v8::String::NewFromTwoByte().

[dherman]

For reference, according to that V8 bug tracker thread mentioned above, the length limit was increased to 2^30 - 25 (i.e., just shy of 1GB) in V8 6.2.10, which (if I'm reading the Node release table correctly) means the new higher limit was available starting in Node 8.10.0.

[danielsokolowski]

This has regressed back to 2^28 due to pointer compression, see: v8/v8@ea56bf5