Prettier crashes on formating large files

[trzecieu]

Hi!
I'm working with large ASM.js files generated by Emscripten, when I debug those I use tools like Prettier to make code at lease a little bit more readable.

In case of prettier I'm getting:

<--- Last few GCs --->

[17699:0x2702f10]    20187 ms: Mark-sweep 1397.2 (1424.1) -> 1396.4 (1423.6) MB, 596.2 / 0.0 ms  (average mu = 0.145, current mu = 0.011) allocation failure scavenge might not succeed
[17699:0x2702f10]    20833 ms: Mark-sweep 1397.4 (1424.1) -> 1396.7 (1424.1) MB, 643.6 / 0.0 ms  (average mu = 0.076, current mu = 0.004) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x2957d235c01d]
    1: StubFrame [pc: 0x2957d2313490]
Security context: 0x16bb6979e549 <JSObject>
    2: willBreak(aka willBreak) [0xccf8baf92a9] [/home/[moderated]/node_modules/prettier/bin-prettier.js:~9173] [pc=0x2957d2724356](this=0x37817e8826f1 <undefined>,doc=0x0388ed06f201 <Object map = 0x2804a52f5a01>)
    3: arguments adaptor frame: 3->1
    4: printArgumentsList(aka printArgumentsList) [0xccf8bafbd69...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8c02c0 node::Abort() [node]
 2: 0x8c030c  [node]
 3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xebe752  [node]
 6: 0xebe858 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xeca982 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xecb2b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xecdf21 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe97444 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x1136d5e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x2957d235c01d
[1]    17699 abort (core dumped)  npx prettier DEF-2515.j

➜ npm --version
6.2.0
➜ node --version
v10.9.0
➜ cat /etc/os-release
NAME=Fedora
VERSION="28 (Workstation Edition)"
ID=fedora
VERSION_ID=28
PLATFORM_ID="platform:f28"
PRETTY_NAME="Fedora 28 (Workstation Edition)"

[suchipi]

It looks like your computer might be running out of memory- see this line:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I'm not saying we couldn't improve this, but as a workaround, you could try formatting the file on a box with more memory.

[trzecieu]

Hi,
I've just tested and I've noticed this error after ~1.4GB of consumed RAM only and I still have a lot of unused memory.
I was able to move limit by calling NODE_OPTIONS=--max_old_space_size=4096 prettier asm.js but still it wasn't enough to format large files.

An additional observation it is that it looks a lot of temporary objects are involved in the process, as JS's GC works intensively:

Probably my case is an edge case - who else prettifies asm.js? Nonetheless, thank you for your attention.

[mkreim]

Same problem for me with a ca. 20Mb file that exports a large array of json objects. I wanted to use this for mocking external data. Memory seems not to be the problem. When prettier crashed I still have 6GB of memory left.

$ prettier --single-quote --trailing-comma es5 --print-width 100 --write "{src,__{tests,mocks}__}/**/*.js"
[...]
<--- Last few GCs --->

[24654:0x33ab300]    59157 ms: Scavenge 1392.4 (1421.8) -> 1391.9 (1422.3) MB, 3.8 / 0.0 ms  (average mu = 0.183, current mu = 0.122) allocation failure 
[24654:0x33ab300]    59167 ms: Scavenge 1392.6 (1422.3) -> 1392.1 (1422.8) MB, 3.9 / 0.0 ms  (average mu = 0.183, current mu = 0.122) allocation failure 
[24654:0x33ab300]    59176 ms: Scavenge 1392.8 (1422.8) -> 1392.3 (1423.8) MB, 3.8 / 0.0 ms  (average mu = 0.183, current mu = 0.122) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x174d6375be1d]
Security context: 0x15e14571e6e1 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x29ac06232bc1] [/home/x/frontend/node_modules/prettier/bin-prettier.js:~24494] [pc=0x174d63e37071](this=0x133f83d026f1 <undefined>,prop=0x29ac06230361 <Object map = 0x15b7f33d49a9>)
    2: arguments adaptor frame: 3->1
    3: map [0x15e145706e59](this=0x29ac0621cee1 <JSArray[86]>,0x29ac06232bc...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8db900 node::Abort() [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 2: 0x8db94c  [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 3: 0xad6c1e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 4: 0xad6e54 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 5: 0xec44e2  [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 6: 0xec45e8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 7: 0xed06c2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 8: 0xed0ff4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
 9: 0xed3c61 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
10: 0xe9d0e4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
11: 0x113c88e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/x/.nvm/versions/node/v10.15.0/bin/node]
12: 0x174d6375be1d 
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[alexander-akait]

@mkreim can you create reproducible test repo?

[suchipi]

@mkreim You might find success by leveraging the NODE_OPTIONS environment variable that @trzecieu mentioned

[mkreim]

@mkreim can you create reproducible test repo?

@evilebottnawi I will ask, if I am allowed to do this. In principle its is one 20Mb file, but it contains data from a customer device.

[alexander-akait]

@mkreim maybe you can create fake data in same format?

[mkreim]

@suchipi Thanks a lot. That actually worked for me:

$ /usr/bin/time --verbose yarn spsprettier
yarn run v1.12.3
$ NODE_OPTIONS=--max_old_space_size=4096 prettier --single-quote --trailing-comma es5 --print-width 100 --write src/spsSampleData_2.js
src/spsSampleData_2.js 26485ms
Done in 27.00s.
        Command being timed: "yarn spsprettier"
        User time (seconds): 41.90
        System time (seconds): 2.03
        Percent of CPU this job got: 161%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:27.22
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2966620
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 848333
        Voluntary context switches: 51786
        Involuntary context switches: 6432
        Swaps: 0
        File system inputs: 0
        File system outputs: 16
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[mkreim]

@mkreim maybe you can create fake data in same format?

I fear the data format is the problem they may be suspicious that someone may get something out of the keys in there. In principle its an array with ca. 10000 entries, where each entry is an Object with around 90 keys. While the values are numbers or short strings. Similar like this:

export default [
  {
   key1: 0.1,
   // until
 key86: "blabla"
},
// repeat 10000 times
]

In the original all code is in one line (output from JSON.stringify)

[lydell]

@mkreim Such an array of objects sounds pretty easy to generate?

[trzecieu]

Can be reproduced with Unity's demo of ASM.js

wget https://files.unity3d.com/jonas/AngryBots/Release/AngryBots.js

[austinbutler]

For a passable alternative, BeatifulSoup's prettify() succeeds where Prettier runs out of memory or ends up with call stack exceeded.

[vecerek]

I see a similar error when using Node 16. Switching to Node 14 runs prettier without an issue. Unfortunately, I can't share a link to the repo because it's private.

[MikaelHashiCorp]

I keep hitting this issue with my large files. Do you have a target date for fixing this?

[MikaelHashiCorp]

allocations.json is 137MB. I tried he NODE_OPTIONS environment variable that @trzecieu mentioned:

NODE_OPTIONS=--max_old_space_size=16384 prettier  --write  ./allocations.json

Node memory peaked at 16.42 GB, then crashed:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Anything larger gave me a

RangeError: Set maximum size exceeded

Any other workaround suggestions are welcome.

[MikaelHashiCorp]

Okay, found a workaround. It requires jq and moreutils, which has sponge. The command that worked for me (including an echo to watch progress):

find . -name '*.json' -exec bash -c 'echo {} ; jq "." {} | sponge {}' \;

[emilienbidet]

This works for me:
pnpm dlx prettier --write YOUR_BIG_FILE_PATH