JavaScript heap out of memory

[katydecorah]

The docs team hasn't been able to complete rolling out the latest version of Batfish (with Webpack 4) as some of our larger sites are running into issues. The most notable issue is that mapbox-gl-js-docs runs out of memory during build.

This site has had problems in the past with running out of memory: #311

No other sites have logged this error.

[2019-11-21 18:50:28 UTC]> batfish build
[2019-11-21 18:50:28 UTC]
[2019-11-21 18:50:30 UTC][18:50:28 Batfish]Compiling css ...
[2019-11-21 18:50:30 UTC][18:50:28 Batfish]Starting build ...
[2019-11-21 18:50:30 UTC][18:50:29 Batfish]Finished compiling css in 1.3s
[2019-11-21 18:50:30 UTC][18:50:29 Batfish]Bundling files for the browser ...
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]<--- Last few GCs --->
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC][2638:0x394d6d0]377071 ms: Mark-sweep 1384.8 (1448.4) -> 1384.7 (1448.4) MB, 355.5 / 0.0 ms allocation failure GC in old space requested
[2019-11-21 18:56:47 UTC][2638:0x394d6d0]377437 ms: Mark-sweep 1384.7 (1448.4) -> 1384.7 (1413.9) MB, 366.4 / 0.0 ms last resort GC in old space requested
[2019-11-21 18:56:47 UTC][2638:0x394d6d0]377772 ms: Mark-sweep 1384.7 (1413.9) -> 1384.7 (1413.9) MB, 334.3 / 0.0 ms last resort GC in old space requested
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]<--- JS stacktrace --->
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]==== JS stack trace =========================================
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]Security context: 0x36699f4a5879 <JSObject>
[2019-11-21 18:56:47 UTC] 2: replace(this=0xb9e0b8787f1 <Very long string[1309050]>,0x18930404a1e9 <JSRegExp <String[44]: [\\\b\f\n\r\v\t\x22\x27\u2028\u2029\0\ufeff]>>,0x18930404a221 <JSFunction (sfi = 0x372dee28b2b9)>)
[2019-11-21 18:56:47 UTC] 3: /* anonymous */(aka /* anonymous */) [/work/mapbox-gl-js-docs/node_modules/terser/dist/bundle.min.js:~1] [pc=0x33291417ea62](this=0x9fd2a5822d1 <undefined>,t=0xb9e0b8787f1 <Very long string[13...
[2019-11-21 18:56:47 UTC]
[2019-11-21 18:56:47 UTC]FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
[2019-11-21 18:56:47 UTC] 1: node::Abort() [batfish]
[2019-11-21 18:56:47 UTC] 2: 0x8c21ec [batfish]
[2019-11-21 18:56:47 UTC] 3: v8::Utils::ReportOOMFailure(char const*, bool) [batfish]
[2019-11-21 18:56:47 UTC] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [batfish]
[2019-11-21 18:56:47 UTC] 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [batfish]
[2019-11-21 18:56:47 UTC] 6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [batfish]
[2019-11-21 18:56:47 UTC] 7: v8::internal::Runtime_RegExpExecMultiple(int, v8::internal::Object**, v8::internal::Isolate*) [batfish]
[2019-11-21 18:56:47 UTC] 8: 0x3329127842fd
[2019-11-21 18:56:47 UTC]Aborted

Progression of error

(Unfortunately I think the build logs expired since they aren't recent. I waited too long to move this into its own issue from #317 but here is a most recent log which is pretty much the same failure for all failed builds.)

Here's a progression of the commits and Batfish versions made to mapbox-gl-docs:

Batfish@1.9.8

This Batfish version uses Webpack@3, I'm providing the latest build on this version in case it's helpful to compare the logs.

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/commit/3b0395d6bebbaf9336f00403a83c07d1f3b3a956
• ✅ Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=bda69ba7-2913-451f-9246-2cf97a5d75de#L527

Batfish@1.10.2

This Batfish version upgrades to Webpack@4 (plus bug fixes).

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/pull/60/commits/a149c15ab901700951c680cee8ed8d63ed87122e
• ❌Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=6266e924-e909-4178-9e8f-a0a5e992f807#L528

When debugging with node --inspect, the code had pointed at writing the stats.json file as running out of memory. So we shipped #307 in Batfish@1.11.0 to place stats.json behind a flag.

However, I'm seeing in this build log that it was also pointing the the terser plugin which handles minification. It's likely failing because the chunks are too large (the api and style-spec pages are long).

Batfish@1.11.0

This Batfish version places the generation of stats.json behind a flag. #307

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/pull/60/commits/5ca5149c38bd3b7bbbefda15d298c5367e6e0b5f
• ❌Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=cc912089-c771-42c3-962b-ed0323338fde#L528

Batfish@1.11.0 + increased Node memory

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/commit/458d392281682c0db39515224e53a466e467a03b
• ✅Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=ebde70c8-66cf-4033-b753-2963386c776c#L529

Batfish@1.11.1

This Batfish version increased maxAsyncRequests and maxInitialRequests. #317

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/pull/60/commits/95241238108308794325ff8d06a56c06ad8590fe
• ✅Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=c750aa26-bdf2-46da-ba9a-dce90e9b22c6#L529

Note: we reverted this change in 1.11.2 because increases maxAsyncRequests and maxInitialRequests effects performance on older mobile devices.

Batfish@1.11.2

This version reverted 1.11.1 and fixed an issue where markdown and JS files were not being watched locally and md and js files were being copied on production.

• Commit: https://github.com/mapbox/mapbox-gl-js-docs/commit/8dc25cd819a3e1575c5a7a2b4deda31954047fff
• ❌Build log: https://hey.mapbox.com/log-link/?logGroup=/aws/codebuild/artifacts-production-publisher&logStream=23f50b42-9eba-450d-8816-37eab5f412c4#L538-583

Possible options

• Since the build is running out of memory during minification, I am watching Process aborts with 'out of memory' when using 2.0.0 webpack-contrib/terser-webpack-plugin#143 since it could be related. I confirmed that a bottleneck happens during minification by setting minimize: false: mapbox-gl-js-docs built without running out of memory and in a fraction of the time.
• We could increase the memory via --max-old-space-size to the build script, which would provide mapbox-gl-js-docs enough memory to build. However, this doesn't solve the root problem, but could buy us time until we are able find a fix.

cc @mapbox/docs

[katydecorah]

We were able to track this issue to a utility that transformed lots of markdown to JavaScript in the Mapbox GL JS site. The issue is unrelated to Batfish 😅