This repository has been archived by the owner on Jul 30, 2019. It is now read-only.
Fix multipage building #833
Comments
|
Confirmed. (Just checked.) |
|
I noticed after checking the TravisCI logs that the w3c/html-tools version of the multipage tool is not being used... so I'm flattening that with adrianba's latest copy, and then switching the build tools to work off of that copy (so that we don't depend on adrianba's private GitHub repo... :) |
|
OK w3c/html-tools#5 is the PR that updates html-tools. |
|
And this commit swaps the build infrastructure... 424e74d |
|
So does this mean a need to change the build documentation? |
|
I watched the NodeJS process hit 1.99GB of memory before the GC allocator finally tossed-up its hands. |
travisleithead
added a commit
to w3c/html-tools
that referenced
this issue
Mar 31, 2017
Fixes an issue of linear memory growth in the NodeJS multipage.js script
which was causing out-of-memory exceptions on Windows. After testing the
issue, I discovered that the splitter reaches about 2GB of memory and then
fails to complete due to the inability of the NodeJS GC to allocate objects.
I also noticed that the splitter became progressively slower as each new
section was generated.
The HTML build process executes the splitter with a few flags (that appear
to have worked around the problem?):
node --max_old_space_size=2048 --expose-gc ./tools/multipage.js single-page.html ./out/
After reviewing the splitter script's code, it became apparent that there was a memory
growth problem in the way that several "whacko" documents were being linked together in
the final section-building loop. The memory structure was basically the following:
sections = [
{
id: <string>,
title: <string>,
node: <ref to a section from single_page.html whacko document>
}
];
As each section building loop ran, the code placed the section[i].node into a new
whacko document representing the template skeleton for that section.
<template whacko document>
<nodes in template whacko document>
..
<ref to single_page.html whacko document>
</..>
</..>
Apparently, the way that a whacko document works is that it use jQuery under-the-hood,
and jQuery links nodes into the document tree in a graph of relationships such that
the inclusion of a ref to the single_page.html's whacko document meant that the
section[i].node reference was now keeping the entire <template whacko document> alive.
As each new <template whacko document> was created and had pieces of it mixed in from
<singe_page.html whacko document> the prior document was unable to be collected, and
memory continued to grow.
The change serializes the section[i].node contents so that there is no GC graph link
possible between whacko documents. Sure enough, the GC is able to kick in and I see
memory growth hit 1GB and then drop back to 500MB in cycles.
|
I have a fix for this now over in the html-tool repo: w3c/html-tools#6 |
travisleithead
added a commit
to w3c/html-tools
that referenced
this issue
Mar 31, 2017
Fix for HTML issue: w3c/html#833
|
Merged. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Reportedly runs windows machines out of memory. /@LJWatson
The text was updated successfully, but these errors were encountered: