Heap out of memory when reading events from file (50,000 tests)

[barakman]

Running a massive set of tests, I get this:

  51973 passing (5h)

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewUninitializedFixedArray
 5: v8::internal::WasmDebugInfo::SetupForTesting
 6: v8::internal::interpreter::BytecodeArrayRandomIterator::UpdateOffsetFromIndex
 7: 000001FEEE7043C1

The contents of file .solcover.js:

module.exports = {
    port:           8555,
    norpc:          true,
    testCommand:    "node --max-old-space-size=8192 ../../node_modules/truffle/build/cli.bundled.js test --network=coverage",
    compileCommand: "node --max-old-space-size=8192 ../../node_modules/truffle/build/cli.bundled.js compile --network=coverage",
};

Environment:

• Windows 10
• node 8.11.2
• npm 6.1.0

[cgewecke]

@barakman Modified your issue title to disambiguate your case from the one we already advise about in the FAQ.

I don't know how to fix this. If anyone has any ideas, please weigh in.

[barakman]

@cgewecke:

I did a quick drill-through on the solidity-coverage source code.

The reason for this out-of-memory-problem is the line:

.on('line', line => this.events.push(line))

In file app.js / function generateReport.

I believe that a working solution would be to save these lines into a text file instead of an array.

You will then also need to change function generate (in file coverageMap.js), to take a text file instead of an array as input.

UPDATE:

In the solution above, you would be replicating the allFiredEvents stream and then pass it from function generateReport to function generate. A simpler, cleaner and more efficient solution would be to just pass the original stream as is. Technically, this requires very little changes.

In file app.js / function generateReport:

Change this:

      this.events = [];
      reader
        .on('line', line => this.events.push(line))
        .on('close', () => {
          ...
        });

To this:

      this.events = reader;
      ...

In file coverageMap.js / function generate:

Change this:

    for (let idx = 0; idx < events.length; idx++) {
      const event = JSON.parse(events[idx]);
      ...
    }

To this:

    events.on('line', function (line) {
      const event = JSON.parse(line);
      ...
    });

[barakman]

@cgewecke:

For anyone reading this, here is a hot-fix (patch) until the official release is fixed:

In your package.json file, add:

  "scripts": {
    "install": "node fix-solidity-coverage.js"
  }

Next to your package.json file, add file fix-solidity-coverage.js containing:

let fs = require("fs");

function fix(fileName, tokens) {
    let data = fs.readFileSync(fileName, {encoding: "utf8"});
    for (let token of tokens)
        data = data.split(token.prev).join(token.next);
    fs.writeFileSync(fileName, data, {encoding: "utf8"});
    console.log("File " + fileName + " fixed");
}

fix("./node_modules/solidity-coverage/lib/app.js", [
    {prev: "      this.events = [];", next: "      this.events = reader;"},
    {prev: "      reader\n        .on('line', line => this.events.push(line))\n        .on('close', () => {", next: ""},
    {prev: "          }\n        });\n    });\n  }\n", next: "          }\n    });\n  }\n"}]
);

fix("./node_modules/solidity-coverage/lib/coverageMap.js", [
    {prev: "    for (let idx = 0; idx < events.length; idx++) {", next: "    events.on('line', function (line) {"},
    {prev: "      const event = JSON.parse(events[idx]);", next: "      const event = JSON.parse(line);"},
    {prev: "      }\n    }\n", next: "      }\n    });\n"}]
);

[barakman]

Note:

The solution above does not work, because the allFiredEvents stream is being read asynchronously.

Here is a working solution:

In your package.json file, add:

  "scripts": {
    "install": "node fix-solidity-coverage.js"
  }

Next to your package.json file, add file fix-solidity-coverage.js containing:

let fs = require("fs");

function fix(fileName, tokens) {
    console.log("Fixing " + fileName);
    let data = fs.readFileSync(fileName, {encoding: "utf8"});
    for (let token of tokens)
        data = data.split(token.prev).join(token.next);
    fs.writeFileSync(fileName, data, {encoding: "utf8"});
}

fix("./node_modules/solidity-coverage/lib/app.js", [
    {prev: "events.push", next: "coverage.processEvent"}]
);

fix("./node_modules/solidity-coverage/lib/coverageMap.js", [
    {prev: "  generate(events, pathPrefix) {", next: "  processEvent(line) {"},
    {prev: "    for (let idx = 0; idx < events.length; idx++) {", next: ""},
    {prev: "      const event = JSON.parse(events[idx]);", next: "      const event = JSON.parse(line);"},
    {prev: "    // Finally, interpret the assert pre/post events", next: "  generate(events, pathPrefix) {"}]
);

[cgewecke]

Note to self - 0.7.0 will resolve this by using an integer counter for intrumentation hits rather than processing them as individually enumerated events. Mem consumption should be well within bounds (for this step anyway).

It looks like this problem is also affecting colonyNetwork's test suites and forcing them to split their tests into two CI jobs. . .

[cgewecke]

0.7.0 fixes this by handling everything in-process and it should have a much smaller memory footprint since hits are managed with a counter.

Thanks for opening @barakman, hope you have a great new year!

[barakman]

@cgewecke: You're welcome.
This one in specific has been both exhilarating and excruciating to investigate and resolve (well, workaround)... As a matter of fact, in terms of difficulty, I believe that it was second to Truffle 4 issue 729 only.
Thank you for providing this outstanding tool, which IMO is a must in every Ethereum project.
Happy New Year :)