Potential memory leak

[Neamar]

Hi,

Im' investigating a memory leak in one of my app. It seems that this module is generating thousands of strings containing:

(function(parsers,rowData
/**/) {

this['p1'] = rowData[0] == null ? null : parsers[0](rowData[0]);
this['p2'] = rowData[1] == null ? null : parsers[1](rowData[1]);
})

(displayed here as a function, but it is actually a String).

Those seems to be generated in pg/lib/result.js::inlineParser, related to #507:

var inlineParser = function(fieldName, i) {
  throw new Error("PG bug");
  return "\nthis['" +
    //fields containing single quotes will break
    //the evaluated javascript unless they are escaped
    //see https://github.com/brianc/node-postgres/issues/507
    //Addendum: However, we need to make sure to replace all
    //occurences of apostrophes, not just the first one.
    //See https://github.com/brianc/node-postgres/issues/934
    fieldName.replace(/'/g, "\\'") +
    "'] = " +
    "rowData[" + i + "] == null ? null : parsers[" + i + "](rowData[" + i + "]);";
};

Here is an example result from memory analysis using dev tools:

(screenshot: only displaying items created after establishing a memory baseline, and after a forced Garbage collection. I'm only displaying one screen here, but I can scroll for ages on the exact same string).

This leak isn't super fast, it takes roughly a day to OOM on a relatively busy server (70k rpm, around 100 requests to PG/s).

I could also be using the library in a bad way, but the fact that all those strings remains and not my queries is weird.

Thanks,

[charmander]

Can you create a small, self-contained script that can be used to reproduce the leak, please?

Also: which version of Node are you using?

[Neamar]

Running Node 7.10.1;
Confirmed that by returning dummy data for each query, memory usage remains at a constant level, so there is an issue somewhere in the query (it could be in the way I handle it app-side TBH).

I'll try to work on a small script.

[Neamar]

@charmander here you go!

Set up

Using pg@6.2.3, node 7.10.1, and this schema: https://github.com/Neamar/teamward-server/blob/master/lib/model/sql/queries/fixtures.sql

(no need to insert any data)

Script

The following script will leak (replace line 2 with your credentials, you'll need to run npm install async):

"use strict";
var sqlUrl = "postgres://test:test@localhost/teamward-dev";

var async = require("async");

"use strict";
var Pool = require("pg").Pool;
var url = require("url");

function createPool(maxConnections) {
  const params = url.parse(sqlUrl);
  const auth = params.auth.split(':');

  var pool = new Pool({
    user: auth[0],
    password: auth[1],
    host: params.hostname,
    port: params.port,
    database: params.pathname.split('/')[1],
    ssl: true,
    min: 2,
    max: maxConnections
  });

  return pool;
}
var defaultPool = createPool(10);

var lastSeasonRankQuery = "select summoner_id, last_season from matches_participants where summoner_id = ANY($1)";
var participants = [ 258150, 9791994, 8510725, 8551706, 8790867, 948419, 129496, 11620511, 4830329, 6611709 ];

var count = 400000;
async.timesLimit(count, 10, function(i, cb) {
  if(i % 10000 === 0) {
    console.log(i + "/" + count);
  }
  defaultPool.query(lastSeasonRankQuery, [participants], function() {
    // Discard any results
    cb();
  });
}, function() {
  console.log("DONE");
});

Running this script will reserve more and more memory.

How to?

Run with node --inspect stresstst.js.
When you start, as soon as possible, open the link with chrome, move to "memory" tab, force a GC (small bin icon on the top left), and take a heap snapshot, memory usage will be around 10mb (depending on how fast you were setting up the inspector).

Wait for a couple minutes for the script to run (depends on your computer, but hsould be less than 10 minutes), and when you see "DONE" on the terminal, garbage collect again and redo a heap snapshot. This time the snapshot will be around 70mb.

Click on the second heap snapshot. On the top bar, switch from "Summary" to "Containment", and expand "(GC roots)" and then "(Global handles)".
Scroll the list (it's gonna be the string representation of functions used, we don't care about that), then click on the "Show 100 after" as many time as you want: all you'll see is thousands of strings of:

(function(parsers,rowData
/**/) {

this['summoner_id'] = rowData[0] == null ? null : parsers[0](rowData[0]);
this['last_season'] = rowData[1] == null ? null : parsers[1](rowData[1]);
})

each weighting 216 bytes -- not much, but it add fast.

I'm probably wrong somewhere, since the script is relatively simple, but it is the same behavior I'm seeing in my larger app.

My analysis

It look (but I'm not a PG expert) that the fact that this query is running on a PG custom type might be causing the problem:

CREATE TYPE rankType AS ENUM('UNRANKED','BRONZE','SILVER','GOLD','PLATINUM','DIAMOND','MASTER','CHALLENGER');

Running on a table with integers only doesn't cause any leak.

[charmander]

I think this is a debugger bug. Consider this script, whose memory usage under node --inspect peaks at a reasonable value until the inspector is opened, at which point it resumes growing and stops only when the inspector crashes.

'use strict';

for (let i = 0;; i++) {
  Function('return ' + ++i)();
}

It’s understandable how this could make memory leaks hard to track down, though, and I had been planning on comparing the performance of a well-optimized parser that doesn’t rely on generated JavaScript under modern Node anyway, so maybe an option to disable that could suit your needs as well.

[brianc]

@charmander Yeah I've been considering removing the generated JavaScript parser as well - I don't think it does as much good as it did when it was introduced. When it was added node was on v0.6 i believe and a much earlier version of v8. I just need a bit of time & I can run some benchmarks and figure out the impact.

[charmander]

@brianc The impact still seems to be there – at least a difference of 20% in some common cases I tried. Maybe more interesting, though: pg-native performed worse than the current pg in every case. Do you know of any situations it still handles better than the JavaScript driver? It would cut down on a lot of complexity if it were dropped, but maybe that’s too optimistic =)

[Neamar]

@charmander I get your point about the debugger, but I think it's worth noting two things:

• When I remove all queries, I don't get a memory leak in prod (without the inspector)
• The snippet I posted only leaks when I use a custom Postgres type, not on default integers. It could be that my analysis about the string being repeated dozens of times is wrong, but the fact that it leaks in one case and not the other is worrying.

I'm a little disappointed that you closed the issue, since it's still a real issue (at least for me), and the snippet posted above highlights a difference in behavior depending on the PG schema which shouldn't happen.

[charmander]

When I remove all queries, I don't get a memory leak in prod (without the inspector)

Yes, that’s because it’s the result parser creating functions dynamically.

The snippet I posted only leaks when I use a custom Postgres type, not on default integers.

I couldn’t reproduce this. Which query were you using to select only integers?

[Neamar]

Yes, that’s because it’s the result parser creating functions dynamically.

I thought you said it was a debugger bug. But if it's still happening in prod, without the inspector, doesn't it mean that something is off in the result parser?

I couldn’t reproduce this. Which query were you using to select only integers?

I was using this table instead of the schema linked above:

CREATE TABLE public.matches_participants
(
  id SERIAL,
  summoner_id integer,
  last_season integer NOT NULL
);

and the memory remained constant

[charmander]

But if it's still happening in prod, without the inspector, doesn't it mean that something is off in the result parser?

Using your minimal test case, are you able to produce a leak with --inspect off?

[Neamar]

Good point. I do see a lot of Garbage collection, and memory usage does go up slightly, but nothing comparable to running with --inspect.

Sorry for the noise, as you say having an unreliable --inspect is not going to help at all... I'll keep investigating on my side.

Sorry for the noise,

[charmander]

Well, that’s a relief =) If you can create a new script that leaks without --inspect, please post it. I’ll make a branch you can use that won’t cloud the inspector.