Skip to content

Memory leak #8

Open
pekim opened this Issue Sep 28, 2012 · 2 comments

3 participants

@pekim
pekim commented Sep 28, 2012

I have some code that uses jsonparse (via JSONStream) to parse a file that is about 170MB. The heap keeps growing, and eventually almost continual gc grinds the process almost to a halt.

I thought at first the leak was caused by dominictarr/JSONStream, but I think that I've narrowed the leak down to jsonparse.

This code causes a leak, that I don't think should happen.

var Parser = require('jsonparse');

var string = (new Array(10 * 1024 + 1)).join("x");

var parser = new Parser();
// parser.onValue = function(value) {
//   //console.log('received:', value);
// };

parser.write('[')
while (true) {
  parser.write('"' + string + '",')
}

It streams a never ending array of strings to jsonparse. It's silly, but it seemed to be a simple way to simulate parsing a large file and provoke the leak.

Running with the -trace_gc flag shows that the heap grows rapidly, gc is unable to reclaim much from the heap, and the heap is quickly exhausted.

I don't see why this code shouldn't be able to run indefinitely. Until it does, I'm probably not going to be able to process large files with jsonparse (which is a shame).

@lancecarlson

I have the same issue, posted it on JSONStream, but I think jsonparse it is the culprit. Here was the message from the issue ( dominictarr/JSONStream#32 ):

request = require('request'),
Stream = require('stream').Stream,
Parser = require('jsonparse');

var dbName = 'test'
var p = new Parser();
p.onValue = function(value) {
  console.log(value)
}

var down = new Stream()
down.writable = true
down.write = function(data) {
  p.write(data)
  return true
}
down.end = function() {
  console.log('end')
}

var host = process.env.DB_HOST
var path = '_all_docs?include_docs=true'
var url = host + '/' + dbName + '/' + path
request(url).pipe(down)

Memory leak steadily increases from 50 to 260MB's, then midway through it jumps to 500-600MB's. I feel like the json parser queues up data for something then does something else with the queued data afterwards.

@ralphtheninja

@lancecarlson Is this still an issue with jsonparse?

@davidrapin davidrapin referenced this issue in dominictarr/JSONStream Feb 10, 2015
Open

memory leak #32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.