Memory leak #8

Open
pekim opened this Issue Sep 28, 2012 · 5 comments

Comments

Projects
None yet
5 participants
@pekim

pekim commented Sep 28, 2012

I have some code that uses jsonparse (via JSONStream) to parse a file that is about 170MB. The heap keeps growing, and eventually almost continual gc grinds the process almost to a halt.

I thought at first the leak was caused by dominictarr/JSONStream, but I think that I've narrowed the leak down to jsonparse.

This code causes a leak, that I don't think should happen.

var Parser = require('jsonparse');

var string = (new Array(10 * 1024 + 1)).join("x");

var parser = new Parser();
// parser.onValue = function(value) {
//   //console.log('received:', value);
// };

parser.write('[')
while (true) {
  parser.write('"' + string + '",')
}

It streams a never ending array of strings to jsonparse. It's silly, but it seemed to be a simple way to simulate parsing a large file and provoke the leak.

Running with the -trace_gc flag shows that the heap grows rapidly, gc is unable to reclaim much from the heap, and the heap is quickly exhausted.

I don't see why this code shouldn't be able to run indefinitely. Until it does, I'm probably not going to be able to process large files with jsonparse (which is a shame).

@lancecarlson

This comment has been minimized.

Show comment
Hide comment
@lancecarlson

lancecarlson May 7, 2013

I have the same issue, posted it on JSONStream, but I think jsonparse it is the culprit. Here was the message from the issue ( dominictarr/JSONStream#32 ):

request = require('request'),
Stream = require('stream').Stream,
Parser = require('jsonparse');

var dbName = 'test'
var p = new Parser();
p.onValue = function(value) {
  console.log(value)
}

var down = new Stream()
down.writable = true
down.write = function(data) {
  p.write(data)
  return true
}
down.end = function() {
  console.log('end')
}

var host = process.env.DB_HOST
var path = '_all_docs?include_docs=true'
var url = host + '/' + dbName + '/' + path
request(url).pipe(down)

Memory leak steadily increases from 50 to 260MB's, then midway through it jumps to 500-600MB's. I feel like the json parser queues up data for something then does something else with the queued data afterwards.

I have the same issue, posted it on JSONStream, but I think jsonparse it is the culprit. Here was the message from the issue ( dominictarr/JSONStream#32 ):

request = require('request'),
Stream = require('stream').Stream,
Parser = require('jsonparse');

var dbName = 'test'
var p = new Parser();
p.onValue = function(value) {
  console.log(value)
}

var down = new Stream()
down.writable = true
down.write = function(data) {
  p.write(data)
  return true
}
down.end = function() {
  console.log('end')
}

var host = process.env.DB_HOST
var path = '_all_docs?include_docs=true'
var url = host + '/' + dbName + '/' + path
request(url).pipe(down)

Memory leak steadily increases from 50 to 260MB's, then midway through it jumps to 500-600MB's. I feel like the json parser queues up data for something then does something else with the queued data afterwards.

@ralphtheninja

This comment has been minimized.

Show comment
Hide comment
@ralphtheninja

ralphtheninja Aug 7, 2014

@lancecarlson Is this still an issue with jsonparse?

@lancecarlson Is this still an issue with jsonparse?

@davidrapin davidrapin referenced this issue in dominictarr/JSONStream Feb 10, 2015

Open

memory leak #32

@hayes

This comment has been minimized.

Show comment
Hide comment
@hayes

hayes Jan 10, 2017

Contributor

Its a bit of a hack, but you can add:

if (parser.value) {
  parser.value = {}
}

to parser.onToken, and it will then store the accrued string on that empty value instead, which should get GCd at the end of the onToken method.

Not sure if that helps

Contributor

hayes commented Jan 10, 2017

Its a bit of a hack, but you can add:

if (parser.value) {
  parser.value = {}
}

to parser.onToken, and it will then store the accrued string on that empty value instead, which should get GCd at the end of the onToken method.

Not sure if that helps

@lojzatran

This comment has been minimized.

Show comment
Hide comment
@lojzatran

lojzatran May 31, 2017

Hi, is this error still valid? Does the hack from @hayes helped? Thanks.

Hi, is this error still valid? Does the hack from @hayes helped? Thanks.

@pekim

This comment has been minimized.

Show comment
Hide comment
@pekim

pekim May 31, 2017

I'm no longer have any code that uses this library. So I'm happy for this to issue to be closed.

pekim commented May 31, 2017

I'm no longer have any code that uses this library. So I'm happy for this to issue to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment