memory issue

[funston]

Trying to debug. So not sure if this is a bug, or maybe ideas to fix. I have a basic app, and parser setup, loading a 300MB xml file.

Getting the following error I believe from inside the parser somewhere:

FATAL ERROR: JS Allocation failed - process out of memory

Just curious if you've tested anything large, before I dig into my code/parsing, although I'm not hitting even the first open tag before memory exhausts. I can post a working gist, but essentially this is what's going on :

var fs = require('fs');
var parser = sax.parser();

var fileStream = fs.createReadStream(process.argv[2],
{'bufferSize': 4 * 1024}
);
var xml = '';
fileStream.addListener("data",function(chunk){
xml += chunk;
});
fileStream.addListener("close", function(){
console.log("file read into xml");
parser.write(xml).close();
});

tnks

[funston]

btw, i also tried to just write "chunks" to the parser inside the file read stream, but getting this error:

/usr/local/lib/node/.npm/sax/0.1.2/package/lib/sax.js:257
while (parser.c = c = chunk.charAt(i++)) {

"write - Write bytes onto the stream. You don't have to do this all at once. You can keep writing as much as you want".

do you have a simple example of not writing an entire buffer but writing chunks to to the parser. perhaps that will help with the memory issue?

[funston]

okay, i plowed away and saw the prettyprint example, i noticed i wasn't passing the encoding of utf8 into the filestream, and with that the parser error above goes away (as does the memory issue if i write in chunks, not the entire buf).

sorted.