Memory usage with fs.createReadStream()

[julien-f]

require('trace')

var stream = require('fs').createReadStream('huge file here')
stream.resume()

It seems that the entire file is loaded in memory.

[AndreasMadsen]

There doesn't appear to be any memory leaks, but there is an odd behaviour in the RSS part which usually comes from new Buffer allocations. I have no idea what it is about, trace doesn't change the behaviour of node in any of those areas.

baseline

with trace

[julien-f]

Side question: which tool are you using to analyse memory usage in Node?

[AndreasMadsen]

I just wrote something simple: 7e5496a

maybe I will make a real module some day :D

[AndreasMadsen]

7b91e4a should fix this. If you try it out (please do) be aware that you will need at least node version 5.2.0 for now. However I have been promised that the necessary node features will land soon in 4.x.

[AndreasMadsen]

I don't get why this is an issue. I suspect it might be a v8 bug in the gc, 7b91e4a basically just moved the stack object into an id-map and does the memory management itself. For future reference I also tried using a WeekMap, so I din't have to use the gc destroy event, there was an improvement but not enough.

WeakMap using handle object

Map using uid and destructor event

[AndreasMadsen]

Thanks for creating the issue. I really appreciate the time you use on trace and stack-chain 👍

[julien-f]

Glad to help.

Thanks for your work :)