Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High memory usage (11x). #46

Closed
ChALkeR opened this issue Mar 14, 2015 · 2 comments
Closed

High memory usage (11x). #46

ChALkeR opened this issue Mar 14, 2015 · 2 comments

Comments

@ChALkeR
Copy link
Contributor

ChALkeR commented Mar 14, 2015

This code:

var x = '601141b710b1f1b8a916f51dfbb3006fcff6b9d22a278ae1b39577fc769f19117bb4a1b87028c1d97b1d1cadcb0833a0000000000000000000007e2058e704be20dcbffaa883cca0ee64f9e4646985321a110f1e3e9ab96bef86768e958fa793f569d03fe8e18cfe74ea9e2772942aa734c3e761efe7b919240c95c1d26b563b591be9074af935899a61f0158b674de70eea061ce0e7cbf00fc799180ec22d5d6fb769cda33332fe646ba9002b2f9832555f9a5e904e4bd2f43cfoofoofoofoofoofoofoo15153e156fc9534bafd6b5d66fc6275d 601141b710b1f1b8a916f51dfbb3006fcff6b9d22a278ae1b39577fc769f19117bb4a1b87028c1d97b1d1cadcb0833a0000000000000000000007e2058e704be20dcbffaa883cca0ee64f9e4646985321a110f1e3e9ab96bef86768e958fa793f569d03fe8e18cfe74ea9e2772942aa734c3e761efe7b919240c95c1d26b563b591be9074af935899a61f0158b674de70eea061ce0e7cbf00fc799180ec22d5d6fb769cda33332fe646ba9002b2f9832555f9a5e904e4bd2f43cfoofoofoofoofoofoofoo15153e156fc9534bafd6b5d66fc6275d';

var m = [];
for (var i = 0; i < 50000; i++) {
    var y = i + x;
    y = LZString.compress(y); // Without lz-string: 17M. With lz-string: 900M.
    m.push(y);
}

Takes about 900M in v8 and about 700M (inaccurate) in Firefox.
That memory is released later by the gc, but the peak memory usage is very high, and if you increase the number of rounds even more, gc will kick in at the middle and will slow things down.

The x string in the code above isn't special, that happens generally.

You could test the memory usage with console.log(process.memoryUsage()); at the end of the loop in iojs/node.

@ChALkeR
Copy link
Contributor Author

ChALkeR commented Mar 14, 2015

Almost all of this comes from the way you are building the result string, context_data_string += … lines.

Replacing context_data_string with an Array and doing push/join('') decreases 900M to 80M.
You could also preallocate the Array of some fixed length (depending on the input length) and manually use [i++] on that. I haven't measured which of those the variants (string, Array.prototype.push, preallocated Array) is faster.

@ChALkeR
Copy link
Contributor Author

ChALkeR commented Mar 14, 2015

Ok, here are the test results with iojs:

50000 iterations, String concat (current): 31 seconds, 905 MiB.
50000 iterations, Array push/join: 29 seconds, 81 MiB.
50000 iterations, Array prealloc/[i++]/join: 32 seconds, 81 MiB.
50000 iterations, Array push/join + ES6: 15 seconds, 79 MiB.

100000 iterations, String concat (current): 1212 seconds, 1760 MiB, the gc kicked in,
100000 iterations, Array push/join: 57 seconds, 114 MiB.
100000 iterations, Array prealloc/[i++]/join: 69 seconds, 115 MiB.
50000 iterations, Array push/join + ES6: 30 seconds, 107 MiB.

200000 iterations, String concat (current): I terminated it after 100 minutes, 2450 MiB, the gc kicked in,
200000 iterations, Array push/join: 120 seconds, 172 MiB.
200000 iterations, Array prealloc/[i++]/join: 139 seconds, 172 MiB.
50000 iterations, Array push/join + ES6: 61 seconds, 164 MiB.

There is enough free memory and swap is disabled, so it didn't affect the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants