High RAM usage

[markon1]

Hi, I made this simple node.js app:
http://pastebin.com/Ua9Q6zqj

As you can see, I load database.json file, do some alasql query on it and return that data as response.
database.json is ~50MB in size.
My problem is that this app uses 400-600MB of RAM!

I made this version too, where I open database.json file using .require, but memory consumption is same!

http://pastebin.com/EEhcPmyG

What am I doing wrong, I suppose it is some issue regarding the size of my JSON file? It is array of 55000 records, each one having 20 fields, if that info helps you?

[nickdeis]

nodejs/express/V8 engine use a lot of memory.
The garbage collector for V8 is also notoriously slow.
In my experience, a simple application eats like 350Mb-500Mb.
What I would do in this situation is try read in a JSON that is a 10% sample of the current JSON, and then see what the mem usage is.

[mathiasrw]

@markon1 Did you try what nickdeis said?

[mathiasrw]

I will close this. Please reopen if anything new comes up.

[guptavijay208]

Any update on above? How we can optimize the RAM uses for ALASQL, I have a simple App made in Codeiniter PHP, however all the code are written in Javascript, Using PHP just to login.
I have a json file approx 16MB (size will grow day by day, at-least 0.5 MB daily), and I need to generate 20 C3.js charts via fetching data from json and passing to C3.js javascript function.

Earlier I was querying my json file like below.

alasql(['select customer_c, sum(spocto_id) as spocto_id, sum(POS) as pos FROM JSON("../bfl.json") where row_num=1 group by customer_c']).then(function(res){
}).catch(function(err){
console.log('Chart 1 & 2, Total Records & POS - Does the file exist? There was an error:', err);
});

I was doing total 20 query for my 20 charts and after doing some calculation was passing it to C3JS javascript function to draw the chart.

I have some filters to apply after I generate all the charts, below is screenshot of my full page, kindly ignore the black patches I did that to hide the names.

When I open google chrome it uses approx 200MB of RAM however when I load my page it goes upto 500-600 MB of RAM uses, and when I apply filters again and again to see specific data, RAM uses keeps increasing dramatically and the application crashes, see below screenshots, RAM uses goes upto 3GB.

RAM USES

BROWSE CRASHED

Because of this reason my App does not even open on low configuration 2GB RAM PCs :(

Later on I thought might be I am loading file again and again like below
FROM JSON("../bfl.json")
in ALASQL that's why it's taking a lot of RAM.

Then I changed the way and included json file in my page. and assigned all the data to a variable "json_data"

<script src="../bfl.json" type="text/javascript"></script>

Then started querying like below
var res = alasql('select customer_c FROM ? group by customer_c order by customer_c',[json_data]);

I noticed that my speed increased a bit while applying filters however the RAM uses is same, it keeps increasing as I use my filters and fetch data again and again.

I checked same with enabling and disabling the cache like below however the RAM uses is same.
alasql.options.cache = false; // Use cache

Is there any way to free up the memory or any other way to use ALASQL effectively, this is very critical for me, and I haven't seen any optimization technique after googling.

Can Anyone help or suggest something please???

[mathiasrw]

Put it into a webworker - goes more smooth.

[thekennysong]

@guptavijay208 were you able to figure out how to use the web worker with es6 modules?