-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage and Garbage Collector #7
Comments
I made different loads during the day and the memory footprint of the process was less than 2GB for the 4 tables with 100k/per batch and 200k/per batch . |
Here you are some stats for a run of 12 hours: influxdb-zabbix process
postgresql backend
|
what's your configuration ? |
Basically the server have the same specs
Latest version of influxdb |
which GO version ? try to update with latest version. |
I am using Go 1.7.4 and the index has not been created. Regarding the config : But now regarding the indexes, it's just a good to have and should . not have any impact on the memory consumption on the client side. The thing is I have a 500 GB zabbix DB (most of the data is for the history table and I don't want to add more indexing weight. Having a limit on the row to return is a workaround but not the real solution (on a DBA part...) for me. For a one year of historical the overload on the DB is just to much |
@menardorama a moving window based on number of days is now implemented |
Hi Thanks for your feedback, it's much better now on the DB side. But there is still something wrong, I think I pointed out but I am not enough good in Go to propose a patch. I'll try to explain my observation. From what I understand, you app works in two steps
My concern is that I have 57 millions of rows per week, and it does not fit in memory. Another approach could be to process a batch of rows (at the fetch level) and insert them in influxdb. This would be more scalable instead of waiting for the full fetch. Another idea would be to spool the result to a tempfile at the fetch level and pass the filename to the influxdb processor. Once again I'm sorry I am not good enough in Go to do it. What do you think ? |
You can now split the dataset to multiple dataset with the conf paramaters daysperbatch. startdate="2017-01-01T00:00:00" => process will start for data with timestamp between ]2017-01-01 and 2017-01-16[ and it will continue with the increment of 15 days, [2017-01-16 to 2017-01-31[, etc Have you tested the last version ? |
Yes my comment was regarding the latest version. And I can't put more RAM on my server (32GB already) Setting daysperbatch=1 help a bit but it's already 57 Millions of rows and it consume all memory |
57 Millions for 1 day ? you just said it was for 1 week. Anyway, that's huge... |
Sorry you're right 51 millions is per week...
But having à batch per day don't fit either in memory....
Thanks anyway.
Le 9 juin 2017 4:36 PM, "zensqlmonitor" <notifications@github.com> a écrit :
57 Millions for 1 day ? you just said it was for 1 week. Anyway, that's
huge...
Sorry but won't spool the result in disk.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#7 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AF0vYLQF0yTpF3sy4DVB4w3rzRT-14Zcks5sCVh0gaJpZM4NtKzr>
.
|
Let's do it more granular. |
Hi
after fixing the long historical data issue (even if I would'nt have done like that) I am facing another issue.
The process consume all memory on the server while waiting for the result of the db.
It's like if the the GC is not working at all.
The result for me is that I can't transfert the data as it consume the whole 32 GB of RAM and OOM Killer kill the process.
The text was updated successfully, but these errors were encountered: