Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate if there are any memory leak in the Archiving process (yearly archives run out of memory) #6846

Closed
mattab opened this issue Dec 12, 2014 · 2 comments
Closed

Investigate if there are any memory leak in the Archiving process (yearly archives run out of memory) #6846

mattab opened this issue Dec 12, 2014 · 2 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone
2.12.0

Comments

@mattab
Copy link
Member

mattab commented Dec 12, 2014

The goal of this issue is to investigate the memory usage of Archiving to check whether there are memory leaks and if it can be improved. In particular we will focus on one particular Piwik PRO client that required more than 4Gb of memory to process the 2014 Yearly report.

Error was: PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 8208 bytes) in piwik-2.8.3/core/DataTable.php on line 1156

Maybe a XHProf run could help us find some memory issues, or maybe we will conclude that the current implementation of sub-tables is simply not fitting for this (next step would be: #6763)...
@mattab mattab added the Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. label Dec 12, 2014
@mattab mattab added this to the Short term milestone Dec 12, 2014
@mattab mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Dec 12, 2014
@mnapoli mnapoli mentioned this issue Jan 15, 2015
Closed
mnapoli referenced this issue Jan 15, 2015
@mnapoli
Archiver optimization: gained 800ms in my local test (1.5%)
c133683
mnapoli referenced this issue Jan 15, 2015
@mnapoli
Optimize how archive data is fetched from database
0c90701 
Now we try to make only 1 query per archive table (before there could be multiple queries per table).

In weekly archiving with very few data the number of SQL queries went from 2500 to 1000, saved 6% of execution time, 5% of memory usage.
@mattab mattab mentioned this issue Feb 10, 2015
Closed
@mattab mattab modified the milestones: Piwik 2.13.0, Short term Mar 19, 2015
@mattab
Copy link
Member Author

mattab commented Mar 19, 2015

we've made a lot of progress in performance, memory, CPU usage of Piwik in 2.12.0 - refs #6759

The archiver was profiled and tweaked several times. We consider this issue done for now (first big step). if anyone still experiences 1G memory exhausted issue for yearly archiving or other periods, then please create a new issue and we will follow up!

@mattab mattab closed this as completed Mar 19, 2015
@mattab mattab modified the milestones: Piwik 2.12.0, Piwik 2.13.0 Mar 19, 2015
@tsteur
Copy link
Member

tsteur commented Mar 19, 2015

FYI: I did some tests on the big instance and it takes about 4GB of memory to archive the year when having lots of data with a total execution time of 50 minutes. 95% of the time is spent in archiving action urls. 3.6GB of memory were used at peak by Piwik\DataTable::addRowsFromSerializedArray. 4GB still sounds ok for me. I don't think there is an actual memory leak! It is just what it needs to build thousands of datatables etc.

@mattab mattab added the c: Performance For when we could improve the performance / speed of Matomo. label Mar 19, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: Performance For when we could improve the performance / speed of Matomo. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Milestone
2.12.0
Development

No branches or pull requests
2 participants
@tsteur @mattab