Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Performance: Fix memory leak during Archiving process #5794

Closed
mattab opened this Issue · 20 comments

3 participants

@mattab
Owner

The following php memory error is showing up when there is too much data.

``` Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes) in /home/www/piwik/demo/core/DataTable.php on line 807```

There are improvements to be made in the archiving process to make it more memory / computation efficient.

@anonymous-piwik-user

[this topic](watching) I have the same problem with month and year view, because my shared hosting allows only 32MB Php-Memory (was ticket:371 )

@mattab
Owner

fixed in 0.2.26

@anonymous-piwik-user

I am sorry, but I have still some problems. Generating grafics for Month doesn’t work. Generating grafics and calculation numbers for Year doesn’t work. And in visitors>overview I get:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 71 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 822

@robocoder
Collaborator

Is this still occuring in 0.2.28?

@anonymous-piwik-user

Glad you ask. I have no errors within Month and year views anymore. (But I also have 64MB php memory now)

However, Piwik is still not able to display the sparklines and the grafics “last visitors” and “returnig visitors”. I think it could be some kind of timeout.

@robocoder
Collaborator

Firefox’s Error Console reports:

Warning: 09 is not a legal ECMA-262 octal constant Source File: http://servername/index.php?module=CoreHome&action=index&idSite=1&period=month&date=yesterday Line: 40, Column: 132 Source Code: piwik.idSite = “1”; piwik.period = “month”; piwik.currentDateString = “2009-01-17”; piwik.minDateYear = 2008; piwik.minDateMonth = 09; piwik.minDateDay = 11;

@robocoder
Collaborator

In plugins/CoreHome/templates/js_global_variables.tpl, change these lines: ``` {if isset($minDateMonth)}piwik.minDateMonth = {$minDateMonth};{/if} {if isset($minDateDay)}piwik.minDateDay = {$minDateDay};{/if} ``` to: ``` {if isset($minDateMonth)}piwik.minDateMonth = parseInt({$minDateMonth}, 10);{/if} {if isset($minDateDay)}piwik.minDateDay = parseInt({$minDateDay}, 10);{/if} ```

@anonymous-piwik-user

Thanks for the help!!! I changed the line, but the error was still there. Then I changed it to: ``` {if isset($minDateMonth)}piwik.minDateMonth = parseInt(“{$minDateMonth}”, 10);{/if} {if isset($minDateDay)}piwik.minDateDay = parseInt(“{$minDateDay}”, 10);{/if} ``` Now the error is gone, but I have still no flash chart with the last visitors :-(

Anyway it looks like this is problem is not connected with the memory leak?

@robocoder
Collaborator

This is the error from the data URL from your Last visits graph and Visitors Evolution graph (day=yesterday, period=month):

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 72 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 836

Matt?

@mattab
Owner

vipsoft, what happens is that, for a given website, we store all data in memory (in ArchiveProcessing_Record_*), and at the end of processing all the processed statistics for a given website, we store all this data in DB.

What we could do is:

  • process each piece of data (ie. visits by keywords)
  • directly save it in DB
  • flush it from memory

Downside is that we would not have this data anymore, if other plugins want to access data that has just been processed by other plugins, to enrich/change it, for example. However this use case is not yet found in Piwik code, so we could assume it is a change we can make.

What would be changed in Piwik_ArchiveProcessing_Record_Manager:

  • get rid of Piwik_ArchiveProcessing_Record_Manager class,
  • after each creating each Piwik_ArchiveProcessing_Record ( new Piwik_ArchiveProcessing_Record_*) put $archiveProcessing→insertRecord()
@mattab
Owner

konus, vipsoft, committed octal-javascript bug in 898

@anonymous-piwik-user

Hello,

I’ve also got this problem, and I had to re-install Piwik. I do not think that is a good option, I do think I will get the same problem again when the archive is big enough.

Is there any work-around for this issue?

@mattab
Owner

(In 987) – refs #5794 test post commit

@mattab
Owner

(In 1035) – memory and speed optimizations of archiving, refs #5794 (hopefully fixing it but not sure yet) - added support for profiling memory & time in Piwik via the events mechanism - small other changes

@mattab
Owner

fixed! the culprit was PHP bug http://bugs.php.net/bug.php?id=33595 to get around I had to follow recommendations on http://paul-m-jones.com/?p=262

apparently bug has been fixed in php 5.3 so we shall see if php is dealing better with deleting old objects with cross references.

the memory leak problem is GONE! we were able to process stats of piwik.org for the last few months and year without any problem for 100k+ unique visitors

@anonymous-piwik-user

Thank you very much! I am happy to report, that piwik is now reporting within month and year timeframe.

@anonymous-piwik-user

Hi guys,
I have the same problem. My piwik has only a domain with 11.000 unique visitors / avg a day.
I dont know if this issue will fix in the next release or not but I need a workaround and I think wait for 5.3 php release is not a good idea.

any help please?

Cheers.

@robocoder
Collaborator

macada: there is a fix in Piwik that avoids this particular memory leak situation, and does not require php 5.3. If you are still encountering a problem with archiving, please open a new ticket and provide detailed information.

@anonymous-piwik-user

The new ticket is #685

@mattab
Owner

This ticket was reopened in #766

@mattab mattab added this to the RobotRock milestone
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.