Tracking servers CPU saturation after upgrade from 2.1.0 to 2.2.0

[anonymous-matomo-user]

Hi, I just upgraded Piwik from 2.1.0 to 2.2.0.
Everything was working fine (on a 3 millions pv per day website) before upgrade but after the update, the CPU stays at 100% (7% before upgrade).

I'm running it on Debian Wheezy, Nginx and php 5.5 and Piwik is totally innaccessible !

I tried to restart nginx, php-fpm, clear the Piwik tmp folder and reboot the server but nothing solved my problem, I still have the CPU at 100%.

Thanks for your help,

[anonymous-matomo-user]

I confirm that the problem comes from 2.2.0.
I switched back to 2.1.0 and everything works fine again : CPU never goes higher than 7%

[anonymous-matomo-user]

can confirm this.

also on a 3-4m pv / day website:

after update, the load was about 10-15, after the downgrade back to 2.1 the load also goes down to normal (~ 3).

rhel 6.5 with php 5.3 and lighttpd with fcgi

[mattab]

Thanks for your report!

We haven't seen such difference in our tests, but it is important we can help you make 2.2.0 work well for you.

where was the load located on the system? PHP or Mysql database?

If the database, please enable Mysql slow query log and write on pastebin the log (processed via this tool)

[anonymous-matomo-user]

From what I can see on Amazon graphs, only the php web server CPU was impacted .
On MySQL RDS instance, the CPU stays at its usual value (around 7%).

I attached the graphs (too bad I can't zoom on RDS graph).

[anonymous-matomo-user]

Attachment:
php.png

[anonymous-matomo-user]

Attachment:
mysql.png

[mattab]

That's very strange. do you use any custom plugin? is the CPU used by piwik.php (tracker) do you think? or was it peaking when the core:archive cron script was running?

[anonymous-matomo-user]

i only can provide a graph for the load (the upgrade was tue, shortly before 12:00), but i checked the server with top (mysql is running on the same machine) and i saw only the php-fcgi prozesses running high (20-30% each, normal is 1-2%), the mysql was about 40-50 which is normal mysql load with version 2.1 too).

there was no archive job running, the load was only from piwik.php requests.

[anonymous-matomo-user]

Attachment:
linux_load.png

[sgiehl]

Could you provide a list with all plugins you have enabled in Piwik?

[anonymous-matomo-user]

All standard Core plugins except the following:

DBStats, DevicesDetection, ExampleCommand, ExampleSettingsPlugin, ExampleUI, ExampleVisualization, Feedback.

for geoip PECL is used.

[anonymous-matomo-user]

I confirm that the problem came from the tag and not from archiving (archiving is scheduled and was not running when I upgraded).

when I removed the tag from my webpages, the server CPU drops down to 0%.

The web interface was then running fine and displayed stats normally.

[mattab]

An idea, disable the DeviceDetection plugin and use 2.2.0 -> does it help maybe?

[sgiehl]

As nx969 mentioned he has DevicesDetection disabled, it seems not to be a problem of that plugin.

[anonymous-matomo-user]

with version 2.2.1, load is slightly better, but still far away from 2.1 performance.

[tsteur]

Ok here some questions to narrow this down (maybe):

• One thing we fixed was "ScheduledTasks" were never triggered when "Browser archiving" was enabled. Do you have "Browser archiving" enabled? You can find out by going to "Settings" => "General settings". Under "Archiving settings" there is the point "Allow Piwik archiving to trigger when reports are viewed from the browser".
• In your tracking requests, do you set the token_auth for authenticated tracking requests?
• If possible, can you update to latest 2.2, and after there were some tracking requests post the result of "tmp/cache/tracker/general.php "?

Another change was the replacement of UserAgentParser with DeviceDetector. @Stefan can this cause any change in terms of CPU usage?

[tsteur]

BTW: I compared 2.2.0 with 2.2.1 and didn't see anything that could have improved load slightly. Maybe there were just less requests or so.

[anonymous-matomo-user]

Hi,

• browser archiving is disabled (its done by a cronjob once a day)
• authenticated tracking requests are not used
• i just moved my piwik 2.2 folder, so i still have the tmp folder. the contents of my general.php after a few thousend requests:

<?php
$content   = array (
  'isBrowserTriggerEnabled' => false,
  'lastTrackerCronRun' => false,
  'PrivacyManager.useAnonymizedIpForVisitEnrichment' => true,
  'PrivacyManager.ipAddressMaskLength' => 1,
  'PrivacyManager.doNotTrackEnabled' => true,
  'PrivacyManager.ipAnonymizerEnabled' => false,
  'currentLocationProviderId' => 'geoip_pecl',
  'CustomVariables.MaxNumCustomVariables' => 5,
);
$expires_on   = 1398170524;
$cache_complete   = true;

[tsteur]

Thx a lot. Sounds like the DeviceDetector is most likely the reason for this? Need to check...

[anonymous-matomo-user]

as already mentioned, we have the DeviceDetector Plugin disabled.

[tsteur]

It's used in the tracker no matter it is enabled or not

[anonymous-matomo-user]

Hopefully this is related.

I also get 100% cpu usage on a httpd process, which indicates that PHP is using all the cpu. the mysqld process goes up and down, so it doesn't seem to be the issue.

I've have allocated 1.5G for php and I get this error :

Allowed memory size of 1610612736 bytes exhausted (tried to allocate 32 bytes) in /var/www/piwik/core/DataTable.php on line 1156, referer: http://somehost/index.php?module=CoreHome&action=index&idSite=2&period=range&date=2013-02-02,2014-05-03&activated=

This line seems to be the issue :

   $serialized = unserialize($stringSerialized);

as of now I'm trying with 2G memory, but it halts because of executiontime which is set to 600 seconds.

As you can see the range is over an year. Could it be a memory leak?

We have around 40000 page vists a day, which afaik is not a lot.

I'm running 2.2.0.

[mattab]

@klausdk this is not related I don't think, unless this process was started by a piwik.php? URL?

see this faq: http://piwik.org/faq/troubleshooting/faq_95/

[tsteur]

I debugged it with xhprof and noticed 92% is in my case spent in DeviceDetector / Spyc library. I think the problem could be a missing "setCache()" in https://github.com/piwik/piwik/blob/master/core/Tracker/Settings.php#L39

As a cache is optional and there is no cache set it will always parse the yml file causing a lot of CPU usage

[tsteur]

In 075e6ba: refs #5019 using the cache reduced the time spent in tracking (piwik.php) from 2seconds to 200ms (on my local env)

[tsteur]

In 1f0e262: refs #5019 should fix tests

[tsteur]

In 7385bda: refs #5019 added missing parse