Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking servers CPU saturation after upgrade from 2.1.0 to 2.2.0 #5019

Closed
anonymous-matomo-user opened this issue Apr 18, 2014 · 27 comments
Closed
Labels
Bug For errors / faults / flaws. c: Performance For when we could improve the performance / speed of Matomo.
Milestone

Comments

@anonymous-matomo-user
Copy link

Hi, I just upgraded Piwik from 2.1.0 to 2.2.0.
Everything was working fine (on a 3 millions pv per day website) before upgrade but after the update, the CPU stays at 100% (7% before upgrade).

I'm running it on Debian Wheezy, Nginx and php 5.5 and Piwik is totally innaccessible !

I tried to restart nginx, php-fpm, clear the Piwik tmp folder and reboot the server but nothing solved my problem, I still have the CPU at 100%.

Thanks for your help,

@anonymous-matomo-user
Copy link
Author

I confirm that the problem comes from 2.2.0.
I switched back to 2.1.0 and everything works fine again : CPU never goes higher than 7%

@anonymous-matomo-user
Copy link
Author

can confirm this.

also on a 3-4m pv / day website:

after update, the load was about 10-15, after the downgrade back to 2.1 the load also goes down to normal (~ 3).

rhel 6.5 with php 5.3 and lighttpd with fcgi

@mattab
Copy link
Member

mattab commented Apr 27, 2014

Thanks for your report!

We haven't seen such difference in our tests, but it is important we can help you make 2.2.0 work well for you.

where was the load located on the system? PHP or Mysql database?

If the database, please enable Mysql slow query log and write on pastebin the log (processed via this tool)

@anonymous-matomo-user
Copy link
Author

From what I can see on Amazon graphs, only the php web server CPU was impacted .
On MySQL RDS instance, the CPU stays at its usual value (around 7%).

I attached the graphs (too bad I can't zoom on RDS graph).

@anonymous-matomo-user
Copy link
Author

Attachment:
php.png

@anonymous-matomo-user
Copy link
Author

Attachment:
mysql.png

@mattab
Copy link
Member

mattab commented Apr 28, 2014

That's very strange. do you use any custom plugin? is the CPU used by piwik.php (tracker) do you think? or was it peaking when the core:archive cron script was running?

@anonymous-matomo-user
Copy link
Author

i only can provide a graph for the load (the upgrade was tue, shortly before 12:00), but i checked the server with top (mysql is running on the same machine) and i saw only the php-fcgi prozesses running high (20-30% each, normal is 1-2%), the mysql was about 40-50 which is normal mysql load with version 2.1 too).

there was no archive job running, the load was only from piwik.php requests.

@anonymous-matomo-user
Copy link
Author

Attachment:
linux_load.png

@sgiehl
Copy link
Member

sgiehl commented Apr 28, 2014

Could you provide a list with all plugins you have enabled in Piwik?

@anonymous-matomo-user
Copy link
Author

All standard Core plugins except the following:

DBStats, DevicesDetection, ExampleCommand, ExampleSettingsPlugin, ExampleUI, ExampleVisualization, Feedback.

for geoip PECL is used.

@anonymous-matomo-user
Copy link
Author

I confirm that the problem came from the tag and not from archiving (archiving is scheduled and was not running when I upgraded).

when I removed the tag from my webpages, the server CPU drops down to 0%.

The web interface was then running fine and displayed stats normally.

@mattab
Copy link
Member

mattab commented Apr 28, 2014

An idea, disable the DeviceDetection plugin and use 2.2.0 -> does it help maybe?

@sgiehl
Copy link
Member

sgiehl commented Apr 28, 2014

As nx969 mentioned he has DevicesDetection disabled, it seems not to be a problem of that plugin.

@anonymous-matomo-user
Copy link
Author

with version 2.2.1, load is slightly better, but still far away from 2.1 performance.

@tsteur
Copy link
Member

tsteur commented May 14, 2014

Ok here some questions to narrow this down (maybe):

  • One thing we fixed was "ScheduledTasks" were never triggered when "Browser archiving" was enabled. Do you have "Browser archiving" enabled? You can find out by going to "Settings" => "General settings". Under "Archiving settings" there is the point "Allow Piwik archiving to trigger when reports are viewed from the browser".
  • In your tracking requests, do you set the token_auth for authenticated tracking requests?
  • If possible, can you update to latest 2.2, and after there were some tracking requests post the result of "tmp/cache/tracker/general.php "?

Another change was the replacement of UserAgentParser with DeviceDetector. @Stefan can this cause any change in terms of CPU usage?

@tsteur
Copy link
Member

tsteur commented May 14, 2014

BTW: I compared 2.2.0 with 2.2.1 and didn't see anything that could have improved load slightly. Maybe there were just less requests or so.

@anonymous-matomo-user
Copy link
Author

Hi,

  • browser archiving is disabled (its done by a cronjob once a day)
  • authenticated tracking requests are not used
  • i just moved my piwik 2.2 folder, so i still have the tmp folder. the contents of my general.php after a few thousend requests:
<?php
$content   = array (
  'isBrowserTriggerEnabled' => false,
  'lastTrackerCronRun' => false,
  'PrivacyManager.useAnonymizedIpForVisitEnrichment' => true,
  'PrivacyManager.ipAddressMaskLength' => 1,
  'PrivacyManager.doNotTrackEnabled' => true,
  'PrivacyManager.ipAnonymizerEnabled' => false,
  'currentLocationProviderId' => 'geoip_pecl',
  'CustomVariables.MaxNumCustomVariables' => 5,
);
$expires_on   = 1398170524;
$cache_complete   = true;

@tsteur
Copy link
Member

tsteur commented May 15, 2014

Thx a lot. Sounds like the DeviceDetector is most likely the reason for this? Need to check...

@anonymous-matomo-user
Copy link
Author

as already mentioned, we have the DeviceDetector Plugin disabled.

@tsteur
Copy link
Member

tsteur commented May 15, 2014

It's used in the tracker no matter it is enabled or not

@anonymous-matomo-user
Copy link
Author

Hopefully this is related.

I also get 100% cpu usage on a httpd process, which indicates that PHP is using all the cpu. the mysqld process goes up and down, so it doesn't seem to be the issue.

I've have allocated 1.5G for php and I get this error :

Allowed memory size of 1610612736 bytes exhausted (tried to allocate 32 bytes) in /var/www/piwik/core/DataTable.php on line 1156, referer: http://somehost/index.php?module=CoreHome&action=index&idSite=2&period=range&date=2013-02-02,2014-05-03&activated=

This line seems to be the issue :

   $serialized = unserialize($stringSerialized);

as of now I'm trying with 2G memory, but it halts because of executiontime which is set to 600 seconds.

As you can see the range is over an year. Could it be a memory leak?

We have around 40000 page vists a day, which afaik is not a lot.

I'm running 2.2.0.

@mattab
Copy link
Member

mattab commented May 24, 2014

@klausdk this is not related I don't think, unless this process was started by a piwik.php? URL?

see this faq: http://piwik.org/faq/troubleshooting/faq_95/

@tsteur
Copy link
Member

tsteur commented May 25, 2014

I debugged it with xhprof and noticed 92% is in my case spent in DeviceDetector / Spyc library. I think the problem could be a missing "setCache()" in https://github.com/piwik/piwik/blob/master/core/Tracker/Settings.php#L39

As a cache is optional and there is no cache set it will always parse the yml file causing a lot of CPU usage

@tsteur
Copy link
Member

tsteur commented May 25, 2014

In 075e6ba: refs #5019 using the cache reduced the time spent in tracking (piwik.php) from 2seconds to 200ms (on my local env)

@tsteur
Copy link
Member

tsteur commented May 26, 2014

In 1f0e262: refs #5019 should fix tests

@tsteur
Copy link
Member

tsteur commented May 26, 2014

In 7385bda: refs #5019 added missing parse

@anonymous-matomo-user anonymous-matomo-user added this to the 2.3.0 - Piwik 2.3.0 milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws. c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

4 participants