Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Tracking servers CPU saturation after upgrade from 2.1.0 to 2.2.0 #5019

Closed
anonymous-piwik-user opened this Issue · 27 comments

4 participants

Anonymous Piwik user Matthieu Aubry Stefan Giehl Thomas Steur
Anonymous Piwik user

Hi, I just upgraded Piwik from 2.1.0 to 2.2.0.
Everything was working fine (on a 3 millions pv per day website) before upgrade but after the update, the CPU stays at 100% (7% before upgrade).

I'm running it on Debian Wheezy, Nginx and php 5.5 and Piwik is totally innaccessible !

I tried to restart nginx, php-fpm, clear the Piwik tmp folder and reboot the server but nothing solved my problem, I still have the CPU at 100%.

Thanks for your help,

Anonymous Piwik user

I confirm that the problem comes from 2.2.0.
I switched back to 2.1.0 and everything works fine again : CPU never goes higher than 7%

Anonymous Piwik user

can confirm this.

also on a 3-4m pv / day website:

after update, the load was about 10-15, after the downgrade back to 2.1 the load also goes down to normal (~ 3).

rhel 6.5 with php 5.3 and lighttpd with fcgi

Matthieu Aubry
Owner

Thanks for your report!

We haven't seen such difference in our tests, but it is important we can help you make 2.2.0 work well for you.

where was the load located on the system? PHP or Mysql database?

If the database, please enable Mysql slow query log and write on pastebin the log (processed via this tool)

Anonymous Piwik user

From what I can see on Amazon graphs, only the php web server CPU was impacted .
On MySQL RDS instance, the CPU stays at its usual value (around 7%).

I attached the graphs (too bad I can't zoom on RDS graph).

Matthieu Aubry
Owner

That's very strange. do you use any custom plugin? is the CPU used by piwik.php (tracker) do you think? or was it peaking when the core:archive cron script was running?

Anonymous Piwik user

i only can provide a graph for the load (the upgrade was tue, shortly before 12:00), but i checked the server with top (mysql is running on the same machine) and i saw only the php-fcgi prozesses running high (20-30% each, normal is 1-2%), the mysql was about 40-50 which is normal mysql load with version 2.1 too).

there was no archive job running, the load was only from piwik.php requests.

Stefan Giehl
Collaborator

Could you provide a list with all plugins you have enabled in Piwik?

Anonymous Piwik user

All standard Core plugins except the following:

DBStats, DevicesDetection, ExampleCommand, ExampleSettingsPlugin, ExampleUI, ExampleVisualization, Feedback.

for geoip PECL is used.

Anonymous Piwik user

I confirm that the problem came from the tag and not from archiving (archiving is scheduled and was not running when I upgraded).

when I removed the tag from my webpages, the server CPU drops down to 0%.

The web interface was then running fine and displayed stats normally.

Matthieu Aubry
Owner

An idea, disable the DeviceDetection plugin and use 2.2.0 -> does it help maybe?

Stefan Giehl
Collaborator

As nx969 mentioned he has DevicesDetection disabled, it seems not to be a problem of that plugin.

Anonymous Piwik user

with version 2.2.1, load is slightly better, but still far away from 2.1 performance.

Thomas Steur
Owner

Ok here some questions to narrow this down (maybe):

  • One thing we fixed was "ScheduledTasks" were never triggered when "Browser archiving" was enabled. Do you have "Browser archiving" enabled? You can find out by going to "Settings" => "General settings". Under "Archiving settings" there is the point "Allow Piwik archiving to trigger when reports are viewed from the browser".
  • In your tracking requests, do you set the token_auth for authenticated tracking requests?
  • If possible, can you update to latest 2.2, and after there were some tracking requests post the result of "tmp/cache/tracker/general.php "?

Another change was the replacement of UserAgentParser with DeviceDetector. @Stefan can this cause any change in terms of CPU usage?

Thomas Steur
Owner

BTW: I compared 2.2.0 with 2.2.1 and didn't see anything that could have improved load slightly. Maybe there were just less requests or so.

Anonymous Piwik user

Hi,

  • browser archiving is disabled (its done by a cronjob once a day)

  • authenticated tracking requests are not used

  • i just moved my piwik 2.2 folder, so i still have the tmp folder. the contents of my general.php after a few thousend requests:

<?php
$content   = array (
  'isBrowserTriggerEnabled' => false,
  'lastTrackerCronRun' => false,
  'PrivacyManager.useAnonymizedIpForVisitEnrichment' => true,
  'PrivacyManager.ipAddressMaskLength' => 1,
  'PrivacyManager.doNotTrackEnabled' => true,
  'PrivacyManager.ipAnonymizerEnabled' => false,
  'currentLocationProviderId' => 'geoip_pecl',
  'CustomVariables.MaxNumCustomVariables' => 5,
);
$expires_on   = 1398170524;
$cache_complete   = true;

Thomas Steur
Owner

Thx a lot. Sounds like the DeviceDetector is most likely the reason for this? Need to check...

Anonymous Piwik user

as already mentioned, we have the DeviceDetector Plugin disabled.

Thomas Steur
Owner

It's used in the tracker no matter it is enabled or not

Anonymous Piwik user

Hopefully this is related.

I also get 100% cpu usage on a httpd process, which indicates that PHP is using all the cpu. the mysqld process goes up and down, so it doesn't seem to be the issue.

I've have allocated 1.5G for php and I get this error :

Allowed memory size of 1610612736 bytes exhausted (tried to allocate 32 bytes) in /var/www/piwik/core/DataTable.php on line 1156, referer: http://somehost/index.php?module=CoreHome&action=index&idSite=2&period=range&date=2013-02-02,2014-05-03&activated=

This line seems to be the issue :

   $serialized = unserialize($stringSerialized);

as of now I'm trying with 2G memory, but it halts because of executiontime which is set to 600 seconds.

As you can see the range is over an year. Could it be a memory leak?

We have around 40000 page vists a day, which afaik is not a lot.

I'm running 2.2.0.

Matthieu Aubry
Owner

@klausdk this is not related I don't think, unless this process was started by a piwik.php? URL?

see this faq: http://piwik.org/faq/troubleshooting/faq_95/

Thomas Steur
Owner

I debugged it with xhprof and noticed 92% is in my case spent in DeviceDetector / Spyc library. I think the problem could be a missing "setCache()" in https://github.com/piwik/piwik/blob/master/core/Tracker/Settings.php#L39

As a cache is optional and there is no cache set it will always parse the yml file causing a lot of CPU usage

Thomas Steur
Owner

In 075e6ba: refs #5019 using the cache reduced the time spent in tracking (piwik.php) from 2seconds to 200ms (on my local env)

Thomas Steur
Owner

In 1f0e262: refs #5019 should fix tests

Thomas Steur
Owner

In 7385bda: refs #5019 added missing parse

Frederic Hemberger fhemberger referenced this issue from a commit
Thomas Steur tsteur refs #5019 using the cache reduced the time spent in tracking (piwik.…
…php) from 2seconds to 200ms (on my local env)
075e6ba
Frederic Hemberger fhemberger referenced this issue from a commit
Thomas Steur tsteur refs #5019 should fix tests 1f0e262
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.