Load Testing System #8477
Labels
c: Performance
For when we could improve the performance / speed of Matomo.
c: Platform
For Matomo platform changes that aren't impacting any of our APIs but improve the core itself.
c: Tests & QA
For issues related to automated tests or making it easier to QA & test issues.
wontfix
If you can reproduce this issue, please reopen the issue or create a new one describing it.
It is not currently possible to test Piwik under high load. The current approach is to use xhprof while tracking one or many requests and while archiving. Since archiving happens once per hour and there is never more than once 'archiving process' running at once (other core:archive processes should continue the existing run), using xhprof alone is ok. Tracker performance, on the other hand is not so easily gauged.
Sure, using xhprof on a single request or multiple requests will provide some useful information and allow you to optimize, but it says nothing about how the tracker will behave under high levels of concurrency. Simulating high load requires many concurrent tracking requests and special monitoring.
Enter Locust
Locust (http://locust.io/) is a load testing framework that can simulate millions of concurrent users. It also looks very easy to use. Though it requires writing python code. (I'm ok w/ that, not sure if everyone else is.)
We can create a system that uses EC2 & locust to automate load testing the tracker. Then after the tracker is tested, we can automatically launch archiving w/ xhprof using the data that was just tracked.
This can provide us with performance regression testing under high load, but more importantly, we can test the effect plugins have on tracker & archiving performance automatically. By comparing the results of a test when a plugin is enabled to the test for core, we can see just how much overhead a plugin adds, and maybe even find ways to boost performance.
The System
Plugins should be able to create a custom locust profile describing test visits and test setup. For example, Goals would make sure goals are defined for the test websites being tracked, and define visits that triggered conversions. An example locust file w/ Piwik visit data might look like:
A new command
tests:load [--plugin=...]
should be created that will start up an EC2 instance w/ Piwik installed, then a EC2 instance w/ locust installed. Later, if it's needed to run tests against millions of concurrent visits, perhaps clusters can be created. [Note: Perhaps puppet files or something else can be used to specify different Piwik instance configurations. For example, normal w/ no redis cache, normal w/ redis cache, w/ queued tracking installed, etc.]The command will run the load test and save metric data to the filesystem, then use xhprof on a core:archive run and save the profile.
If possible, the command should get database performance metrics. Something that can be used to pinpoint problematic queries or tell if caches aren't being effective.
The text was updated successfully, but these errors were encountered: