We have done a lot of work on unit testing, integration testing and come up with an excellent Hudson setup. These have proved incredibly useful and time saving.
Now, it would be great to be able to automatically tell whether Piwik overall performance is impacted by a change or set of changes.
There are a few facts that we want to keep an eye on, and learn more about:
This is my proposal for continuous performance testing in Piwik. Please comment if you have any feedback or idea.
Performance test script
Manually run this script
A manual run of the script, with very high speed, concurrent connections, is equivalent to a stress test. It will highlight what is the limit of traffic piwik can handle.
Continuously run this script
The goal is to run this script as part of our continuous integration process.
qa.piwik.org:8080/hudson currently runs php-cgi 5.3.5 (latest). jetty + cgi is not optimized for speed; it's geared to flexibility (eg switching php versions on demand without restarting jetty; no build dependencies, eg mod_php5) and use with hudson.
I think we can setup the performance test as a remote job, and have it report its "build" status to hudson dashboard.
I suggest setting up lighttpd or nginx (on another port, or replace Apahe httpd at port 80), plus php-fcgi-fpm.
See also http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
See also: Plugin for Jenkins? #3280
XHProf is now integrated in our testing suite, see commits at: #3177
next logical step will be to have it run on Jenkins, and run a daily real life performance test (log import), with web access to the XHProf reports in the morning. :-) Stay tuned!
nice article Push our limits - reliability testing at Twitter
I'm closing this ticket for now, as it's not directly related to Piwik core. We're working on such setup internally as we need infrastructure to monitor performance. Good times!