To give the child a name, Pete simply stands for PErformance TEst.
Learning new languages like Elixir alone is already exciting but make it a bit more exciting by trying new frameworks and compare their speed against each other. Because I'm a PHP (mostly Drupal) guy and just discovered Elixir I want to take it as opportunity to learn functional programming paradigms but also see how Elixir's (Erlang) concurrency model speeds out our traditional languages such as PHP. Therefore I took the Elixir web framework Phoenix Framework and try to compare it in a similar structure and setup to Slim Framework (PHP) and Phalcon (PHP as C extension) and see how the performance differs.
I choose to do a very simplistic gallery app, with no model and database backend. Only router, controller, view and templates are used.
Image gallery:
- Headline
- 4 images (filenames as array/list with copyright info)
- 3 Templates/partials (base, gallery, image)
- CSS
- No model and database backend (only controller / view / templates)
I'm not familar with any of the frameworks I have chosen here, so bare with me if some structure is a complete mess :)
See subdirectories for framework specific information.
I made multiple test runs on Raspberry Pi 2 (4 cores) and some Digitalocean VMs with 2 and 12 cores to check how the frameworks deal with multiple cores and concurrency.
Update 2016-01-05:
After changing php.ini settings (see configuration settings below) PHP frameworks where
able to improve by 10 - 40%. Updated test results. Would be great if we could figure out why Phoenix
is not using all system ressources on 12-core and it get's back to an interesting first place race with Phalcon at the top.
Update 2016-01-06:
Ok, the history has to be rewritten.
Because I used a slow way to
include the images in the template file the performance was really bad. We tracked that issue down
and now we really see the performance of Phoenix (Elixir/Erlang). I expected Phoenix to be faster
than PHP frameworks but that much of lead, I did not expect. Compared to the last test Phoenix speedup was 4x to 14x!
But take a look below yourself, unbelievable fast. And the craziest thing is, no matter how much load you throw at it, it seems
to never return a bad request (3xx or 5xx) status code. (At least bandwidth seems to be maxed out so unable to throw more load at it :))
Using wrk as benchmarking tool with this or similar command:
$ wrk -t4 -c100 -d60s --timeout 2000 http://ip-or-host.tld/gallery
PHP 5: PHP 5.6.14-0-deb8u1 as PHP-FPM, OPcache enabled, Nginx 1.6.2
PHP 7: PHP 7.0.1 (self compiled) as PHP-FPM, OPcache enabled, Nginx 1.6.2
Phoenix: Erlang 18.2.1 (SMP + hipe enabled), Elixir 1.2.0, Phoenix 1.1.0
One nice thing about the Raspberry Pi is that the hardware is cheap and easy to get and test results can (hopefully) be reproduced and compared easier.
| Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
|---|---|---|---|
| Phoenix | 2247.92 | 47.92 | 36.19 |
| Phalcon | 586.41 | 170.22 | 18.68 |
| Slim (PHP 5.6) | 132.75 | 748.62 | 104.27 |
| Slim (PHP 7.0) | 27.71 | >3500.00 | 553.34 |
Detailed results and specs: results--raspberry-pi2.md
Basic digitalocean VM with SSD and 2 cores, 2gb ram
Pretty interesting that with fewer cores Phoenix is close but does not take the crown this time.
| Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
|---|---|---|---|
| Phalcon | 1716.16 | 58.28 | 16.74 |
| Slim (PHP 7.0) | 1394.63 | 71.26 | 13.56 |
| Phoenix | 1347.11 | 54.47 | 16.32 |
| Slim (PHP 5.6) | 693.11 | 143.40 | 20.78 |
Detailed results and specs: results--2-core-vm.md
But now time to get really into multi core business and see who can handle concurrency best:
App VM: Digitalocean VM with SSD and 12 cores, 32gb RAM
Benchmark VM: Digitalocean VM with SSD and 4 cores, 8 gb RAM
Connected over 1 Gb/s
We do the same number of connections but with more threads first, then let's try something insane and quadruple the number of connections and see what happens.
# ./wrk -t12 -c100 -d60s --timeout 1000
| Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
|---|---|---|---|
| Phoenix | 38399.63 | 2.56 | 2.28 |
| Phalcon | 9669.83 | 11.25 | 14.32 |
| Slim (PHP 7.0) | 7162.05 | 13.44 | 3.70 |
| Slim (PHP 5.6) | 3419.78 | 28.11 | 6.80 |
Ok, get a bit extreme and throw 12 threads with 400 connections onto the app server:
# ./wrk -t12 -c400 -d180s --timeout 1000
| Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
|---|---|---|---|
| Phoenix | ** | ** | ** |
| Phalcon | - | - | - |
| Slim (PHP 7.0) | - | - | - |
| Slim (PHP 5.6) | - | - | - |
** 2016-01-06: will need to run this test sometime again, old value was 3000 req/s but it handles now way more. See insane test below :)
Ok, so I really want to make Phoenix throw some errors now and do something insane. 48 treads 500 connections each.
# ./wrk -t48 -c500 -d60s --timeout 1000
| Framework | Throughput (req/s) | Latency avg (ms) | Stdev (ms) |
|---|---|---|---|
| Phoenix | 41162.79 | 43.55 | 111.75 |
| Phalcon | - | - | - |
| Slim (PHP 7.0) | - | - | - |
| Slim (PHP 5.6) | - | - | - |
This is just outstanding, take a look at the details below and look at the response times etc.
Detailed results, specs and comments: results--12-core-vm.md
Some configs for PHP and Nginx can be found in configs directory:
Raspberry PI + 2 core VM configs for Nginx + PHP-FPM pool:
configs/raspberry-pi2/nginx.conf
configs/raspberry-pi2/php-fpm_pool.d_www.conf
12 Core VM
configs/big-machine/nginx.conf
configs/big-machine/php-fpm_pool.d_www.conf
Common configs (Nginx vhosts + php.ini used an all testsystems):
configs/common/phalcon_nginx_vhost
configs/common/slim_nginx_vhost
configs/common/php.ini
Adding the following php.ini opcache-options improved above mentioned throughput for the PHP frameworks
about 10 - 20 % of improvement (+ ~40% for Slim on 12-core!). Thanks to @andresgutierrez for suggesting these tweaks.
opcache.revalidate_freq = 300
opcache.enable_file_override = On
(Keep in mind that this revalidates opcache code only every 5 minutes, so you definetely want that option on
production when you want to squeeze everything out and code does not change)
configs/common/phpinfo.html
I was inspired by these great guys but wanted to do it my way and see the results with a slightly more complex testapp.
http://blog.onfido.com/using-cpus-elixir-on-raspberry-pi2/
(title is a bit misleading because also PHP with FPM uses all CPUs pretty much, but more to that in a blog post)
Comparison from Chris McCord (creator of Phoenix) with Rails:
http://www.littlelines.com/blog/2014/07/08/elixir-vs-ruby-showdown-phoenix-vs-rails/
Followup to above mentioned comparison but extended to Go, Ruby, NodeJS frameworks:
https://github.com/mroth/phoenix-showdown
License: MIT