Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
769 lines (598 sloc) 21.8 KB
layout status published title author wordpress_id wordpress_url date date_gmt categories tags comments
post
publish
true
Benchmark Helloworld in different frameworks #myelixirstatus [Update #3, now with Go]
display_name login email url
750
2015-10-08 22:23:37 +0200
2015-10-08 20:23:37 +0200
Entwicklung
Elixir
Phoenix
true

When I started with Elixir and Phoenix I wanted to see with my own eyes the speed of functional approach and Erlang in comparing with other well known languages. I just want to compare the simplest possible "Hello world" in Javascript, PHP and Elixir.

Every implementation is benchmarked with the Apache HTTP server benchmarking tool ab. I run 1000 requests on 10 and 100 concurrent connections with ab -n 1000 -c 10 http://127.0.0.1:../. The return of "Hello world" ist quite enough but it differs a little bit from test to test. So no HTML, no pre/postprocessing or something. Well, its a bit "comparing apples with peaches".

[see also https://www.techempower.com/]

My test machine is my faithful MacbookPro (Mid 2010) with i5 2,53Ghz, 8 GB RAM, El Capitan.

Beside the plain implementation I also tried a standard framework each. So here we go:

  • Node.JS v4.1.1: Module http and Express v4.13.3
  • PHP v5.5.29 + v7.0.0RC4: built-in Http-Server and Silex v1.3
  • Elixir v1.1.1: Plug (with Cowboy) and Phoenix v0.15
  • Go v1.5.2: net/http

(Update Added version numbers)

Please find the source here: github.com/ronnyhartenstein/benchmarking-helloworld-http

HowTo Apache benchmark

Update: Some words about benchmarking. Normally you should benchmark some workflows - the fast ones (landing page) and the slow ones (e.g. checkout or login). And then the longest request of these. Because if there are a lot of parallel request are going fast and some of them are really slow, these some users just traps into it and getting a damn bad UX. This video of a talk How NOT to Measure Latency explains it in detail.

(Thanks to @MorrisJbk and @Hisako1337 for their explanations)

This posts on serverfault.com are also revealing:

PHP plain

Version 5.5.29 stable

The code:

<?="Hello World"?>

Start the PHP internal server with

php -S localhost:3000

The ab -n 1000 -c 10 http://localhost:3000/ result:

Time per request:       5.418 [ms] (mean)
Time per request:       0.542 [ms] (mean, across all concurrent requests)
Transfer rate:          219.89 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       4
Processing:     1    5   2.8      4      16
Waiting:        0    5   2.6      4      16
Total:          1    5   2.8      4      18

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      6
  80%      7
  90%      9
  95%     12
  98%     14
  99%     15
 100%     18 (longest request)

The ab -n 1000 -c 100 http://localhost:3000/ result:

Time per request:       60.948 [ms] (mean)
Time per request:       0.609 [ms] (mean, across all concurrent requests)
Transfer rate:          195.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4   5.6      1      38
Processing:     3   54  20.5     58      92
Waiting:        3   52  20.9     55      91
Total:         16   57  17.1     59      92

Percentage of the requests served within a certain time (ms)
  50%     59
  66%     69
  75%     72
  80%     73
  90%     78
  95%     81
  98%     85
  99%     87
 100%     92 (longest request)

Update: I guess this dev server is probably single processing, so the parallel request will be processed round robin. So the times increases nearly with the concurrency factor.

Version 7.0.0 RC4 bleeding edge

The ab -n 1000 -c 10 http://localhost:3000/ result:

Time per request:       5.418 [ms] (mean)
Time per request:       0.542 [ms] (mean, across all concurrent requests)
Transfer rate:          219.89 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       4
Processing:     1    5   2.8      4      16
Waiting:        0    5   2.6      4      16
Total:          1    5   2.8      4      18

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      6
  80%      7
  90%      9
  95%     12
  98%     14
  99%     15
 100%     18 (longest request)

The ab -n 1000 -c 100 http://localhost:3000/ result:

Time per request:       39.738 [ms] (mean)
Time per request:       0.397 [ms] (mean, across all concurrent requests)
Transfer rate:          926.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.9      0       9
Processing:     2   37   9.5     35      59
Waiting:        2   36   9.5     35      59
Total:         12   37   8.5     36      59

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     41
  75%     45
  80%     47
  90%     49
  95%     51
  98%     52
  99%     58
 100%     59 (longest request)

So PHP 7 is a bit faster than 5.5. Not bad at all. Lets see the other ones.

NodeJS plain with http

The code:

var http = require('http');
var server = http.createServer(function (request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.end("Hello World\n");
});
server.listen(8000);

The ab -n 1000 -c 10 http://localhost:8000/ result:

Time per request:       6.429 [ms] (mean)
Time per request:       0.643 [ms] (mean, across all concurrent requests)
Transfer rate:          171.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:     1    6   4.8      5      44
Waiting:        1    6   4.7      4      44
Total:          2    6   5.0      5      47

Percentage of the requests served within a certain time (ms)
  50%      5
  66%      6
  75%      7
  80%      7
  90%     10
  95%     13
  98%     21
  99%     36
 100%     47 (longest request)

The ab -n 1000 -c 100 http://localhost:8000/ result:

Time per request:       46.297 [ms] (mean)
Time per request:       0.463 [ms] (mean, across all concurrent requests)
Transfer rate:          238.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   3.3      0      16
Processing:     9   42  10.2     43      60
Waiting:        8   42  10.2     42      60
Total:         22   44   8.9     46      60

Percentage of the requests served within a certain time (ms)
  50%     46
  66%     50
  75%     51
  80%     52
  90%     53
  95%     59
  98%     60
  99%     60
 100%     60 (longest request)

Node.JS is at the same speed than PHP, but did the Webserver for himself. Don't mention it.

Elixir nearly plain with Plug

(based on github.com/elixir-lang/plug)

After bootstrapping Elixir with mix new elixir_plug.. The code in mix.exs

defp deps do
  [{:cowboy, "~> 1.0.0"},
 {:plug, "~> 1.0"}]
end

The code in lib/elixir_plug.ex:

defmodule ElixirPlug do
  import Plug.Conn

  def init(options) do
    # initialize options

    options
  end

  def call(conn, _opts) do
    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "Hello from Plug")
  end
end

Start with iex -S mix and then

{:ok, _} = Plug.Adapters.Cowboy.http ElixirPlug, []

The ab -n 1000 -c 10 http://127.0.0.1:4000/ result: (Elixir and Phoenix don't like localhost)

Time per request:       3.509 [ms] (mean)
Time per request:       0.351 [ms] (mean, across all concurrent requests)
Transfer rate:          556.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.6      0       6
Processing:     0    3   3.7      2      35
Waiting:        0    2   3.6      2      34
Total:          1    3   3.8      2      36
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      4
  80%      4
  90%      6
  95%      7
  98%     11
  99%     34
 100%     36 (longest request)

The ab -n 1000 -c 100 http://127.0.0.1:4000/ result:

Time per request:       30.279 [ms] (mean)
Time per request:       0.303 [ms] (mean, across all concurrent requests)
Transfer rate:          645.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6   4.0      5      21
Processing:     3   24  11.7     22      54
Waiting:        2   20  10.5     18      48
Total:          7   29  10.1     27      57

Percentage of the requests served within a certain time (ms)
  50%     27
  66%     33
  75%     36
  80%     38
  90%     43
  95%     46
  98%     51
  99%     53
 100%     57 (longest request)

PHP with Silex

The composer.json

{
    "require": {
        "silex/silex": "~1.3"
    }
}

The code of index.php

<?php
require __DIR__ . '/vendor/autoload.php';
$app = new \Silex\Application();
$app['debug'] = false;
$app->get('/', function() {
    return "HelloWorld";
});
$app->run();

Start the PHP internal dev server with php -S localhost:3000. Well, this dev server is (probably) single processing, so the parallel request will be processed round robin. But hey, for fun..

with PHP 5.5.29

The results of ab -n 1000 -c 10 http://localhost:3000/

Time per request:       203.522 [ms] (mean)
Time per request:       20.352 [ms] (mean, across all concurrent requests)
Transfer rate:          6.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:   166  202  10.1    200     342
Waiting:       49  182  10.7    180     324
Total:        170  202  10.2    200     343

Percentage of the requests served within a certain time (ms)
  50%    200
  66%    203
  75%    205
  80%    206
  90%    210
  95%    214
  98%    224
  99%    240
 100%    343 (longest request)

Update: I spare the 100 concurrent requests test, because it would'nt be better results.

with PHP 7

The results of ab -n 1000 -c 10 http://localhost:3000/

Time per request:       176.218 [ms] (mean)
Time per request:       17.622 [ms] (mean, across all concurrent requests)
Transfer rate:          13.91 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:    26  175  22.1    177     229
Waiting:       24  174  22.0    176     229
Total:         26  175  22.1    177     230

Percentage of the requests served within a certain time (ms)
  50%    177
  66%    189
  75%    193
  80%    196
  90%    201
  95%    207
  98%    215
  99%    224
 100%    230 (longest request)

So it's ~30% faster looking at the longest request. But the 99% and other values are near to the 5.5. Ok, I run this test without opcache, maybe thats a problem.

NodeJS with Express

(inspired by expressjs.com/starter/hello-world.html)

The package.json

{
  ..
  "dependencies": {
    "express": "^4.13.3",
    "express-cluster": "0.0.4"
  }
}

as single instance

The code app_single.js

var express = require('express');
var app = express();
app.get('/', function (req, res) {
  res.send('Hello World!');
});
var server = app.listen(3000, function () {
  var host = server.address().address;
  var port = server.address().port;
  console.log('Listening at http://%s:%s', host, port);
});

The result of ab -n 1000 -c 10 http://localhost:3000/

Time per request:       9.832 [ms] (mean)
Time per request:       0.983 [ms] (mean, across all concurrent requests)
Transfer rate:          204.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     5   10   6.0      8      51
Waiting:        4    9   6.0      7      51
Total:          5   10   6.1      8      51

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      9
  75%     11
  80%     12
  90%     15
  95%     18
  98%     33
  99%     46
 100%     51 (longest request)

The result of ab -n 1000 -c 100 http://localhost:3000/

Time per request:       84.539 [ms] (mean)
Time per request:       0.845 [ms] (mean, across all concurrent requests)
Transfer rate:          237.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   2.0      0      12
Processing:    12   80  17.8     82     111
Waiting:       12   80  17.8     82     111
Total:         24   81  16.7     83     112

Percentage of the requests served within a certain time (ms)
  50%     83
  66%     93
  75%     94
  80%     95
  90%     99
  95%    101
  98%    105
  99%    109
 100%    112 (longest request)

as a cluster with express-cluster:

The code app_cluster.js

var express = require('express');
var cluster = require('express-cluster');
cluster(function(worker) {
    var app = express();
    app.get('/', function(req, res) {
        res.send('Hello World from worker #' + worker.id);
    });
    server = app.listen(3000, function () {
      var host = server.address().address;
      var port = server.address().port;
      console.log('Listening at http://%s:%s', host, port);
    });
    return server;
}, {count: 4})

(If you have more cores, than increase count: n)

The result for ab -n 1000 -c 10 http://localhost:3000/

Requests per second:    910.30 [#/sec] (mean)
Time per request:       10.985 [ms] (mean)
Time per request:       1.099 [ms] (mean, across all concurrent requests)
Transfer rate:          196.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:     1   11  13.3      6     104
Waiting:        1   10  12.6      6     104
Total:          1   11  13.3      7     105

Percentage of the requests served within a certain time (ms)
  50%      7
  66%      9
  75%     12
  80%     13
  90%     22
  95%     40
  98%     55
  99%     72
 100%    105 (longest request)

The result for ab -n 1000 -c 100 http://localhost:3000/.

Requests per second:    1155.37 [#/sec] (mean)
Time per request:       86.552 [ms] (mean)
Time per request:       0.866 [ms] (mean, across all concurrent requests)
Transfer rate:          249.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   6.8      0      28
Processing:     3   80  17.3     82     126
Waiting:        3   79  18.2     82     126
Total:         31   83  12.7     83     127

Percentage of the requests served within a certain time (ms)
  50%     83
  66%     87
  75%     89
  80%     91
  90%     95
  95%    102
  98%    112
  99%    118
 100%    127 (longest request)

Hm, no better performance for me now. Maybe, because "cluster" don't really mean "threading on multiple cores". Because it is a npm package it is running within the NodeJS instance and can't be multi-threading by default. (poor NodeJS). But there are other libs ideas like Nginx as proxy or clusting outside of NodeJS.

Elixir with Phoenix

The mix.ex

..
defp deps do
  [{:phoenix, "~> 0.15"},
   {:phoenix_ecto, "~> 0.8"},
   {:postgrex, ">= 0.0.0"},
   {:phoenix_html, "~> 1.4"},
   {:phoenix_live_reload, "~> 0.5", only: :dev},
   {:cowboy, "~> 1.0"}]
end

After bootstrapping Phoenix with mix phoenix.new hello_world we had to tune it a little bit.

We define a /hello/:name route for fun in file web/router.ex.

..
scope "/", HelloPhoenix do
  ..
  get "/hello/:name", HelloController, :world
  ..
end
..

Our layout should be nearly plain in web/templates/layout/app.html.eex

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Hello Phoenix!</title>
  </head>
  <body>
      <%= @inner %>
  </body>
</html>

The template for our route is also more or less minimal - file web/templates/hello/world.html.eex

<h1>From template: Hello <%= String.capitalize @name %>!</h1>

Well, it is'nt so plain like the other ones. The route /hello/:name works with the value of the parameter. But still ..

Update: Start Phoenix in production mode with MIX_ENV=prod PORT=4000 mix do compile, phoenix.digest, phoenix.server. (thanks for the tip @josevalim)

Update: The results for ab -n 1000 -c 10 http://127.0.0.1:4000/hello/benchmark:

Time per request:       4.781 [ms] (mean)
Time per request:       0.478 [ms] (mean, across all concurrent requests)
Transfer rate:          804.77 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       6
Processing:     1    4   2.9      3      24
Waiting:        1    4   2.7      3      24
Total:          1    5   3.0      4      24

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      5
  80%      6
  90%      8
  95%     11
  98%     14
  99%     16
 100%     24 (longest request)

Update: The results for ab -n 1000 -c 100 http://127.0.0.1:4000/hello/benchmark:

Time per request:       52.396 [ms] (mean)
Time per request:       0.524 [ms] (mean, across all concurrent requests)
Transfer rate:          734.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7   7.2      5      30
Processing:     3   42  15.6     41      85
Waiting:        3   39  14.5     38      83
Total:         13   50  13.2     50      86

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     55
  75%     58
  80%     61
  90%     69
  95%     73
  98%     75
  99%     78
 100%     86 (longest request)

Go with net/http

The code:

package main

import (
	"io"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		io.WriteString(w, "Hello world!")
	})
	http.ListenAndServe(":8080", nil)
}

The results for ab -n 1000 -c 10 http://127.0.0.1:8080/:

Requests per second:    3854.59 [#/sec] (mean)
Time per request:       2.594 [ms] (mean)
Time per request:       0.259 [ms] (mean, across all concurrent requests)
Transfer rate:          485.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.7      1       5
Processing:     0    2   0.9      1       7
Waiting:        0    1   0.7      1       6
Total:          1    2   1.1      2       7
WARNING: The median and mean for the processing time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      4
  95%      5
  98%      7
  99%      7
 100%      7 (longest request)

The ab -n 1000 -c 100 http://127.0.0.1:8080/ result:

Requests per second:    4097.54 [#/sec] (mean)
Time per request:       24.405 [ms] (mean)
Time per request:       0.244 [ms] (mean, across all concurrent requests)
Transfer rate:          516.19 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    9   5.2      8      31
Processing:     3   13   6.3     12      36
Waiting:        2   10   5.2      9      31
Total:          9   22   7.6     21      45

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     23
  75%     24
  80%     26
  90%     35
  95%     41
  98%     42
  99%     44
 100%     45 (longest request)

So well, Go is faster than Elixir with Plug. It's really really flippin fast! But hey, its like programming a lean webservice in C 😄

Conclusion

So well, the fully featured fat-lady framework Phoenix with its rounting, view, layout and template runs at the same speed like a pure NodeJS with http. And PHP plain just returning a string is not significant faster. Get your own tests and think about it.

Elixir and Phoenix is functional. Thats really another paradigm. That changes a lot. The way of programming, thinking, structuring the code. Its not easy, its not hard, its just different. But hey, I think its worth the efford of learning it.