bench some web microframeworks (Ruby)

Because we don't have enough drama, let's bench some hello world apps!

Why?

As part of some research, collected some benchmark around Ruby web microframeworks.

The idea is bench the performance from Rack, throught the stack of the framework to the minimum representation of an application (Hello World apps).

It is not the goal to bechmark hits to the database or external services, nor to do a silly fibonacci-as-a-service, ok?

What?

The following microframeworks were considered when doing this research:

• Brooklyn - brooklyn.ru
• Cuba - cuba.ru
• Gin - gin.ru
• Hobbit - hobbit.ru
• Lotus (Router) - lotus-router.ru
• Nancy - nancy.ru
• NYNY - nyny.ru
• Mustermann - mustermann.ru
• Rack - rack.ru + rack-response.ru
• Rails - rails.ru
• Ramaze - ramaze.ru
• Rambutan - rambutan.ru
• Roda - roda.ru
• Scorched - scorched.ru
• Sinatra - sinatra.ru

Please note that while Rails has been added to the list, it is just a minimalistic representation (using Metal, no full middleware stack, etc). You shouldn't take the performance numbers mentioned here about Rails (or any other) as scientific and decision-taking references.

How?

Used wrk to benchmark, locally, a burst of requests (in 2 threads) over 10 seconds. The command line used was:

$ wrk -t 2 http://localhost:9292/

All the frameworks were run using Puma on Ruby 2.1, in production mode and using 16 threads:

$ puma -e production -t 16:16 apps/<framework.ru>

Run benchmark for all frameworks

# use `bundle exec`, if needed
$ bench/summary-memory
$ bench/summary-speed

Have some numbers around?

Yup, I do:

Requests/sec

Framework            Requests/sec  % from best
----------------------------------------------
rack                     18979.54       100.0%
mustermann               16466.79       86.76%
roda                     15329.83       80.77%
syro                     15312.13       80.68%
cuba                     13921.44       73.35%
hobbit                   13804.59       72.73%
rack-response            13564.87       71.47%
lotus-router             13341.14       70.29%
rambutan                 10982.39       57.86%
brooklyn                 10136.64       53.41%
nancy                     8966.56       47.24%
nyny                      8107.73       42.72%
rails                     5773.34       30.42%
gin                       5670.34       29.88%
sinatra                   4544.52       23.94%
ramaze                    2005.40       10.57%

Memory Allocation/Request

Framework       Allocs/Req Memsize/Req
--------------------------------------
rack                    38        3256
syro                    43        3536
hobbit                  45        3808
roda                    45        3752
mustermann              51        4112
cuba                    53        4072
rack-response           56        5312
lotus-router            66        4944
brooklyn                69        6412
nancy                   75        7644
nyny                    82        8396
rambutan                85        6848
rails                  113       10311
gin                    198       17219
sinatra                212       15783
ramaze                 454       39670

These numbers were collected on:

• Ubuntu 15.10 64-bit (kernel: 4.2.0-19-generic)
• Dell XPS 13 (9343, QHD, Developer Edition)
• Intel® Core™ i7-5500U CPU @ 2.40GHz × 4
• 8GB RAM
• 256GB SSD
• ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]