bench some web microframeworks (Ruby)

Because we don't have enough drama, let's bench some hello world apps!

Why?

As part of some research, collected some benchmark around Ruby web microframeworks.

The idea is bench the performance from Rack, throught the stack of the framework to the minimum representation of an application (Hello World apps).

It is not the goal to bechmark hits to the database or external services, nor to do a silly fibonacci-as-a-service, ok?

What?

The following microframeworks were considered when doing this research:

• Brooklyn - brooklyn.ru
• Cuba - cuba.ru
• Gin - gin.ru
• Hobbit - hobbit.ru
• Lotus (Router) - lotus-router.ru
• Nancy - nancy.ru
• NYNY - nyny.ru
• Mustermann - mustermann.ru
• Rack - rack.ru + rack-response.ru
• Rails - rails.ru
• Ramaze - ramaze.ru
• Rambutan - rambutan.ru
• Roda - roda.ru
• Scorched - scorched.ru
• Sinatra - sinatra.ru
• Syro - syro

Please note that while Rails has been added to the list, it is just a minimalistic representation (using Metal, no full middleware stack, etc). You shouldn't take the performance numbers mentioned here about Rails (or any other) as scientific and decision-taking references.

How?

Used wrk to benchmark, locally, a burst of requests (in 2 threads) over 10 seconds. The command line used was:

$ wrk -t 2 http://localhost:9292/

All the frameworks were run using Puma on Ruby 2.1, in production mode and using 16 threads:

$ puma -e production -t 16:16 apps/<framework.ru>

Run benchmark for all frameworks

# use `bundle exec`, if needed
$ bench/summary-memory
$ bench/summary-speed

Have some numbers around?

Yup, I do:

Requests/sec

Framework            Requests/sec  % from best
----------------------------------------------
rack                     19138.63       100.0%
mustermann               15877.18       82.96%
roda                     15773.44       82.42%
syro                     15677.54       81.92%
cuba                     14664.93       76.62%
hobbit                   14346.23       74.96%
lotus-router             13298.08       69.48%
rack-response            12980.98       67.83%
rambutan                 10817.10       56.52%
brooklyn                 10116.29       52.86%
nancy                     9118.49       47.64%
nyny                      8170.25       42.69%
rails                     5847.11       30.55%
gin                       5689.37       29.73%
sinatra                   4559.17       23.82%
ramaze                    1986.19       10.38%

Memory Allocation/Request

Framework       Allocs/Req Memsize/Req
--------------------------------------
rack                    38        3256
syro                    43        3536
hobbit                  45        3808
roda                    45        3752
mustermann              51        4112
cuba                    53        4072
rack-response           56        5312
lotus-router            66        4944
brooklyn                69        6412
nancy                   75        7644
nyny                    82        8396
rambutan                85        6848
rails                  113       10311
gin                    198       17219
sinatra                212       15783
ramaze                 454       39670

These numbers were collected on:

• Ubuntu 15.10 64-bit (kernel: 4.2.0-19-generic)
• Dell XPS 13 (9343, QHD, Developer Edition)
• Intel® Core™ i7-5500U CPU @ 2.40GHz × 4
• 8GB RAM
• 256GB SSD
• ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]