Benchmarks
Here is a preliminary report:
Each program was run on MacOS 11.1 using machine with Apple M1 (4+4 cores, 16GB RAM).
All data were compared with CRuby3.0.0.
Optcarrot is a NES emulator, and a semi-official benchmark program for Ruby, on https://github.com/mame/optcarrot.
You can see the results of optcarrot benchmark for ruruby and other Ruby implementations here.
| benchmark | CRuby | ruruby | rate |
|---|---|---|---|
| optcarrot | 56.09 ± 0.13 fps | 34.80 ± 0.08 fps | x 1.61 |
| optcarrot --opt | 130.53 ± 0.82 fps | 101.85 ± 1.08 fps | x 1.28 |
* With --opt option, optcarrot rewrites its own code for optimization.
Frames per seconds are shown.
The higher, the better.
| benchmark | CRuby | ruruby | rate |
|---|---|---|---|
| accessor_get.rb | 0.42 ± 0.00 s | 0.43 ± 0.00 s | x 1.02 |
| accessor_set.rb | 0.48 ± 0.00 s | 0.53 ± 0.00 s | x 1.10 |
| ivar_get.rb | 0.52 ± 0.00 s | 0.63 ± 0.00 s | x 1.21 |
| ivar_set.rb | 0.52 ± 0.01 s | 0.66 ± 0.01 s | x 1.25 |
| loop_times.rb | 0.72 ± 0.01 s | 0.60 ± 0.00 s | x 0.84 |
| loop_for.rb | 0.78 ± 0.01 s | 0.71 ± 0.00 s | x 0.91 |
| loop_whileloop.rb | 0.52 ± 0.00 s | 0.48 ± 0.01 s | x 0.93 |
| so_concatenate.rb | 0.51 ± 0.01 s | 0.48 ± 0.01 s | x 0.94 |
| string_scan_str.rb | 0.63 ± 0.00 s | 0.88 ± 0.01 s | x 1.40 |
| string_scan_re.rb | 0.95 ± 0.00 s | 0.87 ± 0.01 s | x 0.92 |
| fiber_allocate.rb | 0.59 ± 0.01 s | 0.41 ± 0.04 s | x 0.69 |
| fiber_switch.rb | 0.64 ± 0.00 s | 0.52 ± 0.01 s | x 0.82 |
| so_mandelbrot.rb | 1.18 ± 0.04 s | 1.36 ± 0.01 s | x 1.15 |
| app_mandelbrot.rb | 0.84 ± 0.00 s | 0.95 ± 0.00 s | x 1.13 |
| app_fibo.rb | 0.54 ± 0.01 s | 0.68 ± 0.00 s | x 1.25 |
| app_aobench.rb | 5.89 ± 0.01 s | 9.74 ± 0.03 s | x 1.65 |
| so_nbody.rb | 0.76 ± 0.01 s | 0.94 ± 0.00 s | x 1.22 |
Mean and SD of execution time (executed 8 times) in seconds are shown.
The lower, the better.
These programs are available on https://github.com/ruby/ruby/blob/master/benchmark.
The characteristics of the benchmark programs are described below.
Simply execute a times method with block repeatedly.
Simply execute a for-loop repeatedly.
Simply execute a while-loop repeatedly.
Join small strings repeatedly.
A lot of arythmetic calculations with local variables in for-loop, producing almost no objects.
Produce and consume a lot of objects, and execute numerous method calls.
A huge number of recursive method calls.
Produce and consume a lot of objects, executing a huge number of floating point calculations and method calls.
To investigate memory consumption, maximum resident set size was measured using gnu-time. Memory consumption of ruruby is around 20%-25% as compared with CRuby.
| benchmark | CRuby | ruruby | rate |
|---|---|---|---|
| simple_call.rb | 26.7M | 3.5M | x 0.13 |
| accessor_get.rb | 26.6M | 3.5M | x 0.13 |
| accessor_set.rb | 26.7M | 3.5M | x 0.13 |
| ivar_get.rb | 26.6M | 3.5M | x 0.13 |
| ivar_set.rb | 26.6M | 3.5M | x 0.13 |
| loop_times.rb | 26.7M | 3.5M | x 0.13 |
| loop_for.rb | 26.8M | 3.5M | x 0.13 |
| loop_whileloop.rb | 26.7M | 3.5M | x 0.13 |
| so_concatenate.rb | 75.8M | 27.3M | x 0.36 |
| string_scan_str.rb | 28.2M | 5.7M | x 0.20 |
| string_scan_re.rb | 28.5M | 5.5M | x 0.19 |
| fiber_allocate.rb | 209.7M | 229.8M | x 1.10 |
| fiber_switch.rb | 26.7M | 3.5M | x 0.13 |
| so_mandelbrot.rb | 27.2M | 4.5M | x 0.16 |
| app_mandelbrot.rb | 27.1M | 4.5M | x 0.16 |
| app_fibo.rb | 26.7M | 3.5M | x 0.13 |
| app_aobench.rb | 27.3M | 5.8M | x 0.21 |
| so_nbody.rb | 26.8M | 3.7M | x 0.14 |
| optcarrot | 100.9M | 65.2M | x 0.65 |
| optcarrot --opt | 117.2M | 95.1M | x 0.81 |