Oj is not always faster than MessagePack

[frsyuki]

I run "test/perf_strict.rb" but it showed different result on my environment than yours.
Could you tell me your environment? I want to improve msgpack.

My environment is:

$ ruby --version
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.2.0]
$ gem list | grep json
json (1.7.3, 1.6.5, 1.6.3, 1.6.1, 1.5.2, 1.5.0)
$ gem list | grep oj
oj (1.2.8)
$ gem list | grep msgpack
msgpack (0.4.7, 0.4.6, 0.4.4, 0.4.3)

It showed same tendency on following environment:

$ ruby --version
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]

[ohler55]

If you create a pull request for improving the way I use msgpack in the tests I will be very happy to merge the pull request. As it is now, you are using an older version of Ruby and I can not reproduce the tests results.

I don't remember what machine I ran the tests on. I think it was a MacBook Air. I can update the tests and rerun on a iMac once that is done. I would be glad to rerun the tests then.

BTW, nice work on msgpack.

[frsyuki]

I created another pull request to improve the way to use msgpack: #19

And I ran the tests using the latest ruby, ruby-1.9.3-p194. Here is the result:
https://gist.github.com/2692477

[ohler55]

new pull request takes precedence

[ohler55]

I ran the updated tests on a iMac 2.8GHz i7, OS X 10.7.3, Ruby 1.9.3-p194. The results are below. I'm not sure why there is a difference from what you got. I plan on seeing if I can improve the performance on the dump() still. I will update the README but I'd like to figure out why we get different results before I do.

ohler@saba test (master)> perf_strict.rb -no
--------------------------------------------------------------------------------
Load/Parse Performance
JSON::Ext.parse 10000 times in 0.281 seconds or 35540.265 parse/sec.
JSON::Pure.parse 10000 times in 2.948 seconds or 3392.587 parse/sec.
Oj:compat.load 10000 times in 0.167 seconds or 60056.814 load/sec.
Oj.load 10000 times in 0.165 seconds or 60437.568 load/sec.
Yajl.parse 10000 times in 0.309 seconds or 32367.173 parse/sec.
Ox.load 10000 times in 0.203 seconds or 49222.288 load/sec.
MessagePack.unpack 10000 times in 0.188 seconds or 53178.195 unpack/sec.

Summary:
     System  time (secs)  rate (ops/sec)
-----------  -----------  --------------
         Oj       0.165       60437.568
  Oj:compat       0.167       60056.814
MessagePack       0.188       53178.195
         Ox       0.203       49222.288
  JSON::Ext       0.281       35540.265
       Yajl       0.309       32367.173
 JSON::Pure       2.948        3392.587

Comparison Matrix
(performance factor, 2.0 means row is twice as fast as column)
                      Oj    Oj:compat  MessagePack           Ox    JSON::Ext         Yajl   JSON::Pure
-----------  -----------  -----------  -----------  -----------  -----------  -----------  -----------
         Oj         1.00         1.01         1.14         1.23         1.70         1.87        17.81
  Oj:compat         0.99         1.00         1.13         1.22         1.69         1.86        17.70
MessagePack         0.88         0.89         1.00         1.08         1.50         1.64        15.67
         Ox         0.81         0.82         0.93         1.00         1.38         1.52        14.51
  JSON::Ext         0.59         0.59         0.67         0.72         1.00         1.10        10.48
       Yajl         0.54         0.54         0.61         0.66         0.91         1.00         9.54
 JSON::Pure         0.06         0.06         0.06         0.07         0.10         0.10         1.00


--------------------------------------------------------------------------------
Dump/Encode/Generate Performance
JSON::Ext.generate 10000 times in 0.447 seconds or 22353.962 generate/sec.
JSON::Pure.generate 10000 times in 1.659 seconds or 6026.282 generate/sec.
Oj.dump 10000 times in 0.111 seconds or 89974.987 dump/sec.
Oj:compat.dump 10000 times in 0.104 seconds or 96213.054 dump/sec.
Yajl.encode 10000 times in 0.129 seconds or 77380.814 encode/sec.
Ox.dump 10000 times in 0.118 seconds or 84840.669 dump/sec.
MessagePack.pack 10000 times in 0.076 seconds or 130725.790 pack/sec.

Summary:
     System  time (secs)  rate (ops/sec)
-----------  -----------  --------------
MessagePack       0.076      130725.790
  Oj:compat       0.104       96213.054
         Oj       0.111       89974.987
         Ox       0.118       84840.669
       Yajl       0.129       77380.814
  JSON::Ext       0.447       22353.962
 JSON::Pure       1.659        6026.282

Comparison Matrix
(performance factor, 2.0 means row is twice as fast as column)
             MessagePack    Oj:compat           Oj           Ox         Yajl    JSON::Ext   JSON::Pure
-----------  -----------  -----------  -----------  -----------  -----------  -----------  -----------
MessagePack         1.00         1.36         1.45         1.54         1.69         5.85        21.69
  Oj:compat         0.74         1.00         1.07         1.13         1.24         4.30        15.97
         Oj         0.69         0.94         1.00         1.06         1.16         4.03        14.93
         Ox         0.65         0.88         0.94         1.00         1.10         3.80        14.08
       Yajl         0.59         0.80         0.86         0.91         1.00         3.46        12.84
  JSON::Ext         0.17         0.23         0.25         0.26         0.29         1.00         3.71
 JSON::Pure         0.05         0.06         0.07         0.07         0.08         0.27         1.00

[frsyuki]

I'm not sure why there is a difference from what you got

I think the difference is within a margin of error. They show the same tendency.

It's a known thing that it's difficult to measure meaningful difference using this kind of microbenchmarks because influence of nondeterministic factors such as GC or CPU cache is relatively large.

I'd like to figure out why we get different results before I do.

Since the environment you used is unknown, it's hard to investigate, but I guess following possibilities:

• it used different ruby (e.g. rubinius)
• version of msgpack was old
  • I released msgpack v0.4.7 on May 5, 2012
  • last major performance improvements of #to_msgpack methods were on Feb 2009, though
• the compiler was different
  • It may make big difference because msgpack code depends on compilers' inline expansion optimization
• compilation flags were different
  • I changed them for msgpack at v0.4.6, which was released on Aug 2011.
• influence of other nondeterministic factors

[ohler55]

For Oj, Rubinius is supported but it does not perform as well as I have not optimized for the missing parts in Rubinius. The tests above used msgpack 0.4.7. You may have missed the environment, iMac 2.8GHz i7, OS X 10.7.3, Ruby 1.9.3-p194. I suppose, if you do not want to push it further we can just let it rest. If you want to tweak msgpack to make it faster I will hold off on publishing the revised results.

[frsyuki]

Sorry, I found I misunderstood your comment "I'd like to figure out why we get different results before I do" as "I'd like to figure out why I got different results from the result I wrote to README.md, before I do."

So, I just think the difference is within a margin of error. I mean both of our results are correct. They show the same tendency:

• deserialization: Oj:compact ≒ MessagePack ≒ Oj > Ox > JSON::Ext ≒ Yajl >> JSON::Pure
• serialization: MessagePack > Oj:compact ≒ Oj ≒ Ox ≒ Yajl > JSON::Ext >> JSON::Pure