Unusually slow performance on "sig" benchmarks

[headius]

The "sig" library provides type-checking for Ruby. The benchmarks reported in their README don't reflect the performance we'd expect to see on JRuby 9000.

I've filed janlelis/sig#1 to hopefully get to the bottom of it.

[enebo]

A few observations after a casual glance:

1. Module prepends a define_method (which ends up having us go through block for execution)
2. this method calls super and I believe as a proc we end up fallng back to unresolved super
3. The method accepts **kwrest. Our kwarg impl is pretty slow atm

The first two we will eventually be able to eliminate in post 9k work since we plan on converting define_method procs to actual methods. This will also allow us to make the unresolved super resolved. The kwargs overhead I have no doubt we will be improving a bit early in 9k cycle.

[headius]

I updated the referenced issue with results from running the benchmark directly. The original results were run via a Rakefile, so the block-based IPS bench never jitted. Our performance is orders of magnitude better than what is in the README if JIT runs with indy enabled.

Our performance on the sig and contracts part of the bench now is closer to where it should be, but not where I'd like to see it:

[] ~/projects/sig $ rvm ruby-2.2 do ruby -I lib blah.rb 
ruby version: 2.2.0
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure    85.808k i/100ms
                 sig    11.573k i/100ms
           contracts     6.723k i/100ms
-------------------------------------------------
                pure      6.281M (±11.6%) i/s -     30.977M
                 sig    163.279k (± 6.0%) i/s -    821.683k
           contracts     83.552k (± 5.4%) i/s -    416.826k

Comparison:
                pure:  6281161.5 i/s
                 sig:   163278.7 i/s - 38.47x slower
           contracts:    83552.4 i/s - 75.18x slower


[] ~/projects/sig $ jruby -Xcompile.invokedynamic=true -Ilib blah.rb
ruby version: 2.2.2
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure    99.780k i/100ms
                 sig    11.177k i/100ms
           contracts     1.528k i/100ms
-------------------------------------------------
                pure     46.102M (±18.4%) i/s -    215.724M
                 sig    186.208k (± 5.5%) i/s -    938.868k
           contracts     51.995k (± 8.1%) i/s -    258.232k

Comparison:
                pure: 46101534.6 i/s
                 sig:   186207.7 i/s - 247.58x slower
           contracts:    51994.7 i/s - 886.66x slower

The sig library uses prepend, so there's no way to compare with JRuby 1.7.

[headius]

I realized that the type-checking wrapper sig generates uses define_method, and so it will never JIT in the normal situation (yet). Results with -Xjit.threshold=0 are better:

[] ~/projects/sig $ jruby -Xcompile.invokedynamic=true -Xjit.threshold=0 -Ilib blah.rb
ruby version: 2.2.2
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure   200.902k i/100ms
                 sig    20.817k i/100ms
           contracts     4.867k i/100ms
-------------------------------------------------
                pure     31.938M (±15.7%) i/s -    152.686M
                 sig    386.760k (± 6.4%) i/s -      1.936M
           contracts    125.764k (± 8.1%) i/s -    627.843k

Comparison:
                pure: 31937956.7 i/s
                 sig:   386760.3 i/s - 82.58x slower
           contracts:   125764.4 i/s - 253.95x slower

All numbers are comfortably ahead of MRI here.

The numbers for rbx are very confusing, though. They show a significant drop in perf for contracts but nearly identical performance between the pure and sig benchmarks. That doesn't seem possible; there's a lot of overhead in the type checking logic.

[headius]

This is no longer an issue; with proper configs/flags we are easily the fastest at these benchmarks.

The rbx numbers turned out to be bogus because they have a JIT bug preventing sig's typechecking from running in the jitted version of those methods.

@janlelis If you want to get accurate JRuby results, try running with -Xcompile.invokedynamic=true -Xjit.threshold=0 to use invokedynamic and compile everything right away. I think the rbx results should just be removed, since they aren't accurate at all.