JRuby fatal crash when reloading

[kares]

LS 7.6.1 (JRuby 9.2.9.0) sometimes fails bad while pruning metric-store (on reload) with a:
Java::JavaLang::ArrayIndexOutOfBoundsException: Index 10 out of bounds for length 10

org.jruby.runtime.ThreadContext.getNextFrame(ThreadContext.java:550)
org.jruby.runtime.ThreadContext.pushFrameForBlock(ThreadContext.java:896)
org.jruby.runtime.ThreadContext.preYieldNoScope(ThreadContext.java:1058)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_6.lib.concurrent_minus_ruby.concurrent.map.RUBY$block$keys$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/map.rb)
org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)
org.jruby.runtime.IRBlockBody.yieldSpecificMultiArgsCommon(IRBlockBody.java:123)
org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:135)
org.jruby.runtime.Block.yieldSpecific(Block.java:157)
com.concurrent_ruby.ext.JRubyMapBackendLibrary$JRubyMapBackend.each_pair(JRubyMapBackendLibrary.java:227)
com.concurrent_ruby.ext.JRubyMapBackendLibrary$JRubyMapBackend$INVOKER$i$0$0$each_pair.call(JRubyMapBackendLibrary$JRubyMapBackend$INVOKER$i$0$0$each_pair.gen)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)
org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)
org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)
org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_6.lib.concurrent_minus_ruby.concurrent.map.RUBY$method$each_pair$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/map.rb:262)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_6.lib.concurrent_minus_ruby.concurrent.map.RUBY$method$each_pair$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/map.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_6.lib.concurrent_minus_ruby.concurrent.map.RUBY$method$keys$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/map.rb:224)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_6.lib.concurrent_minus_ruby.concurrent.map.RUBY$method$keys$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/map.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
usr.share.logstash.logstash_minus_core.lib.logstash.instrument.metric_store.RUBY$block$prune$1(/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:188)
org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)
org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:85)
org.jruby.runtime.Block.yieldSpecific(Block.java:139)
org.jruby.ext.thread.Mutex.synchronize(Mutex.java:164)
org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)
usr.share.logstash.logstash_minus_core.lib.logstash.instrument.metric_store.RUBY$method$prune$0(/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:187)
usr.share.logstash.logstash_minus_core.lib.logstash.instrument.metric_store.RUBY$method$prune$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
usr.share.logstash.logstash_minus_core.lib.logstash.instrument.collector.RUBY$method$clear$0(/usr/share/logstash/logstash-core/lib/logstash/instrument/collector.rb:64)
usr.share.logstash.logstash_minus_core.lib.logstash.instrument.collector.RUBY$method$clear$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/instrument/collector.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$clear_pipeline_metrics$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:459)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$clear_pipeline_metrics$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:376)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop.RUBY$block$execute$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop.rb:14)
org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)
org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
org.jruby.runtime.Block.yield(Block.java:170)
org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:499)
org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)
usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:88)
usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop.rb:13)
usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:84)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)
usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
org.jruby.runtime.Block.call(Block.java:125)
org.jruby.RubyProc.call(RubyProc.java:274)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
java.base/java.lang.Thread.run(Thread.java:834)

had a theory that this relates to crossing the Ruby->Java boundary on Concurrent::Map's keys passing a block down to each_pair which calls a native super. however no luck reproducing.

also no hints at JRuby's end - there seems to be no issues indicating missing frames in 9.2.9+.

note that, if this turns out to be related to block's framing, it's worth also reproducing in JRuby 9.2.10.0+ due work towards reduced blocks cost: jruby/jruby#6005

[jsvd]

For those hitting pipeline reload crashes please test 7.8.1, which contains the change #11900.

This change removes some back and forth bettween jruby and java regarding threads. With this removed I could no longer replicated these crashes.

Other similar stack traces:

[2020-08-12T14:27:37,600][FATAL][1|LogStash::Runner:logstash.runner ] An unexpected error occurred! {:error=>java.lang.ArrayIndexOutOfBoundsException: -3, :backtrace=>["org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:537)", "org.jruby.runtime.ThreadContext.postYieldNoScope(ThreadContext.java:1083)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:330)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:125)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:745)"]}