Catch juc.TimeoutException errors on reconnect

[andrewvc]

This automatically recovers in the event of all network failures now

Fixes #107 (comment)

I inspected this in pry if you're curious, here's the info from there:

From: /Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb @ line 502 MarchHare::Session#reconnecting_on_network_failures:

    494: def reconnecting_on_network_failures(interval_in_ms, &fn)
    495:   begin
    496:     fn.call
    497:   rescue IOError, MarchHare::ConnectionRefused, java.io.IOException => e
    498:     java.lang.Thread.sleep(interval_in_ms)
    499:
    500:     retry
    501:   rescue => e
 => 502:     require 'pry'; binding.pry
    503:   end
    504: end

[1] pry(#<MarchHare::Session>)> e.class
=> Java::JavaUtilConcurrent::TimeoutException
[2] pry(#<MarchHare::Session>)> e.backtrace
=> ["com.rabbitmq.utility.BlockingCell.get(com/rabbitmq/utility/BlockingCell.java:76)",
 "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(com/rabbitmq/utility/BlockingCell.java:110)",
 "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(com/rabbitmq/utility/BlockingValueOrException.java:36)",
 "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(com/rabbitmq/client/impl/AMQChannel.java:372)",
 "com.rabbitmq.client.impl.AMQConnection.start(com/rabbitmq/client/impl/AMQConnection.java:298)",
 "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:886)",
 "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:839)",
 "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:797)",
 "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:930)",
 "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:497)",
 "RUBY.new_connection_impl(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:518)",
 "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
 "RUBY.converting_rjc_exceptions_to_ruby(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:477)",
 "RUBY.new_connection_impl(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:512)",
 "RUBY.automatically_recover(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:267)",

<page break> --- Press enter to continue ( q<enter> to break ) --- <page break>

 "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
 "RUBY.reconnecting_on_network_failures(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:496)",
 "RUBY.automatically_recover(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:263)",
 "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
 "RUBY.converting_rjc_exceptions_to_ruby(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:477)",
 "RUBY.automatically_recover(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:262)",
 "RUBY.add_automatic_recovery_hook(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/session.rb:243)",
 "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)",
 "RUBY.shutdownCompleted(/Users/andrewvc/Downloads/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/march_hare-2.19.0-java/lib/march_hare/shutdown_listener.rb:12)",
 "MarchHare$$ShutdownListener_176621868.shutdownCompleted(MarchHare$$ShutdownListener_176621868.gen:13)",
 "MarchHare$$ShutdownListener_176621868.shutdownCompleted(MarchHare$$ShutdownListener_176621868.gen:13)",
 "com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(com/rabbitmq/client/impl/ShutdownNotifierComponent.java:77)",
 "com.rabbitmq.client.impl.AMQConnection$MainLoop.run(com/rabbitmq/client/impl/AMQConnection.java:593)",

<page break> --- Press enter to continue ( q<enter> to break ) --- <page break>

 "java.lang.Thread.run(java/lang/Thread.java:745)"]

[andrewvc]

Thanks @michaelklishin ! Much appreciated :) Do you plan on releasing a new gem soon?

[michaelklishin]

@andrewvc sure, I can do one tomorrow if you feel confident about this without more end-to-end testing with Logstash.

[andrewvc]

@michaelklishin I'd it if you could do one tomorrow. I'm pretty sure this is the fix. I'm not 100% sure that there aren't other possible exceptions, but this one would be amazing to get out to our users as a hotfix. I've reproduced this several times as has @jordansissel , so I'm not too worried about it.

Once again, thanks for all your help :)

[michaelklishin]

March Hare 2.20.0 is up.