Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO#read_nonblock blocks sometimes for SSL Sockets #1129

Closed
mohamedhafez opened this Issue Oct 15, 2013 · 8 comments

Comments

Projects
None yet
2 participants
@mohamedhafez
Copy link
Contributor

mohamedhafez commented Oct 15, 2013

IO#read_nonblock blocks sometimes for SSL Sockets. Based on Aaron Patterson's monkey patch in http://gist.github.com/251244, I added some logging in Thread.current[] for debugging and monkey patched Net::BufferedIO#rbuf_fill to be:

def rbuf_fill
    Thread.current[:rbuf_fill_loops] = 0

    begin
      Thread.current[:rbuf_fill_loops] += 1
      Thread.current[:rbuf_fill_stage] = "pre read_nonblock #{Time.now}"
      @rbuf << @io.read_nonblock(BUFSIZE)
    rescue OpenSSL::SSL::SSLErrorReadable, Errno::EWOULDBLOCK, Errno::EAGAIN => e
      Thread.current[:rbuf_fill_stage] = "rescue"
      raise if e.is_a?(OpenSSL::SSL::SSLErrorReadable) && e.message != "read would block"
      Thread.current[:rbuf_fill_error] = "#{$!.class}: #{$!.message}"
      Thread.current[:rbuf_fill_stage] = "pre io select"
      retry if IO.select [@io], nil, nil, 30.0
      raise Timeout::Error, e.message
    end

    Thread.current[:rbuf_fill_stage] = nil
    Thread.current[:rbuf_fill_loops] = nil
    Thread.current[:rbuf_fill_error] = nil
    @rbuf
end

Usually it works great (I use Net::HTTP to scrape some sites, which in turn uses this method), but every now and then it hangs at the read_nonblock for SSL sockets. If you wrap the whole method in a Timeout::timeout set for 5 minutes or so, you can see that it rarely but occasionally goes off and that when you examine Thread.current[:rbuf_fill_stage] it reads "pre read_nonblock ..." with a timestamp approx 5 minutes before the timeout fired. Also, for the 10 or so cases I've observed so far, Thread.current[:rbuf_fill_loops] is always 2, and Thread.current[:rbuf_fill_error] is always "OpenSSL::SSL::SSLErrorReadable: read would block".

I'm using JRuby 1.7.3 in ruby 1.9 mode, I know there are later JRuby releases but none seem to have any patches related to read_nonblock so I figure this is still a bug even in those releases.

@headius

This comment has been minimized.

Copy link
Member

headius commented Oct 21, 2013

Can you get it to block and then get a thread dump from JRuby/JVM? That would be ctrl+\ in running console or send QUIT signal or use "jstack" command. That would confirm that it's actually read_nonblock blocking.

I'll have a look at the code to see if I can see a path where this might happen.

@headius

This comment has been minimized.

Copy link
Member

headius commented Oct 21, 2013

After a quick look I could not see how read_nonblock (implemented by sysread_nonblock) would block in any case; the selector logic it does should see we're doing a nonblock and only select currently-ready channels, and if none are ready it should raise the appropriate error before doing a blocking read.

Get that thread dump to me and we'll see what we can see.

@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Oct 21, 2013

Sure thing, I'll set up something to capture that right now. It might take a few days or even a week to get it to you since I can't make it happen deterministically tho, I'll just post it here when I get it. Thanks so much for looking in to this!

@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Nov 4, 2013

Actually, I just got a timeout with a stack trace for a thread that did block on read_nonblock, for about 5 minutes (The timestamp i saved in Thread.current[:rbuf_fill_stage] was just under 5 minutes before the timeout). here's the stack trace below: (the line /var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:37 is @rbuf << @io.read_nonblock(BUFSIZE))

"RubyThread-306: /var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:4" daemon prio=10 tid=0x00007f8d41bb6800 nid=0x4b2b runnable [0x00007f8c90b01000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.NativeThread.current(Native Method)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:305)
        - locked <0x000000076aa38b68> (a java.lang.Object)
        - locked <0x000000076aa38b48> (a java.lang.Object)
        at org.jruby.ext.openssl.SSLSocket.readAndUnwrap(SSLSocket.java:513)
        at org.jruby.ext.openssl.SSLSocket.read(SSLSocket.java:501)
        at org.jruby.ext.openssl.SSLSocket.do_sysread(SSLSocket.java:613)
        at org.jruby.ext.openssl.SSLSocket.sysread_nonblock(SSLSocket.java:636)
        at org.jruby.ext.openssl.SSLSocket$INVOKER$i$0$1$sysread_nonblock.call(SSLSocket$INVOKER$i$0$1$sysread_nonblock.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:650)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:213)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.openssl.buffering.method__11$RUBY$read_nonblock(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/openssl/buffering.rb:174)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$openssl$buffering$method__11$RUBY$read_nonblock.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$openssl$buffering$method__11$RUBY$read_nonblock)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
        at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:260)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_bufferedio_rbuf_fill_patch.chained_4_rescue_1$RUBY$SYNTHETICrbuf_fill(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:37)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_bufferedio_rbuf_fill_patch.method__3$RUBY$rbuf_fill(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:30)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.protocol.chained_23_rescue_3$RUBY$SYNTHETICreaduntil(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/protocol.rb:122)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.protocol.method__22$RUBY$readuntil(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/protocol.rb:120)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__22$RUBY$readuntil.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__22$RUBY$readuntil)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
        at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:260)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.protocol.method__25$RUBY$readline(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/protocol.rb:132)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.http.method__217$RUBY$read_status_line(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/http.rb:2570)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.http.method__216$RUBY$read_new(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/http.rb:2559)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_http_ssl_connection_timeout.block_1$RUBY$transport_request(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_http_ssl_connection_timeout.rb:118)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$block_1$RUBY$transport_request.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$block_1$RUBY$transport_request)
        at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:139)
        at org.jruby.runtime.Block.yield(Block.java:130)
        at org.jruby.RubyContinuation.enter(RubyContinuation.java:107)
        at org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1181)
        at org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1174)
        at org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_http_ssl_connection_timeout.method__9$RUBY$transport_request(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_http_ssl_connection_timeout.rb:112)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$method__9$RUBY$transport_request.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$method__9$RUBY$transport_request)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
        at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_3.lib.ruby.$1_dot_9.net.http.method__74$RUBY$request(/usr/local/rvm/rubies/jruby-1.7.3/lib/ruby/1.9/net/http.rb:1301)
        at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__74$RUBY$request.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_3$lib$ruby$$1_dot_9$net$http$method__74$RUBY$request)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:217)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:356)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:213)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
        at org.jruby.runtime.callsite.CachingCallSite.callVarargsIter(CachingCallSite.java:124)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.mechanize_1_patch.chained_5_rescue_1$RUBY$SYNTHETICfetch_page(/var/www/vhosts/www.substitutealert.com/current/lib/ext/mechanize_1_patch.rb:162)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.mechanize_1_patch.method__4$RUBY$fetch_page(/var/www/vhosts/www.substitutealert.com/current/lib/ext/mechanize_1_patch.rb:160)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at var.www.vhosts.www_dot_substitutealert_dot_com.shared.bundle.jruby.$1_dot_9.gems.mechanize_minus_1_dot_0_dot_0.lib.mechanize.method__34$RUBY$post_form(/var/www/vhosts/www.substitutealert.com/shared/bundle/jruby/1.9/gems/mechanize-1.0.0/lib/mechanize.rb:457)
        at var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__34$RUBY$post_form.call(var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__34$RUBY$post_form)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:225)
        at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:270)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:236)
        at var.www.vhosts.www_dot_substitutealert_dot_com.shared.bundle.jruby.$1_dot_9.gems.mechanize_minus_1_dot_0_dot_0.lib.mechanize.method__23$RUBY$post(/var/www/vhosts/www.substitutealert.com/shared/bundle/jruby/1.9/gems/mechanize-1.0.0/lib/mechanize.rb:357)
        at var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__23$RUBY$post.call(var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__23$RUBY$post)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:217)
        at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:265)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.block_1$RUBY$post_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:59)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$block_1$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$block_1$RUBY$post_with_retry)
        at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:121)
        at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:96)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.chained_3_rescue_1$RUBY$SYNTHETICrequest_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:33)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.method__2$RUBY$request_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:32)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__2$RUBY$request_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__2$RUBY$request_with_retry)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.method__6$RUBY$post_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:59)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.block_17$RUBY$try_to_retrieve_available_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:294)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$block_17$RUBY$try_to_retrieve_available_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$block_17$RUBY$try_to_retrieve_available_jobs)
        at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:121)
        at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:96)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.chained_57_rescue_7$RUBY$SYNTHETICcatch_connection_errors(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:510)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.method__56$RUBY$catch_connection_errors(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:509)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__56$RUBY$catch_connection_errors.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__56$RUBY$catch_connection_errors)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.method__42$RUBY$try_to_retrieve_available_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:293)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__42$RUBY$try_to_retrieve_available_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__42$RUBY$try_to_retrieve_available_jobs)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__42$RUBY$try_to_retrieve_available_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__42$RUBY$try_to_retrieve_available_jobs)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.subfinder_task.chained_10_rescue_1$RUBY$SYNTHETICget_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task/subfinder_task.rb:91)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.subfinder_task.method__9$RUBY$get_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task/subfinder_task.rb:72)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__9$RUBY$get_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__9$RUBY$get_jobs)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__9$RUBY$get_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__9$RUBY$get_jobs)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.block_7$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:108)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_7$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_7$RUBY$run_once)
        at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:139)
        at org.jruby.runtime.Block.yield(Block.java:130)
        at org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
        at org.jruby.RubyArray.each(RubyArray.java:1613)
        at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.block_6$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:108)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_6$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_6$RUBY$run_once)
        at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:121)
        at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:96)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.timeoutable_thread.chained_4_ensure_1$RUBY$__ensure__(/var/www/vhosts/www.substitutealert.com/current/lib/ext/timeoutable_thread.rb:26)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.timeoutable_thread.method__3$RUBY$timeoutable(/var/www/vhosts/www.substitutealert.com/current/lib/ext/timeoutable_thread.rb:25)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$timeoutable_thread$method__3$RUBY$timeoutable.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$timeoutable_thread$method__3$RUBY$timeoutable)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.chained_9_rescue_1$RUBY$SYNTHETICrun_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:106)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.method__8$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:97)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__8$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__8$RUBY$run_once)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__8$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__8$RUBY$run_once)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.chained_9_rescue_2$RUBY$SYNTHETICworker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:206)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.chained_8_rescue_1$RUBY$SYNTHETICworker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:177)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.method__7$RUBY$worker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:124)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.block_5$RUBY$start_one_thread(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:116)
        at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$block_5$RUBY$start_one_thread.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$block_5$RUBY$start_one_thread)
        at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:163)
        at org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:91)
        at org.jruby.runtime.Block.call(Block.java:89)
        at org.jruby.RubyProc.call(RubyProc.java:261)
        at org.jruby.RubyProc.call(RubyProc.java:213)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:98)
        at java.lang.Thread.run(Thread.java:722)

@mohamedhafez mohamedhafez reopened this Nov 4, 2013

@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Nov 5, 2013

I've gotten a few more similar stack traces to the one above in the last couple days, so it doesn't just seem to be a fluke.

@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Nov 15, 2013

As a temporary workaround until this gets fixed I've put a timeout around the read_nonblock call:

Timeout::timeout(60, IOReadNonblockTimeoutError) {
    @rbuf << @io.read_nonblock(BUFSIZE)
}

And I'm getting IOReadNonblockTimeoutError thrown every now and then, it seems like this is definitely a problem with read_nonblock hanging...

@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Dec 11, 2013

I thought this might be due to #549, and I updated to JRuby 1.7.9, however I am still getting hanging threads at the @rbuf << @io.read_nonblock(BUFSIZE) line, /var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:22. Here's the new stack trace:

"RubyThread-61: /var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:4" daemon prio=10 tid=0x00007f18a9fb9000 nid=0x3b8d runnable [0x00007f18821ec000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
- locked <0x000000075b48feb8> (a java.lang.Object)
at org.jruby.ext.openssl.SSLSocket.readAndUnwrap(SSLSocket.java:513)
at org.jruby.ext.openssl.SSLSocket.read(SSLSocket.java:501)
at org.jruby.ext.openssl.SSLSocket.do_sysread(SSLSocket.java:613)
at org.jruby.ext.openssl.SSLSocket.sysread_nonblock(SSLSocket.java:636)
at org.jruby.ext.openssl.SSLSocket$INVOKER$i$0$1$sysread_nonblock.call(SSLSocket$INVOKER$i$0$1$sysread_nonblock.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:669)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:214)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.shared.jopenssl19.openssl.buffering.method__11$RUBY$read_nonblock(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/shared/jopenssl19/openssl/buffering.rb:174)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$shared$jopenssl19$openssl$buffering$method__11$RUBY$read_nonblock.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$shared$jopenssl19$openssl$buffering$method__11$RUBY$read_nonblock)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:263)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_bufferedio_rbuf_fill_patch.chained_4_rescue_1$RUBY$SYNTHETICrbuf_fill(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:22)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_bufferedio_rbuf_fill_patch.method__3$RUBY$rbuf_fill(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_bufferedio_rbuf_fill_patch.rb:21)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_bufferedio_rbuf_fill_patch$method__3$RUBY$rbuf_fill)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.protocol.chained_23_rescue_3$RUBY$SYNTHETICreaduntil(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/protocol.rb:122)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.protocol.method__22$RUBY$readuntil(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/protocol.rb:120)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__22$RUBY$readuntil.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__22$RUBY$readuntil)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:263)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.protocol.method__25$RUBY$readline(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/protocol.rb:132)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$protocol$method__25$RUBY$readline)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.http.method__217$RUBY$read_status_line(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/http.rb:2570)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__217$RUBY$read_status_line)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.http.method__216$RUBY$read_new(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/http.rb:2559)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__216$RUBY$read_new)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_http_ssl_connection_timeout.block_1$RUBY$transport_request(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_http_ssl_connection_timeout.rb:118)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$block_1$RUBY$transport_request.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$block_1$RUBY$transport_request)
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyContinuation.enter(RubyContinuation.java:107)
at org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1289)
at org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1282)
at org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.net_http_ssl_connection_timeout.method__9$RUBY$transport_request(/var/www/vhosts/www.substitutealert.com/current/lib/ext/net_http_ssl_connection_timeout.rb:112)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$method__9$RUBY$transport_request.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$net_http_ssl_connection_timeout$method__9$RUBY$transport_request)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at usr.local.rvm.rubies.jruby_minus_1_dot_7_dot_9.lib.ruby.$1_dot_9.net.http.method__74$RUBY$request(/usr/local/rvm/rubies/jruby-1.7.9/lib/ruby/1.9/net/http.rb:1301)
at usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__74$RUBY$request.call(usr$local$rvm$rubies$jruby_minus_1_dot_7_dot_9$lib$ruby$$1_dot_9$net$http$method__74$RUBY$request)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:356)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:213)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
at org.jruby.runtime.callsite.CachingCallSite.callVarargsIter(CachingCallSite.java:124)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.mechanize_1_patch.chained_5_rescue_1$RUBY$SYNTHETICfetch_page(/var/www/vhosts/www.substitutealert.com/current/lib/ext/mechanize_1_patch.rb:162)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.mechanize_1_patch.method__4$RUBY$fetch_page(/var/www/vhosts/www.substitutealert.com/current/lib/ext/mechanize_1_patch.rb:160)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$mechanize_1_patch$method__4$RUBY$fetch_page)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at var.www.vhosts.www_dot_substitutealert_dot_com.shared.bundle.jruby.$1_dot_9.gems.mechanize_minus_1_dot_0_dot_0.lib.mechanize.method__34$RUBY$post_form(/var/www/vhosts/www.substitutealert.com/shared/bundle/jruby/1.9/gems/mechanize-1.0.0/lib/mechanize.rb:457)
at var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__34$RUBY$post_form.call(var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__34$RUBY$post_form)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)
at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:273)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:236)
at var.www.vhosts.www_dot_substitutealert_dot_com.shared.bundle.jruby.$1_dot_9.gems.mechanize_minus_1_dot_0_dot_0.lib.mechanize.method__23$RUBY$post(/var/www/vhosts/www.substitutealert.com/shared/bundle/jruby/1.9/gems/mechanize-1.0.0/lib/mechanize.rb:357)
at var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__23$RUBY$post.call(var$www$vhosts$www_dot_substitutealert_dot_com$shared$bundle$jruby$$1_dot_9$gems$mechanize_minus_1_dot_0_dot_0$lib$mechanize$method__23$RUBY$post)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:268)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.block_1$RUBY$post_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:59)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$block_1$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$block_1$RUBY$post_with_retry)
at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:117)
at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:92)
at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.chained_3_rescue_1$RUBY$SYNTHETICrequest_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:33)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.method__2$RUBY$request_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:32)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__2$RUBY$request_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__2$RUBY$request_with_retry)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.safe_mech.method__6$RUBY$post_with_retry(/var/www/vhosts/www.substitutealert.com/current/lib/ext/safe_mech.rb:59)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$safe_mech$method__6$RUBY$post_with_retry)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.block_16$RUBY$retrieve_current_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:283)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$block_16$RUBY$retrieve_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$block_16$RUBY$retrieve_current_jobs)
at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:117)
at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:92)
at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.chained_57_rescue_7$RUBY$SYNTHETICcatch_connection_errors(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:520)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.method__56$RUBY$catch_connection_errors(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:519)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__56$RUBY$catch_connection_errors.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__56$RUBY$catch_connection_errors)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.method__40$RUBY$retrieve_current_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:282)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__40$RUBY$retrieve_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__40$RUBY$retrieve_current_jobs)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__40$RUBY$retrieve_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__40$RUBY$retrieve_current_jobs)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.chained_71_rescue_10$RUBY$SYNTHETICget_current_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:608)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.app_lib.subfinder_agent.method__70$RUBY$get_current_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/app_lib/subfinder_agent.rb:606)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__70$RUBY$get_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$app_lib$subfinder_agent$method__70$RUBY$get_current_jobs)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:263)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.subfinder_task.method__7$RUBY$get_current_jobs(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task/subfinder_task.rb:53)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__7$RUBY$get_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__7$RUBY$get_current_jobs)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__7$RUBY$get_current_jobs.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$subfinder_task$method__7$RUBY$get_current_jobs)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.block_10$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:126)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_10$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_10$RUBY$run_once)
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
at org.jruby.RubyArray.each(RubyArray.java:1613)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.block_7$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:126)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_7$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$block_7$RUBY$run_once)
at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:117)
at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:92)
at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.timeoutable_thread.chained_4_ensure_1$RUBY$__ensure__(/var/www/vhosts/www.substitutealert.com/current/lib/ext/timeoutable_thread.rb:26)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.ext.timeoutable_thread.method__3$RUBY$timeoutable(/var/www/vhosts/www.substitutealert.com/current/lib/ext/timeoutable_thread.rb:25)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$timeoutable_thread$method__3$RUBY$timeoutable.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$ext$timeoutable_thread$method__3$RUBY$timeoutable)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.chained_11_rescue_1$RUBY$SYNTHETICrun_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:119)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.user_task.method__10$RUBY$run_once(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master/user_task.rb:108)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__10$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__10$RUBY$run_once)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__10$RUBY$run_once.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$user_task$method__10$RUBY$run_once)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.chained_9_rescue_2$RUBY$SYNTHETICworker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:206)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.chained_8_rescue_1$RUBY$SYNTHETICworker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:177)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.method__7$RUBY$worker_thread_method(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:124)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$method__7$RUBY$worker_thread_method)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
at var.www.vhosts.www_dot_substitutealert_dot_com.current.lib.daemons.user_task_master.block_5$RUBY$start_one_thread(/var/www/vhosts/www.substitutealert.com/current/lib/daemons/user_task_master.rb:116)
at var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$block_5$RUBY$start_one_thread.call(var$www$vhosts$www_dot_substitutealert_dot_com$current$lib$daemons$user_task_master$block_5$RUBY$start_one_thread)
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:159)
at org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:290)
at org.jruby.RubyProc.call(RubyProc.java:228)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:97)
at java.lang.Thread.run(Thread.java:722)
@mohamedhafez

This comment has been minimized.

Copy link
Contributor Author

mohamedhafez commented Dec 11, 2013

I believe this is the same issue as #1280, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.