Exception: “write while request pending” #195

Closed
mariusae opened this Issue Aug 19, 2013 · 22 comments

Comments

3 participants
Contributor

mariusae commented Aug 19, 2013

Reported by @rstrickland, with the following stack trace:

2013-08-19 07:55:14,025 ERROR c.w.d.d.FourInfoActor [New I/O  worker #22] <* DC [Failed] : java.lang.Exception: write while request pending (Chan=sms, Prod=severe, Addresses=List(6786707884))
com.twitter.finagle.WriteException$$anon$1: java.lang.Exception: write while request pending
    at com.twitter.finagle.transport.ClientChannelTransport$$anonfun$2.apply(ChannelTransport.scala:99) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport$$anonfun$2.apply(ChannelTransport.scala:96) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.util.Proc$class.$bang(Chan.scala:34) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.util.Proc$$anon$1.$bang(Chan.scala:47) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport.write(ChannelTransport.scala:146) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.dispatch.SerialClientDispatcher.dispatch(ClientDispatcher.scala:37) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.dispatch.SerialClientDispatcher$$anonfun$apply$2.apply(ClientDispatcher.scala:50) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.dispatch.SerialClientDispatcher$$anonfun$apply$2.apply(ClientDispatcher.scala:49) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Future$$anonfun$onSuccess$1.apply(Future.scala:645) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Future$$anonfun$onSuccess$1.apply(Future.scala:644) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise$Monitored.apply(Promise.scala:40) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise$Monitored.apply(Promise.scala:31) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise$$anon$2.run(Promise.scala:519) ~[Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.concurrent.Scheduler$LocalScheduler.run(Scheduler.scala:60) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.concurrent.Scheduler$LocalScheduler.submit(Scheduler.scala:40) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:26) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise.runq(Promise.scala:248) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:492) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Promise.update(Promise.scala:475) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.channel.ConnectionLifecycleHandler$$anonfun$com$twitter$finagle$channel$ConnectionLifecycleHandler$$channelDidConnect$1.apply$mcV$sp(ConnectionLifecycleHandler.scala:19) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.netty3.RichChannelFuture$$anonfun$onSuccessOrFailure$1.apply(ChannelFuture.scala:126) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.netty3.RichChannelFuture$$anonfun$onSuccessOrFailure$1.apply(ChannelFuture.scala:124) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.netty3.RichChannelFuture$$anon$1.operationComplete(ChannelFuture.scala:28) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:428) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:414) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:363) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.AbstractChannel$ChannelCloseFuture.setClosed(AbstractChannel.java:355) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.AbstractChannel.setClosed(AbstractChannel.java:185) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioChannel.setClosed(AbstractNioChannel.java:204) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.NioSocketChannel.setClosed(NioSocketChannel.java:85) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:717) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:99) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:774) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:338) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:338) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.channel.ChannelStatsHandler.closeRequested(ChannelStatsHandler.scala:83) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:97) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelDownstreamHandler.closeRequested(SimpleChannelDownstreamHandler.java:154) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelDownstreamHandler.handleDownstream(SimpleChannelDownstreamHandler.java:76) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelDownstreamHandler.closeRequested(SimpleChannelDownstreamHandler.java:154) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelDownstreamHandler.handleDownstream(SimpleChannelDownstreamHandler.java:76) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:338) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:576) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.close(Channels.java:820) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport.close(ChannelTransport.scala:156) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.util.Closable$class.close(Closable.scala:12) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport.close(ChannelTransport.scala:88) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport.fail(ChannelTransport.scala:115) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.transport.ClientChannelTransport.handleUpstream(ChannelTransport.scala:135) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:156) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:378) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:156) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.channel.ChannelStatsHandler.exceptionCaught(ChannelStatsHandler.scala:94) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:156) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:533) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:652) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.http.codec.ResponseDecoder.messageReceived(ResponseDecoder.scala:20) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:103) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:455) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:538) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:437) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:74) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at com.twitter.finagle.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [Distribution-Server-1.1.4-RELEASE-jar-with-dependencies.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_17]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_17]
    at java.lang.Thread.run(Thread.java:722) [?:1.7.0_17]
Caused by: java.lang.Exception: write while request pending
    ... 139 more

@ghost ghost assigned mariusae Aug 19, 2013

Contributor

mariusae commented Aug 19, 2013

@rstrickland: Would you mind clarifying some things?

The exception is presumably seen at the “top-level” service? I.e. in

val f = service(request)

f is what exhibits the exception.

In the code you’ve shared with me, you configure retries — presumably retries are active at the time you are experiencing this error as well?

Yes retries are active using the default of 3.

Contributor

mariusae commented Aug 19, 2013

Okay thanks. That gives me enough to try to simulate the error.

Contributor

mariusae commented Aug 19, 2013

@rstrickland Do you always use the service (from ClientBuilder.build) directly? Or do you also use it via ServiceFactory?

We always use the ClientBuilder.

Contributor

mariusae commented Aug 19, 2013

@rstrickland Do you know if these clients are using TLS or not?

This one is not using TLS.

Contributor

mariusae commented Aug 19, 2013

@rstrickland Can you do me a favor? Would you be able to try finagle 6.5.2? (Latest release).

Second, do you know if there are any entries like the following: “Exception propagated to the root monitor!” in your logs?

The only way I can make sense of your stack trace is that an exception was thrown in a respond block which propagates. Now, finagle monitors all such exceptions and logs the above message, but a somewhat recent change means that such exceptions aren’t propagated anymore (they never should’ve been).

If this is what’s happening, then you’ll get a proper error (the exception thrown in respond) and the connection state won’t be faulty (subsequent requests should continue to work).

Actually yes, we are getting a handful of these (about 0.005% of requests produce this), and this would be one thing our mock server doesn't emulate. So that would make sense.

Regarding moving up to 6.5.2, we would love to, but we're using Cassie which depends on 6.3. Do you know if the versions are binary compatible?

Contributor

mariusae commented Aug 19, 2013

I think they are binary compatible, but no promises. If that doesn’t work out I can nudge the Cassie maintainers to provide an updated artifact :-)

Ok I will do a little testing tomorrow and see if I can just drop in 6.5.2.

Contributor

mariusae commented Aug 19, 2013

Great, thanks! Let me know.

I haven't tested with 6.5.2 yet, but I updated our mock to emulate the exception that bubbles up to the root monitor. I was able to send 400k messages without error (other than the exception which occurs 0.1% of the time) using the same configuration that's in production. I was hoping I could simply handle this edge case as a triage measure, but I'm not convinced it's the root cause. I am uneasy about upgrading Finagle without extensive testing, because all of our services use Finagle for inter-process communication.

I noticed this morning that Cassie's newest release uses 6.5.0. Is it reasonable to assume that the fix in question would be in that version? I switched our client back to the Apache client as an immediate triage (which seems to be working fine), because I really want to be able to replicate the issue before trying another production fix. If you have any other theories I can check out, I'm happy to do so; I'd much prefer to use Finagle across the board.

Contributor

mariusae commented Aug 20, 2013

Re: reproducing the edge case -- to trigger the issue requires a specific ordering of events which is probably difficult to do synthetically.

Re: 6.5.0 yes that's new enough.

So what I hear you saying is that an upgrade to 6.5.0 will theoretically fix the issue, but there's no way to know for sure without deploying to production? If the unhandled exception puts the client in this state in our production system, why would it not do the same in our test system?

Contributor

mariusae commented Aug 21, 2013

The problem is that it depends on the connection behavior of the server as well. For example, how a server chooses to close an HTTP connection once the result is served. It also depends on which thread ends up executing the respond block which throws an exception (for example if it has been chained with other actions).

The totality of the situation would be very difficult to reproduce in a closed setting.

Contributor

mariusae commented Sep 12, 2013

Did you have a chance to try this out?

It's on our short-term backlog. We had to prioritize getting a major release out the door first, but this is a high priority item for us. I'll let you know when we have testing results.

Contributor

mariusae commented Sep 17, 2013

Sounds good.

Contributor

luciferous commented Jul 22, 2015

It's been two years so I'm going to assume it worked for him.

@luciferous luciferous closed this Jul 22, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment