When recovery fails, and the connection is closed, Lyra may leave pending publishes waiting forever

[spatula75]

If retries and recovery have been exhausted, and no further communication via AMQP is going to be possible, Lyra can be left in a state where there is still a thread running and blocked on a condition, which in turn can prevent the JVM from being able to exit. A thread dump showing an example of this:

"Thread-3" prio=5 tid=0x00007fc3a600b000 nid=0x6403 waiting on condition [0x000000011ac4c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007d81d2e08> (a net.jodah.lyra.internal.util.concurrent.ReentrantCircuit$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at net.jodah.lyra.internal.util.concurrent.ReentrantCircuit.await(ReentrantCircuit.java:63) at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:73) at net.jodah.lyra.internal.ChannelHandler.invoke(ChannelHandler.java:168) at com.sun.proxy.$Proxy25.basicPublish(Unknown Source) at x.sendReply(x.java:111) at x.run(x.java:80) at org.springbyexample.bean.scope.thread.ThreadScopeRunnable.run(ThreadScopeRunnable.java:46) at java.lang.Thread.run(Thread.java:745)

Manually sending the thread (which appears to be waiting on a retry that will never occur) an interrupt() will cause it to finally unblock with an InterruptedException, but it might be nice to handle this kind of occurrence gracefully.

To reproduce:

1. Set up a DefaultConsumer that has a handleDelivery method that spawns a thread to handle the details of the delivery. Configure Lyra to have one retry and one recovery only.
2. Have that delivery wait for 10 seconds or so before sending a publish event.
3. After sending a message to be handled by this DefaultConsumer, forcibly stop the RabbitMQ server.
4. Watch Lyra attempt a recovery, and then give up after a retry.
5. The thread that started finishes sleeping and tries to do a publish, which then blocks forever.

It may be easier to reproduce than this, but this is the way I've managed to do it.

[jhalterman]

Excellent report - off to reproduce this...

[jhalterman]

This fix should take care of things. When the recovery attempts are finished, any waiting threads will be interrupted, resulting in an exception such as AlreadyClosedException being thrown in the thread that was waiting to do a publish.

Let me know if there are still problems.

[spatula75]

Thanks, any idea when you might next roll a release that we can pick up out
of Maven Central?

Nick

On 26 November 2014 at 12:54, Jonathan Halterman notifications@github.com
wrote:

This fix should take care of things. When the recovery attempts are
complete, any waiting threads will be interrupted, resulting in an
exception such as AlreadyClosedException being thrown in the thread that
was waiting to do a publish.

Let me know if there are still problems.

—
Reply to this email directly or view it on GitHub
#40 (comment).

"Courage isn't just a matter of not being frightened, you know. It's being
afraid and doing what you have to do anyway."
-- Doctor Who - Planet of the Daleks
This message has been brought to you by Nick Johnson 2.3b1 and the number 6.