Channel reconnect regression with 3.0.0

[timcharper]

In previous versions of akka-rabbitmq, a connection close would result in automatic connection recovery. With 3.0.0 that behavior appears to be broken. The following scala script code can be evaluated by ammonite (open up ammonite, paste, etc.) to reproduce:

import $ivy.`com.thenewmotion::akka-rabbitmq:3.0.0`

import com.typesafe.config.ConfigFactory
import com.rabbitmq.client.{Channel, ConfirmListener, ShutdownListener, ShutdownSignalException}
import com.rabbitmq.client.AMQP.BasicProperties.Builder
import com.rabbitmq.client.AMQP.BasicProperties
import com.thenewmotion.akka.rabbitmq._
import akka.actor._

implicit val system = ActorSystem("very-system", ConfigFactory.parseString("akka.loglevel = DEBUG"))
val factory = new ConnectionFactory()
val connectionActor: ActorRef = system.actorOf(ConnectionActor.props(factory))
val builder = new Builder
builder.deliveryMode(2)
val props = builder.build

val channelActor = connectionActor.createChannel(
  ChannelActor.props { (channel, actor) =>
    println(s"==================== Channel 1 OPEN")
    channel.addShutdownListener(new ShutdownListener {
      def shutdownCompleted(cause: ShutdownSignalException): Unit =
        println(s"==================== Shutdown channel 1 ${cause}")
    })

    Thread.sleep(1000) // don't go so fast
    channel.basicPublish("somebadexchange", "somebadkey", props, "data".getBytes)
  }
)


val recoveringChannelActor = connectionActor.createChannel(
  ChannelActor.props { (channel, actor) =>
    println(s"==================== Channel 2 OPEN")
    channel.addShutdownListener(new ShutdownListener {
      def shutdownCompleted(cause: ShutdownSignalException): Unit =
        println(s"==================== Shutdown channel 2 ${cause}")
    })
  }
)

Output:

[DEBUG] [01/15/2017 23:25:07.904] [very-system-akka.actor.default-dispatcher-9] [akka://very-system/user/$a] akka://very-system/user/$a in Connected received CreateChannel(Props(Deploy(,Config(SimpleConfigObject({})),NoRouter,NoScopeGiven,,),class com.thenewmotion.akka.rabbitmq.ChannelActor,List($sess.cmd20$$$Lambda$2248/360073749@48c89983)),None): creating child Actor[akka://very-system/user/$a/$g#1999078385] with channel com.rabbitmq.client.impl.recovery.AutorecoveringChannel@376641cb
[DEBUG] [01/15/2017 23:25:07.904] [very-system-akka.actor.default-dispatcher-12] [akka://very-system/user/$a/$g] akka://very-system/user/$a/$g setting up new channel com.rabbitmq.client.impl.recovery.AutorecoveringChannel@376641cb
[DEBUG] [01/15/2017 23:25:07.906] [very-system-akka.actor.default-dispatcher-11] [akka://very-system/user/$a] akka://very-system/user/$a in Connected received CreateChannel(Props(Deploy(,Config(SimpleConfigObject({})),NoRouter,NoScopeGiven,,),class com.thenewmotion.akka.rabbitmq.ChannelActor,List($sess.cmd20$$$Lambda$2249/27082839@be7a0e6)),None): creating child Actor[akka://very-system/user/$a/$h#-1110941847] with channel com.rabbitmq.client.impl.recovery.AutorecoveringChannel@fd83542
==================== Channel 2 OPEN
==================== Channel 1 OPEN
[DEBUG] [01/15/2017 23:25:07.906] [very-system-akka.actor.default-dispatcher-10] [akka://very-system/user/$a/$h] akka://very-system/user/$a/$h setting up new channel com.rabbitmq.client.impl.recovery.AutorecoveringChannel@fd83542
[INFO] [01/15/2017 23:25:07.907] [very-system-akka.actor.default-dispatcher-10] [akka://very-system/user/$a/$h] akka://very-system/user/$a/$h connected
channelActor: ActorRef = Actor[akka://very-system/user/$a/$g#1999078385]
recoveringChannelActor: ActorRef = Actor[akka://very-system/user/$a/$h#-1110941847]


[INFO] [01/15/2017 23:25:08.912] [very-system-akka.actor.default-dispatcher-12] [akka://very-system/user/$a/$g] akka://very-system/user/$a/$g connected
==================== Shutdown channel 1 com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'somebadexchange' in vhost '/', class-id=60, method-id=40)
[DEBUG] [01/15/2017 23:25:08.913] [AMQP Connection 127.0.0.1:5672] [akka://very-system/user/$a/$g] on shutdownCompleted com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'somebadexchange' in vhost '/', class-id=60, method-id=40)

At this point, the first channel shuts down. No attempt is made to reconnect. If this is the expected behavior, then perhaps it should be documented? In previous versions, the channel was recovered.

Also, if I kill RabbitMQ, then channel #2 shuts down. However, when I restart it, the channel is not recreated, although the connection actor obviously still works as I can still create further channels at this point.

[efemelar]

Thank you for report @timcharper and for executable script!

It is certainly result of upgrade to amqp-client 4.0.0 where automatic recovery is turned on by default.
This interferes with com.thenewmotion.akka.rabbitmq.ChannelActor#shutdownSignalAppliesToChannel as reference is changed underneath.

replacing factory initialization with following

val factory = new ConnectionFactory() {
  setAutomaticRecoveryEnabled(false)
}

will yield expected result of channel recoveries.

At the moment i'm tempted to disable automatic recovery at the moment ConnectionActor Props created, publish 3.0.1 bugfix release and work on better approach meanwhile. That should unblock you with op-rabbit progress.

[timcharper]

Thank you for the reply, @fedgehog ! I should be able to work around this in op-rabbit. I'll do a milestone release with a big bold warning for those who are taking control over the connection factory aspect.

[efemelar]

There shouldn't be many of them in Scala-Akka lands, as autorecovery is feature that is there for long time in 3.X.X series of amqp-client. It was just disabled by default, now default is to enable.
Could you try using this branch disable_native_autorecovery @timcharper ?

[efemelar]

@timcharper i've published 3.0.1-SNAPSHOT with same changes applied as in aforementioned branch

[efemelar]

Published 3.0.1 with quickfix

[timcharper]

@fedgehog can you publish 3.0.1 to maven?

[timcharper]

Nevermind :) found 4.0.0