HA issue - ShardManager will try to send a connect message to a dead actor if it receives an updated shard

[jasongoodwin]

It appears if brando has a shard connected to a dead redis instance that actor will die.
If we try to send a new shard to the shard manager for that host, then it will send a Connect message to the dead actor so it can never resolve in some failover scenarios.

I'll look at the code a bit more later - these are high level observations.

Course of action is to make the connections immutable and just poisonpill and recreate connections on Shard messages to the shard manager.

[jasongoodwin]

My observations may not be accurate but I see this in the log - I'll dig into this - I get this in the logs when updating a bad shard

[30/Aug/2013 11:26:57.611] [INFO] [StreamPublisher-akka.actor.default-dispatcher-23] akka.actor.LocalActorRef - Message [brando.Connect] from Actor[akka://StreamPublisher/user/$a/$a/$a/$g#2130182160] to Actor[akka://StreamPublisher/user/$a/$a/$a/$g/$a#-475118153] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Next message to a connected server works. We'll be doing redis ha with sharding testing today.

[jasongoodwin]

this is only a logging issue and does not affect functionality.