enable remember-entities with DData Cluster.Sharding

[Aaronontheweb]

closes #4254

[Aaronontheweb]

Based on the MNTR logs, I think I've figured out the issue here:

[DEBUG][2/26/2020 6:12:04 PM][Thread 0031][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/EntityCoordinator/singleton/coordinator] Shard [1] allocated at [[akka://PersistentClusterShardingRememberEntitiesSpec/system/sharding/Entity#868160794]]
[DEBUG][2/26/2020 6:12:04 PM][Thread 0033][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/Entity] Host shard [1]
[DEBUG][2/26/2020 6:12:04 PM][Thread 0033][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/Entity] Starting shard [1] in region
[DEBUG][2/26/2020 6:12:04 PM][Thread 0032][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/replicator] Received get for key shard-Entity-1-0, local value , consistency: ReadMajority(timeout=00:00:05)
[DEBUG][2/26/2020 6:12:04 PM][Thread 0032][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/replicator] Received get for key shard-Entity-1-1, local value , consistency: ReadMajority(timeout=00:00:05)
[DEBUG][2/26/2020 6:12:04 PM][Thread 0032][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/replicator] Received get for key shard-Entity-1-2, local value , consistency: ReadMajority(timeout=00:00:05)
[DEBUG][2/26/2020 6:12:04 PM][Thread 0032][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/replicator] Received get for key shard-Entity-1-3, local value , consistency: ReadMajority(timeout=00:00:05)
[DEBUG][2/26/2020 6:12:04 PM][Thread 0032][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/replicator] Received get for key shard-Entity-1-4, local value , consistency: ReadMajority(timeout=00:00:05)
[DEBUG][2/26/2020 6:12:04 PM][Thread 0012][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/Entity/1] DDataShard recovery completed shard [1] with [0] entities
[DEBUG][2/26/2020 6:12:04 PM][Thread 0034][akka.tcp://PersistentClusterShardingRememberEntitiesSpec@localhost:53675/system/sharding/Entity] Shard was initialized [1]

We execute gets for all of the missing local keys, but since there's no entity registered with that name, it never gets factored into our recovery for remember entities. I think this is a bug - don't believe we're supposed to have a key for every entity, just the key set itself (which we have.)

[Aaronontheweb]

I take that back - the above calculations are part of the routine GetState process for DData shards

[Aaronontheweb]

On node 2, we never call UpdateState for the entities we're supposed to remember.