Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClusterReceptionist, RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. #3840

Closed
hhko opened this issue Jul 7, 2019 · 4 comments

Comments

@hhko
Copy link

commented Jul 7, 2019

  • Akka 1.3.13

After #2535 issue occurs, RingOrdering throws an exception(IComparer.Compare) repeatedly.
As a result, The 'Akka.Cluster.Tools.Client.ClusterReceptionist' is continuously restarting and Memory is increasing up to 20G.

[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 

[2019/07/01 16:24:14.384][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] An entry with the same key already exists. System.ArgumentException: An entry with the same key already exists.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
   at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
   at Akka.Routing.ConsistentHash.Create[T](IEnumerable`1 nodes, Int32 virtualNodesFactor)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

@hhko hhko changed the title Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. ClusterReceptionist, RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. Jul 7, 2019

@Aaronontheweb

This comment has been minimized.

Copy link
Member

commented Jul 7, 2019

@hhko thanks! We'll look into this.

@Aaronontheweb Aaronontheweb self-assigned this Jul 7, 2019

@kimbyungeun

This comment has been minimized.

Copy link

commented Jul 7, 2019

Me and @hhko is co-worker.

Let me explain about our environment.

  1. We operate many services with Akka.Cluster.
  2. Use Custer Singleton for Actor distribution.
  3. Use the Cluster Client to send messages outside the cluster.

There is an additional explanation for the error.
[2019/07/01 16:24:14.404] More than 26 error messages were output at the same time.
Most services consumed more than 20GB of memory as the same errors were output continuously.

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mls-cluster-system/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mls-cluster-system/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 
[2019/07/01 16:24:14.404][  212][ERROR]akka://mls-cluster-system/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
@Aaronontheweb

This comment has been minimized.

Copy link
Member

commented Jul 22, 2019

Thanks folks - I'm going to get to work on fixing this soon. Flying back from India to the USA at the moment, but we'll take care of this. Sorry for not following up sooner - been traveling continuously since this issue was filed.

@Aaronontheweb

This comment has been minimized.

Copy link
Member

commented Jul 25, 2019

Figured this out in #2535 - it's the comparer that's bad. Working on a multi-node test to verify the fix and to ensure that old nodes properly get pruned out of the state.

Aaronontheweb added a commit that referenced this issue Jul 26, 2019

ClusterClient fixes (#3866)
Fixed broker `IComparer` for ClusterClient hash ring and ported over other handoff fixes.

Close #2535
Close #2312
Close #3840

* implemented akka/akka#24167

* implemented akka/akka#22992

@Aaronontheweb Aaronontheweb added this to the 1.4.0 milestone Jul 26, 2019

Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this issue Jul 26, 2019

ClusterClient fixes (akkadotnet#3866)
Fixed broker `IComparer` for ClusterClient hash ring and ported over other handoff fixes.

Close akkadotnet#2535
Close akkadotnet#2312
Close akkadotnet#3840

* implemented akka/akka#24167

* implemented akka/akka#22992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.