Cluster.Tools.Client.ClusterReceptionist: An entry with the same key already exists.

[cgstevens]

Version 1.3
After having my member .Terminate and shutdown and then come back up when it joins the cluster ALL the other members log the following error.

[akka://mysystem/system/receptionist] - An entry with the same key already exists.

System.ArgumentException: An entry with the same key already exists.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.TreeSet1.AddIfNotPresent(T item) at System.Collections.Generic.SortedDictionary2.Add(TKey key, TValue value)
at Akka.Routing.ConsistentHash.Create[T](IEnumerable`1 nodes, Int32 virtualNodesFactor)
at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[Horusiath]

@cgstevens as I understand shuting down a node gets acknowledged by other nodes by calling it down? Or does it remaing unreachable waiting for reconnection to happen?

It looks like either we could forgot removing an entry somewhere on downing a node event or we should use indexer setter instead of Add method on SortedDictionary.

[cgstevens]

@Horusiath I don't have that answer as I the logs didn't say with running Info.
I am currently setting up a new Dev environment and have not had a chance to confirm this yet.

[mwpro]

I think, that I've encountered similar issue. I'm running a Akka.NET cluster on Kubernetes with Cluster Receptionist and a second application running Cluster Client. After I deploy a new version of cluster application (it's important that since we're using K8s, the nodes are started on random IP/port), the cluster receptionist happens to return contact points to old nodes that already left the cluster to the client.

I've tried to reproduce the issue locally by adding some extra logging to the ClusterReceptionist code and I've found out that Receptionist is handling the MemberRemoved event correctly but ImmutableSortedSet with RingOrder comperer used beneath does not always handle removing nodes correctly.

Issue reproduction:

• https://github.com/mwpro/akka.net/blob/ClusterReceptionistIssue/src/examples/GP/ReceptionistIssue/Program.cs - simple program that show cluster receptionist not having nodes removed from it's internals. It also reproduces "An entry with the same key already exists" exception which seems to be the @cgstevens case.
• https://github.com/mwpro/akka.net/blob/ClusterReceptionistIssue/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/RingOrderingTests.cs - unit tests for RingOrdering that I believe should be passing but are not

Akka.NET version: 1.3.11 and dev branch

Platform: Happens both on Windows dev environment and Docker container running on Kubernetes.

[Aaronontheweb]

@mwpro thanks for reporting this - we'll take a closer look at it.

[Aaronontheweb]

Looks like the issue is here:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Client/ClusterReceptionist.cs

Lines 326 to 333 in cf0cb8c

	public int Compare(Address x, Address y)
	{
	var ha = HashFor(x);
	var hb = HashFor(y);
	if (ha == hb) return 0;
	return ha < hb || Member.AddressOrdering.Compare(x, y) < 0 ? -1 : 1;
	}

This CompareTo function is, for lack of a better word, useless. Fails the transitive property of equality. The function's basic problem is that if hash of addr1 is lower than hash of addr2, the value from the first half of the equation gets used. But if I reverse the call, the second half of the equation gets used - the Address.CompareTo method. There's no guarantee that the hash value and the member address value are both going to sort in the same order.

I rewrote the method to use the following code:

 public int Compare(Address x, Address y)
            {
                var ha = HashFor(x);
                var hb = HashFor(y);

                if (ha == hb) return Member.AddressOrdering.Compare(x, y);
                return ha.CompareTo(hb);
            }

The Member.AddressOrdering only gets called in the rare event that the hashes are equal. Otherwise, we simply return the comparison values of the hashes themselves. All of the tests written by @mwpro now pass as a result.

[Aaronontheweb]

Just wanted to point something out:

It looks like either we could forgot removing an entry somewhere on downing a node event or we should use indexer setter instead of Add method on SortedDictionary.

This would have made the problem worse - the reason why there's an error in the first place inside the ConsistentHash.Create method is because we passed in a range of duplicate items from a SortedSet<T> as a result of the bug in the IComparer - papering over that with a hack would have made the underlying problem more difficult to find.