Akka.Cluster.Tools.Singleton: singleton moves as soon as node with higher AppVersion joins cluster?

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.0
Which Akka.NET Modules? Akka.Cluster.Tools

Describe the bug

Chasing down and issue for a production support customer - they have a custom pbm command for being able to track the location of cluster singletons. They confirmed the singleton was on a specific node and decided to replace that one last during a version upgrade. What they observed was: the singleton moved onto the newest node with the highest AppVersion even before that oldest node was downed!

Expected behavior

As I wrote back to the customer originally, the singleton should only move onto a new node AFTER the node it's currently on begins to leave the cluster. This leads me to believe that the following code might have a bug in how we compute the sort order for who the most suitable location is for a singleton:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/OldestChangedBuffer.cs

Lines 98 to 112 in 3f0be58

	/// <summary>
	/// Creates a new instance of the <see cref="OldestChangedBuffer"/>.
	/// </summary>
	/// <param name="role">The role for which we're watching for membership changes.</param>
	/// <param name="considerAppVersion">Should cluster AppVersion be considered when sorting member age</param>
	public OldestChangedBuffer(string role, bool considerAppVersion)
	{
	_role = role;
	_memberAgeComparer = considerAppVersion
	? MemberAgeOrdering.DescendingWithAppVersion
	: MemberAgeOrdering.Descending;
	_membersByAge = ImmutableSortedSet<Member>.Empty.WithComparer(_memberAgeComparer);
	SetupCoordinatedShutdown();
	}

In fact, I'm almost certain that this is the case.

[Aaronontheweb]

Marking this bug as critical - one of the major side effects from this issue is that we can create split brains with all cluster singletons during deployments when the AppVersion is getting bumped. That can result in problems such as #6973

[Aaronontheweb]

So this bug likely affected less people than I initially thought as

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/reference.conf

Line 49 in d1ed226

consider-app-version = false

Has been set to false this whole time and that's also the default value from the HOCON extractors when this configuration isn't available. That's good news, but it still needed to be fixed.

[Aaronontheweb]

Looks like the original issue reported by the end user wasn't even caused by the AppVersion, but this feature is definitely a footgun and probably needs to be removed.