Akka.Tools.ClusterClient: ClusterClient.Send and ClusterClient.SendToAll aren't serialized using ClusterClientSerializer

[Aaronontheweb]

Version Information
Version of Akka.NET? 1.5.8 and previous
Which Akka.NET Modules?

Describe the bug

ClusterClient.Send and ClusterClient.SendToAll aren't serialized using ClusterClientSerializer - they are instead serialized using the System.Object serializer. This means that if a user defines a custom serializer for messages they wish to send over ClusterClient, none of the messages sent will use that serializer.

Reproduction

Will submit one via PR shortly.

Expected behavior

User-defined messages should be serialized using their own custom serializer, when specified, not overridden by a user-defined serializer

Actual behavior

Newtonsoft.Json serializer is being used instead.

Additional context

Since this involves making a change to the wire format (one of the no-no's outlined in our developer guidelines), we're going to need to add a feature flag to make this backwards compatible and enable that by default:

akka.cluster.client.use-legacy-serialization = on

Users who want to opt-in to the new behavior will be able to do-so by turning off that setting.

[Arkatufus]

The problem is that Send, SendToAll, and Publish is not a cluster client internal message, it is instead a pub-sub message.
We can not change the serializer for these messages because it is tied directly to pub-sub.

[Aaronontheweb]

The problem is that Send, SendToAll, and Publish is not a cluster client internal message, it is instead a pub-sub message. We can not change the serializer for these messages because it is tied directly to pub-sub.

In that case we still have a fixable issue: wrong serializer is getting used for these.

[Arkatufus]

Never mind, there are pairs of Send, SendToAll and Publish classes, one inside pub-sub and one inside cluster client.
When ClusterClient receives its own version of Send or SendToAll, it is converted to the pub-sub version of Send and SendToAll and then forwarded to the pub-sub mediator.
I think the version that does not get serialized properly is the cluster client version of these classes.