Failed to deserialize payload object when deserializing ActorSelectionMessage with payload ...

[fscavo]

Version Information
Version of Akka.NET? 1.4.24
Which Akka.NET Modules? Akka.Cluster

Describe the bug
Our services throws SerizalizationException when sending a message to a singleton actor.

[Aaronontheweb]

Do you have a copy of the serialization exception error message that was thrown? This would be very helpful to diagnose this error.

[fscavo]

This is the full message

Failed to deserialize payload object when deserializing ActorSelectionMessage with payload [SerializerId=14, Manifest=A] addressed to [user,cache-abb-advices-found] Cannot find serializer with id [14] (manifest [A]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems

And here is the call stack

System.Runtime.Serialization.SerializationException:
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Serialization.Serialization.Deserialize (Akka, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Remote.MessageSerializer.Deserialize (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Remote.DefaultMessageDispatcher.Dispatch (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Remote.EndpointReader.b__11_0 (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Runtime.Serialization.SerializationException handled at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary:
at Akka.Serialization.Serialization.Deserialize (Akka, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Remote.Serialization.WrappedPayloadSupport.PayloadFrom (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary (Akka.Remote, Version=1.4.24.0, Culture=neutral, PublicKeyToken=null)

[Aaronontheweb]

If you try doing the following:

yourConfig.WithFallback(ClusterSingletonManager.DefaultConfig())

I think that should resolve this issue. This is something I want to fix with automatic loading in the future.

[fscavo]

@Aaronontheweb Thanks!
May I ask you to elaborate on this issue?

[fscavo]

I thought it might be helpful to post here the configuration

var defaultAkkaConfiguration = ConfigurationFactory.ParseString($@"
                akka {{
                    log-config-on-start = on
                    loglevel = {akkaSettings.AkkaLogLevel},
                    loggers  = [ ""Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"" ]
                    suppress-json-serializer-warning = on
                    actor {{
                        ask-timeout = 15s
                        serializers {{
                            hyperion = ""Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion""
                        }}
                        serialization-bindings {{
                            ""System.Object"" = hyperion
                        }}
                        serialize-messages = {(akkaSettings.ForceMessagesSerialization ? "on" : "off")}
                        serialize-creators = {(akkaSettings.ForceCreatorsSerialization ? "on" : "off")}
                    }}
                }}");

var akkaPersistenceConfiguration = ConfigurationFactory.ParseString($@"
                akka.persistence {{
                    journal {{
                        plugin = ""akka.persistence.journal.sql-server""
                        sql-server {{
                            class = ""Akka.Persistence.SqlServer.Journal.{(akkaSettings.Persistence.Journal.UseBatchingJournal ? "Batching" : string.Empty)}SqlServerJournal, Akka.Persistence.SqlServer""
                            schema-name = dbo
                            auto-initialize = on
                            connection-string = ""{GetDbConnectionString()}""
                            metadata-table-name = AkkaPersistenceMetadata
                            table-name = AkkaPersistenceEventJournal
                            circuit-breaker {{
                                call-timeout = 50s
                            }}
                            event-adapters = {{
                                {GetConfigFromDictionary(akkaSettings.Persistence.Journal.EventAdapters)}
                            }}
                            event-adapter-bindings = {{
                                {GetConfigFromDictionary(akkaSettings.Persistence.Journal.EventAdapterBindings)}
                            }}
                        }}
                    }}
                    snapshot-store {{
                        plugin = ""akka.persistence.snapshot-store.sql-server""
                        sql-server {{
                            class = ""Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer""
                            schema-name = dbo
                            auto-initialize = on
                            connection-string = ""{GetDbConnectionString()}""
                            table-name = AkkaPersistenceSnapshotStore
                            circuit-breaker {{
                                call-timeout = 50s
                            }}
                        }}
                    }}
                    query {{
                        journal {{
                            sql-server {{
                                class = ""Akka.Persistence.Query.Sql.SqlReadJournalProvider, Akka.Persistence.Query.Sql""
                                max-buffer-size = {akkaSettings.Persistence.Query.MaxBufferSize}
                                refresh-interval = {akkaSettings.Persistence.Query.RefreshInterval}
                            }}
                        }}
                    }}
                }}");

private string GetClusterConfig()
        {
            var clusterInfo = clusterResolver.GetClusterInfo();
            var nodes = string.Join(",", clusterInfo.Seeds.Select(t => $@"""akka.tcp://{platformSettings.Name}@{t.Host}:{t.Port}"""));
            return $@"
            akka {{
                actor.provider = cluster
                extensions = [""Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider,Akka.Cluster.Tools""]
                remote {{
                    dot-netty.tcp {{
                        public-hostname = {clusterInfo.CurrentHost.Host}
                        public-port = {clusterInfo.CurrentHost.Port}
                        hostname = 0.0.0.0
                        port = {platformSettings.ClusterPort}
                        maximum-frame-size = 2MB
                        send-buffer-size = 2MB
                        receive-buffer-size = 2MB
                    }}
                }}
                cluster {{
                    seed-nodes = [{nodes}]
                    downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
                    down-removal-margin = 20s # Keep the same as stable-after
                    split-brain-resolver {{
                        active-strategy = keep-majority
                        stable-after = 20s
                        down-all-when-unstable = off
                    }}
                    failure-detector {{
                        acceptable-heartbeat-pause = 15s
                    }}
                    min-nr-of-members = {akkaSettings.MinNumberOfNodesInCluster}
                    sharding {{
                        state-store-mode = persistence
                        least-shard-allocation-strategy {{
                            rebalance-threshold = 2
                            max-simultaneous-rebalance = 3
                        }}
                    }}
                    log-info = on
                    log-info-verbose = off
                    pub-sub = {{
                        role = ""service""
                    }}
                    roles = [""service""]
                    role.[""service""].min-nr-of-members = {akkaSettings.MinNumberOfNodesInCluster}
                    singleton = {{
                        role = ""service""
                    }}
                    singleton-proxy = {{
                        role = ""service""
                    }}
                }}
            }}";
        }

var config = configurationProviders
                .OrderByDescending(c => c.Priority)
                .Where(c => c.Get() != null)
                .Aggregate(Config.Empty, (config, provider) =>
                {
                    logger.LogInformation("Using additional Akka configuration from {Provider}: {Config}", provider.GetType().Name, provider.Get().ToString());
                    return config.WithFallback(provider.Get());
                })
                .WithFallback(GetDispatcherConfig());

            if (akkaSettings.EnableCluster)
            {
                config = config.WithFallback(ConfigurationFactory.ParseString(GetClusterConfig()));
            }

            return config
                .WithFallback(akkaPersistenceConfiguration)
                .WithFallback(defaultAkkaConfiguration);

[Aaronontheweb]

So the issue here is that unless this method gets called:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/ClusterSingletonManagerSettings.cs

Lines 28 to 37 in e27220a

	public static ClusterSingletonManagerSettings Create(ActorSystem system)
	{
	system.Settings.InjectTopLevelFallback(ClusterSingletonManager.DefaultConfig());
	var config = system.Settings.Config.GetConfig("akka.cluster.singleton");
	if (config.IsNullOrEmpty())
	throw ConfigurationException.NullOrEmptyConfig<ClusterSingletonManagerSettings>("akka.cluster.singleton");
	return Create(config).WithRemovalMargin(Cluster.Get(system).DowningProvider.DownRemovalMargin);
	}

By default the Akka.Cluster.Singleton serializer is not loaded, which you can see is serializerId=14 here:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/reference.conf

Lines 72 to 81 in e27220a

	akka.actor {
	serializers {
	akka-singleton = "Akka.Cluster.Tools.Singleton.Serialization.ClusterSingletonMessageSerializer, Akka.Cluster.Tools"
	}
	serialization-bindings {
	"Akka.Cluster.Tools.Singleton.IClusterSingletonMessage, Akka.Cluster.Tools" = akka-singleton
	}
	serialization-identifiers {
	"Akka.Cluster.Tools.Singleton.Serialization.ClusterSingletonMessageSerializer, Akka.Cluster.Tools" = 14
	}

So in those cases, to solve these serialization errors, you need to manually load the ClusterSingletonManager.DefaultConfig() to make sure those serialization bindings are available to Akka.Remote.

[fscavo]

I think this has solved the issue.
Thanks @Aaronontheweb

[Aaronontheweb]

Tagging this on akkadotnet/HOCON#48