Akka.Persistence: need to remove hard-coded Newtonsoft.Json object serializer

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.4.49
Which Akka.NET Modules? Akka.Persistence and all derivatives

Related: #3811

Describe the bug

Per akkadotnet/Akka.Hosting#141

Persistence has its own default System.Object serializer settings here:

akka.persistence.journal-plugin-fallback.serializer = json
akka.persistence.snapshot-store-plugin-fallback.serializer = json

https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka.Persistence/persistence.conf#L178

As far as I can tell, this cannot be overwritten.

I've attempted to override these in both of the following ways:

 public static AkkaConfigurationBuilder AddLegacySerialization(this AkkaConfigurationBuilder builder)
    {
        return builder
            .AddHocon(@"
                akka.persistence.journal-plugin-fallback.serializer = json2
                akka.persistence.snapshot-store-plugin-fallback.serializer = json2
            ", HoconAddMode.Prepend)
        .AddHocon(@$"akka.actor {{
             serializers {{
                 json2 = ""{typeof(JsonSerializerWithManifest).AssemblyQualifiedName}""
             }}
            serialization-identifiers {{
                ""{typeof(JsonSerializerWithManifest).AssemblyQualifiedName}"" = 410
            }}
            serialization-bindings {{
                ""System.Object"" = json2
            }}
        }}", HoconAddMode.Prepend);
    }

I still get Newtonsoft.Json as my serializer, even though I have explicitly overridden all of the appropriate settings. Akka.Remote uses the correct serializer. Persistence does not.

I also tried overriding it programmatically:

 public static AkkaConfigurationBuilder AddLegacySerialization(this AkkaConfigurationBuilder builder)
    {
        return builder.WithCustomSerializer("json2", new[] { typeof(object) },
                system => new JsonSerializerWithManifest(system))
                 .AddHocon(@"
                akka.persistence.journal-plugin-fallback.serializer = json2
                akka.persistence.snapshot-store-plugin-fallback.serializer = json2
            ", HoconAddMode.Prepend)
    }

Also did not work.

Expected behavior

Akka.Persistence should take its directions on serialization from Akka.NET, or at least be overridable using the appropriate settings.

Actual behavior

"Phasers had no effect, captain"

Additional context

I've discussed this before with @Arkatufus and we thought it might be unsafe to make this change - after looking more closely at it I don't think that's the case:

1. All old events serialized with a SerializerId of 1 will still be read using Newtonsoft.Json.
2. Any new events serialized going forward will use the System.Object serializer actually specified by the end-user.

In other words, this isn't a destructive change at all - old data read using old method, new data read using configured method.

[ingted]

Now I got it!!! This is the reason that the input of event adapter is always of Json.Net type...

[Arkatufus]

@Aaronontheweb which persistence plugin did you test this on? You should not override akka.persistence.journal-plugin-fallback.serializer directly, you should override akka.persistence.journal.{plugin-id}.serializer instead

[Aaronontheweb]

Ugh, lol. Needed it on SQL Server.

[ingted]

Hi @Aaronontheweb ,

I found if I try to persist Tagged object, it always return to use Newtonsoft.Json object serializer (id=1) even I configured
"Akka.Persistence.Journal.Tagged, Akka.Persistence, Version=1.5.0.0, Culture=neutral, PublicKeyToken=null" = "FSharpExpr" in serialization-bindings.

(Without tagging, it works very well...)

Is it normal?