Custom Serializer setup not propagating to Persistence

[bauersystems]

Version Information
Akka - v1.4.43
Akka.Hosting - v0.4.3
Akka.Persistence.SqlServer.Hosting - v0.4.3
Akka.Serialization.Hyperion - v1.4.43

Describe the bug
When setting up Hyperion as the default serializer via Akka.Hosting configuration, persistence still uses Json as the default serializer and has to be configured separately to use Hyperion.

To Reproduce
I used a configuration similar to the following:

        services.AddAkka("actor-system", (configurationBuilder, _) =>
        {
            configurationBuilder
                .WithRemoting("localhost", 7910)
                .WithClustering(new ClusterOptions() { Roles = new[] { "Coordinator-Role" }, SeedNodes = Address[] { Address.Parse("akka.tcp://actor-system@localhost:7910") } })
                .WithSqlServerPersistence("Server=localhost,1433; Database=Akka; User Id=sa; Password=[password];")
                .WithCustomSerializer("hyperion", new[] { typeof(object) }, system => new HyperionSerializer(system));
        {

Expected behavior
When setting up Hyperion as the default serializer, this setting should be propagated through-out the Akka.net modules.

Actual behavior
Json was still the default serializer for persistence.

Screenshots

Environment
Windows 11
.Net 6.0

Additional context

[Aaronontheweb]

Thanks @bauersystems - we'll look into this

[Arkatufus]

@bauersystems
Can you tell us what value you get when you run this SQL script on your journal database?

SELECT ej.SerializerId FROM EventJournal ej WHERE ej.PersistenceId = 'client_bauer-systems' AND ej.SequenceNr = 1

[Arkatufus]

The proper SerializerId for Hyperion is -5, the SerializerId for the default JSON serializer is 1.

[Arkatufus]

@bauersystems
It appears that this is a big design flaw with the Akka.Persistence package that never got found until today, I think we might not be able to fix it because it will break a lot of existing implementations.

The workaround is to add these custom HOCON configuration in:

builder
    .AddHocon(@"
akka.persistence.journal.sql-server.serializer = hyperion
akka.persistence.snapshot-store.sql-server.serializer = hyperion", HoconAddMode.Prepend))

These settings are a bit obscure, it overrides the object type serializer for that specific journal/snapshot store configuration.

[Arkatufus]

You can also override the default behaviour by using these HOCON settings:

builder
    .AddHocon(@"
akka.persistence.journal-plugin-fallback.serializer = null
akka.persistence.snapshot-store-plugin-fallback.serializer = null
", HoconAddMode.Prepend)

[ismaelhamed]

@bauersystems It appears that this is a big design flaw with the Akka.Persistence package that never got found until today, I think we might not be able to fix it because it will break a lot of existing implementations.

Actually, this is rather old: akkadotnet/akka.net#3133 (comment)

[Arkatufus]

Changing this would potentially break and/or change the behavior of systems that is already in the wild since it requires changes to be made on Akka.NET core. We will leave this as is.

[Arkatufus]

@Aaronontheweb Maybe we should add a new extension to Akka.Hosting that specifically target this?
Something like .WithDefaultPersistenceSerializer()?
We also need to add this specific peculiarity to the documentation.

[Aaronontheweb]

@bauersystems @Arkatufus I'm thinking this is a bug with Akka.Persistence and it needs to be fixed akkadotnet/akka.net#6389