[feat] Actor redeployment when cluster mode is enabled

[Tochemey]

The cluster implementation simply relies on a distributed sharded key/value store engine. It handles the rebalance and the sharding mechanism.

• A single instance of an actor is created when cluster mode is enabled. The actor identifier is propagated across the cluster.
• Each node in the cluster has a copy of the created actor identifier and can easily refers to the location of the actor.
• When the node where the given actor is created dies, then the other nodes in the cluster are aware of the event. However, at the moment, Go-Akt is not doing actor redeployment which is the issue we need to address.

It is an issue because that actor cannot be found when its node dies

[Tochemey]

There are two ways to go about this:

• Actor serialization and deserialization: this will require serialize the implemented Actor into some bytes array and pass that data over the wire to be able to deserialize it whenever we need to re-create it.
• Listen to cluster events and recreate actors when there really existed. We don't want to create non existing actors. We know that the given actor reference exist in the cluster even if its hosting node leaves the cluster.

[Tochemey]

https://github.com/Tochemey/goakt?tab=readme-ov-file#cluster

[Tochemey]

I will look into actors' redeployment. This may mean changing the underlying cluster engine and falling back to core raft and co. Any help is welcome. cc @AlexanderTar, @chenxyzl and @civilizeddev.

[Tochemey]

Some thoughts here:

• Add a serializer interface to the Actor interface. That way developers can implement their custom binary serialization using binary encoder and binary decoder. This will help serialize Actor implementation and send them over the wire.
• Add an option to cluster mode to start cluster with a minimum number of nodes (at least three). When that number goes below the threshold then we can trigger a panic that the cluster is unbalanced and request an action.
• When a node leaves the cluster, re-creates its actors whenever they are accessed on other nodes in the cluster. This is feasible because the same actor system is run across the cluster. We can leverage on the rebalance mechanism of Olric. Some references:
  • https://github.com/buraksezer/olric?tab=readme-ov-file#read-repair-on-dmaps
  • https://github.com/buraksezer/olric?tab=readme-ov-file#quorum-based-replica-control
  • https://github.com/buraksezer/olric?tab=readme-ov-file#simple-split-brain-protection

[Tochemey]

I will not be able to work on this at the moment.