Use alternative Distribution Protocol

[sleipnir]

In order to achieve our goals of offering a way to reach the edges in a local-first approach, it is necessary that we offer an alternative to the old Erlang Distribution Protocol, because in environments that are really distributed in a fabric that deviates from the basic requirements of a datacenter or public private cloud if requires a protocol that is easily routable over the Internet or over unreliable connections.

I'm inclined to test quic_dist as an alternative to Erlang Distributed, but if anyone has another proposal feel free to present it.

For this to work, it is very important that the alternative protocol chosen is easily pluggable with Elixir and Libcluster.

[sleipnir]

I decided to turn the previous Issue into a discussion as I find it more appropriate to use this channel for this type of subject.
That said, I've found yet another good candidate for a distributed protocol and am experimenting with new ideas in the old eigr-labs/steinadler repository. It is about using the Nats broker as a basis for a distributed protocol. Nats has many interesting features mainly in the areas of security and topology and therefore I think it is a good candidate. However, an integration with it through Steinadler would result in something more invasive and would require more changes in the Spawn code base since Steinadler is a higher level abstraction than Erlang Distributed or quic_dist.
One of the benefits of using Steinadler via Nats is that in theory we would be able to support really large clusters without the cost of a mesh network where all nodes involved need to be directly connected to each other. Each Node would only need a single direct connection to the Nats cluster, which reduces the number of necessary connections and obviously increases the number of possible nodes to be connected. Depending on the implementation of Steinadler, it would also reduce the overhead in registering the processes and consequently improve the lookup of the processes.

[heri16]

Sounds like a single point of failure. Prefer hubs

[sleipnir]

Hi @heri16, glad to see someone moving around here.

Yes, there are tradeoffs, and all models will have their tradeoffs. You can operate on a geographically provisioned Nats cluster which would remove your single point of failure.

A fully connected network like the standard Erlang Dist is very convenient, but it comes with a very high cost in TCP connections and auto discovery is a very expensive operation for large clusters.

The idea of having more than one mode of operation is precisely to give freedom of choice to anyone using Spawn.