-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Discovery and clustering #12
Comments
Gossip protocol is definitely the right way to go. As per our discussion, a new node would be provided IP address of an existing node, so it could jump start the protocol. Few questions/requirements:
We'll also have to determine what our communication protocol would be. I looked at grpc.io, but it's focused around protocol buffers. We're using Flatbuffers, which aren't supported. So, I see no big benefit in going that direction. We should stick to the standard net/rpc package provided by Go. Byte array based request and response should be sufficient, because Flatbuffers. |
Btw, server failure detection isn't really required from Gossip protocol. MultiRaft is going to tackle that part. |
No. Each new node will always connect to at least an existing node. And after connection is established, the topology will be shared periodically. So, let's say node
As I said in the channel, this is definitely one advantage of going with
I'd say we should stick with whatever allows for any payload format. Narrowing it to
Node failure is important in terms of cluster membership management. We're not talking about sharding/replication and its consensus here. |
I think you misunderstood my question. My question is specifically regarding the case when you bring up the cluster the first time, so in your example, bringing up A, B, and C together. In that case, there's no previously existing node; so does that mean, we have to do rolling start? Regarding grpc, as I mentioned above, it uses protocol buffers, but we use Flatbuffers. I don't want to mix up the two, unless absolutely needed. Which is why I propose using
I think there's some sort of overlap here between Raft and Gossip, that we should be careful of, design wise. Probably more of a note for me at this point, than for you. |
Here's what I said:
So A is the existing node, while only B and C connect together.
I don't support going with gRPC as well. Neither choosing |
What new technologies would existing solution bring in? How hard would it be to write our own solution with |
Not sure I follow the question, but basically something like
The discovery process should be pretty doable. The membership management and messaging would be tricky. The network tomography should be hard. |
What I mean is, if you don't use either of these for communication, what do you use? That's what I mean when I ask about the new technologies that your solution would bring in. I'm not asking about the functionality the library would bring in -- that's |
As I said above
I just meant that I don't feel like narrowing the messaging transport in a decision right now. Deciding to go with |
I think it seems like writing our own bare bones Gossip impl would give us the right customizability, without any of the unneeded features at this point at the extra cost they might come in. |
Why not Raft Protocol ? https://github.com/hashicorp/raft |
Raft is only meant for consensus, not discovery or cluster management.
|
We're using RAFT only. When a new node wants to join, it can talk to any member of the cluster. That member can convey the master for the entire cluster. The new node then talks to the master, and joins the cluster. This is sufficient for now. |
Co-authored-by: Sunil Mujumdar <sunil@sunil-mac.local>
Co-authored-by: Sunil Mujumdar <sunil@sunil-mac.local>
For clustering to be achieved automatically, a discovery mechanism needs to be implemented. Multicast-based discovery mechanisms are always prefered because it requires zero-configuration, but since most, if not all cloud providers block multicast traffic I propose the adoption of a gossip protocol like the one implemented by
Serf
.By doing so, there's only one requirement in order to form a cluster, at least one node must exist and it must be reachable by any new nodes. Every new node (member) does a full state sync with the existing member and begins gossiping its existence to the cluster.
Gossip allows to keep an eventually-consistent view of the cluster at any given node.
Thoughts?
The text was updated successfully, but these errors were encountered: