Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEW] Migrate non topology update cluster bus messages to light header type #932

Open
hpatro opened this issue Aug 21, 2024 · 8 comments
Open

Comments

@hpatro
Copy link
Contributor

hpatro commented Aug 21, 2024

With the light message header for cluster bus in place, we can migrate the following messages to use light header type. The following messages can be migrated:

  1. CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST
  2. CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK
  3. CLUSTERMSG_TYPE_UPDATE
  4. CLUSTERMSG_TYPE_MFSTART
  5. CLUSTERMSG_TYPE_MODULE

This should only be used when the cluster is in homogenous state and all of the nodes support the light message header.

@hpatro
Copy link
Contributor Author

hpatro commented Aug 21, 2024

@roshkhatri / @zuiderkwast WDYT ?

@madolson
Copy link
Member

CLUSTERMSG_TYPE_UPDATE

I kind of think this one should stay. We're indicating state changes, it seems prudent to send our state of the world as well.

@zuiderkwast
Copy link
Contributor

zuiderkwast commented Aug 22, 2024

I believe the messages listed here are rare and don't make up a large percentage of the cluster bus traffic, so I don't think we need to prioritize these. The bulk of the cluster bus traffic is PING and PONG. If we can figure out a way to use light ping pong 🏓, then we're saving a lot of cluster bus overhead for large clusters.

A basic idea is that each node remembers what it sent to any other node (for example a hash of the content) and if it hasn't changed since the last ping/pong between the nodes, then the node can send a light PING or PONG, just as a keepalive message and to say that nothing has changed. Do you want to explore this idea?

@madolson
Copy link
Member

The bulk of the cluster bus traffic is PING and PONG.

This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.

@roshkhatri
Copy link
Member

roshkhatri commented Aug 22, 2024

At a high level, it does seem like a nice idea!
However for,

  1. CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST we do require currentEpoch, configEpoch and myslots.
  2. 'CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST' would require mflags

This means we will be adding new field to the clusterMsgLight which will make it a bit large again or add these in some other way in the data field.

@zuiderkwast's idea is what we had discussed some time ago and I do agree with that too and would like to explore.

This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.

Yeah this would he the best gain stating that I am alive and no changes since last time.

@zuiderkwast
Copy link
Contributor

The bulk of the cluster bus traffic is PING and PONG.

This biggest gain might be to send a message that says, "I have no new slot information changes from last time", which I think is the biggest part of the message (That's like 2kb right)? One problem will be that it'll still get rounded up to a full TCP packet.

@madolson Yes, the slot bitmap is 2KB (16K bits). A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?

Anyway, the light packet doesn't solve our scaling problem. The problem is still the all-to-all communication. In a raft cluster, nodes only communicate with the leader, except when the leader isn't reachable anymore. To focus on that might be a better idea.

@roshkhatri
Copy link
Member

A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?

This will be a constant for every msg, right? The thing we can focus on is how we can reduce our overhead for every msg.

@zuiderkwast
Copy link
Contributor

A TCP packet has 20 bytes overhead and an IPv4 header is another 20 bytes overhead. Then, Ethernet has somewhere around 128 bytes overhead I believe (source and target MAC addresses and some CRC-sum). We're at somewhere around 200 bytes in total. On top of this, TLS has some minimup packet size and VPNs and other stuff probably add more overhead. Did I miss anything?

This will be a constant for every msg, right? The thing we can focus on is how we can reduce our overhead for every msg.

Yes, the overhead per message is constant, but it's important to discuss it. It makes little sense to reduce a small message to even smaller (say from 20 to 10 bytes) because we would only save very little compare to all the overhead. If we can reduce a message from 2K to 400 including all protocol overhead, we save 80% so this is good.

We still have very many messages though and this will still be a bottleneck, because the number of ping-pong messages in a cluster grows exponentially with the size of the cluster (right?). If we save 80% of the size of each message, we can maybe achieve a 600 nodes cluster instead of 500 nodes, but if we can reduce the number of message to just grow linearly or quadratically with the size of the cluster, then we can scale to huge clusters for real. We could introduce features like non-voting nodes, and less active nodes that don't ping anyone but they just answer pings from other nodes. Or... we take the step to use a raft protocol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants