New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lots of CPU time spent sorting broadcast bytes #11
Comments
Hah I half expected this to come up 😬 The problem is that custom broadcasts use the same mechanism used for cluster broadcasts: it's a pretty dumb sorted array. SWIM can get by with this inefficient thing because broadcasts are bound by the cluster size and are tiny (it's just cluster updates, and there's invalidation member identity). When I was hacking on it what I wanted was a binary heap with retain capabilities, but didn't want to jump on nightly and was too lazy to roll my own, so I ended up with what we have now heh One way forward is making Broadcasts better. The api surface is small and requirements clear:
Another would be to disconnect the custom broadcasts from the main interaction (I think I mentioned this one before). First alternative is a lot easier I think. I don't think I'll be looking at this until at least the holidays season in December but I'm happy to receive/review pull-requests if you wanna take a stab at it 😁 |
That makes sense. I might've noticed this wasn't a problem when I was using larger max payload size, transmitted via TCP. Would that have made a difference? For reference: my cluster size is now ~300 nodes.
This might be easier on our end! If I didn't use the Broadcast stuff from
All good. I just might take a stab at it yes! I will have more questions if I do that. |
I ended up doing my own broadcasts, handling messages with logic similar to the That solved my CPU issues! |
Sweet! I think I over-promised a bit on the general purpose broadcasting feature 😅 I think when optimizing these sort of broadcasts you might end up wanting to do some fancier way of prioritizing which packets to send first, maybe even some way of feeding back information about the current known global state so you can prune/reorder the queue... It's why I think the way to go is disconnecting broadcasts from foca and just exposing some functionality to enable the use case. From the top of my head, I could expose choose_active_members (without the unneeded |
Yeah, we've ended up implementing some logics for some broadcast messages to go out immediately instead of buffering a few of them. Did |
That's right, foca doesn't try to do anything smart. It simply buffers a broadcast up and tries to send it |
closing stale issues that seem resolved. feel free to reopen |
I'm actually not sure where the time is being spent, but most of the CPU time in my project is currently being used sorting something related to broadcast messages.
Flamegraph: corro.svg.zip
Looks like it goes like this:
Foca::broadcast
Foca::send_message
Broadcasts::fill
core::slice::sort::recurse
core::slice::sort::partial_insertion_sort
PartialOrd::lt
I have a loop that calls
Foca::broadcast
every 200ms to ensure maximum gossip propagation of broadcast messages.The text was updated successfully, but these errors were encountered: