You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have multiple async workers (futures) that all want to process data (send over a network connection in this case).
I want to use flume to let them all receive the same set of messages, where flume picks a receiver which then performs the sending over the network.
What I'm wondering is, how fair is the load distributed across multiple receivers? In my case, when the receiver has processed data, the task isn't actually completed yet: Network data is still in-flight and needs to be processed by the other side. Yet the receiver is already capable of taking in new messages and hence will call recv() again. In this case, I want the other receivers to do work next, before the previous receiver gets new work. This would ensure equal load on all network connections.
Therefore, my question is: Does flume distribute fairly? If a receiver gets a message and then calls recv() again, can it get new work before another receiver, which has been blocked on recv() the entire time, got its share of data?
The text was updated successfully, but these errors were encountered:
The current implementation of flume has receivers make claims on the queue, so starvation/unfairness should not occur. Threads will be woken to receive items in the approximate order that they called .recv().
So, I have the following use case:
I have multiple async workers (futures) that all want to process data (send over a network connection in this case).
I want to use flume to let them all receive the same set of messages, where flume picks a receiver which then performs the sending over the network.
What I'm wondering is, how fair is the load distributed across multiple receivers? In my case, when the receiver has processed data, the task isn't actually completed yet: Network data is still in-flight and needs to be processed by the other side. Yet the receiver is already capable of taking in new messages and hence will call recv() again. In this case, I want the other receivers to do work next, before the previous receiver gets new work. This would ensure equal load on all network connections.
Therefore, my question is: Does flume distribute fairly? If a receiver gets a message and then calls recv() again, can it get new work before another receiver, which has been blocked on recv() the entire time, got its share of data?
The text was updated successfully, but these errors were encountered: