[CELEBORN-2217] Use a separate thread to handle RpcEndpointVerifier messages#3554
[CELEBORN-2217] Use a separate thread to handle RpcEndpointVerifier messages#3554TheodoreLx wants to merge 3 commits intoapache:mainfrom
Conversation
| buildConf("celeborn.rpc.RpcEndpointVerifier.separate.enabled") | ||
| .categories("network") | ||
| .version("0.7.0") | ||
| .doc("dispatcher will process RpcEndpointVerifier's request separately") |
There was a problem hiding this comment.
| .doc("dispatcher will process RpcEndpointVerifier's request separately") | |
| .doc("Whether to enable dispatcher process RpcEndpointVerifier's request separately.") |
| } | ||
|
|
||
| /** Message loop used for dispatching messages. */ | ||
| private class EndpointVerifierMessageLoop extends Runnable { |
There was a problem hiding this comment.
Could you add some test case to verify?
|
This pr look interesting to me. Can you share some test results for this PR? For example, are there any changes in the number of rpc look up timeout exception? |
This is a screenshot of the driver logs from a previous task. If the sum of the In this image, because many RPC messages were waiting in the queue, the After adopting this pull request, we no longer found |
|
Thanks, Merge to main(v0.7.0) |
4 participants
What changes were proposed in this pull request?
An EndpointVerifierMessageLoop is introduced to specifically handle RpcEndpointVerifier messages, and a separate thread is used to execute the EndpointVerifierMessageLoop.
Why are the changes needed?
RpcEndpointVerifier.CheckExistence is a high-priority message that needs to be processed first. In the original model, if the LifecycleManager's RPC message queue accumulates a large backlog, RpcEndpointVerifier.CheckExistence messages cannot be processed immediately, leading to numerous ShuffleClient initialization failures and causing task failures.
Does this PR resolve a correctness bug?
No
Does this PR introduce any user-facing change?
No
How was this patch tested?
Cluster Test