[server] Offload NotifyLeaderAndIsr to dedicated single-thread executor#3647
Conversation
|
Could you help review this? @fresh-borzoni |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@zuston Thank you, makes sense overall, left a suggestion, PTAL
| notifyLeaderAndIsrRequestData, | ||
| result -> response.complete(makeNotifyLeaderAndIsrResponse(result))); | ||
| try { | ||
| replicaStateChangeExecutor.execute( |
There was a problem hiding this comment.
Should stopReplicas go through this executor too?
It takes the same replicaStateChangeLock but still runs on the RPC worker, so a stop can now overtake queued notifies, which the per-connection ordering used to prevent.
Same-executor for both would fix the ordering, and also keep stops from blocking an RPC worker on the lock while a slow transition runs.
WDYT?
There was a problem hiding this comment.
nice suggestion. let me add the remaining rpcs into this dedicated executor.
|
updated. @fresh-borzoni |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@zuston Thank you, LGTM 👍
2 participants
Purpose
Linked issue: close #3646
NotifyLeaderAndIsris currently processed on the RPC worker thread. The underlyingbecomeLeaderOrFolloweroperation may perform relatively expensive replica state transitions, such as initializing local replicas and updating replica fetchers. A slow transition can therefore occupy an RPC worker thread and delay unrelated RPC requests.Brief change log
Introduce a dedicated single-thread executor for
NotifyLeaderAndIsroperations.The executor is created and managed by
TabletServer, injected intoTabletService, and gracefully shut down before replica-related resources are closed.TabletServiceparses the request on the RPC worker thread and submits the replica state transition to the dedicated executor.A single thread is sufficient because replica state changes are already serialized by the replica state change lock. It also preserves the execution order of
NotifyLeaderAndIsrrequests while preventing slow transitions from blocking other RPCs.The RPC semantics remain unchanged: the response future is completed only after
becomeLeaderOrFollowerfinishes.Tests
API and Format
Documentation