[release-1.12] wait event handler completed before start scheduling#5201
[release-1.12] wait event handler completed before start scheduling#5201JesseStutler merged 2 commits intovolcano-sh:release-1.12from
Conversation
Signed-off-by: qi-min <qim_34@163.com>
Signed-off-by: qi-min <qim_34@163.com>
There was a problem hiding this comment.
Code Review
This pull request implements a synchronization mechanism to ensure the Volcano scheduler waits for initial resource data to be fully processed before starting the scheduling cycle. Key changes include the addition of a ResourceSyncTimeout configuration, the introduction of an InitialEventAsyncHandlerTracker to monitor synchronization status, and the transition of internal queues to use a QueueObjectWrapper. The review feedback identifies several opportunities to improve the robustness of this implementation, specifically by adding nil checks for the new trackers to prevent potential panics during initialization and ensuring that initial events are only marked as complete in the tracker if the synchronization logic succeeds without errors.
| for _, handler := range sc.registeredHandlers { | ||
| if !handler.HasSynced() { | ||
| return false, nil | ||
| } | ||
| } |
There was a problem hiding this comment.
Add a nil check for handler before calling HasSynced() to avoid potential panics if a handler registration failed during initialization.
| for _, handler := range sc.registeredHandlers { | |
| if !handler.HasSynced() { | |
| return false, nil | |
| } | |
| } | |
| for _, handler := range sc.registeredHandlers { | |
| if handler != nil && !handler.HasSynced() { | |
| return false, nil | |
| } | |
| } |
| for name, handler := range sc.registeredHandlers { | ||
| if !handler.HasSynced() { | ||
| klog.Errorf("%s handler synchronization is not completed", name) | ||
| } | ||
| } |
There was a problem hiding this comment.
Add a nil check for handler before calling HasSynced() in the error logging loop to prevent potential panics.
| for name, handler := range sc.registeredHandlers { | |
| if !handler.HasSynced() { | |
| klog.Errorf("%s handler synchronization is not completed", name) | |
| } | |
| } | |
| for name, handler := range sc.registeredHandlers { | |
| if handler != nil && !handler.HasSynced() { | |
| klog.Errorf("%s handler synchronization is not completed", name) | |
| } | |
| } |
| err := sc.SyncNode(nodeWrapper.Object) | ||
| if nodeWrapper.IsInInitialList { | ||
| sc.nodeInitialEventTracker.Done(nodeWrapper.Object) | ||
| } | ||
| if err == nil { | ||
| sc.nodeQueue.Forget(nodeName) | ||
| sc.nodeQueue.Forget(nodeWrapper) | ||
| return true | ||
| } |
There was a problem hiding this comment.
Move the Done call after the error check to ensure the initial sync actually succeeded before marking it as complete in the tracker. Also, add a nil check for sc.nodeInitialEventTracker to prevent potential panics.
| err := sc.SyncNode(nodeWrapper.Object) | |
| if nodeWrapper.IsInInitialList { | |
| sc.nodeInitialEventTracker.Done(nodeWrapper.Object) | |
| } | |
| if err == nil { | |
| sc.nodeQueue.Forget(nodeName) | |
| sc.nodeQueue.Forget(nodeWrapper) | |
| return true | |
| } | |
| err := sc.SyncNode(nodeWrapper.Object) | |
| if err == nil { | |
| if nodeWrapper.IsInInitialList && sc.nodeInitialEventTracker != nil { | |
| sc.nodeInitialEventTracker.Done(nodeWrapper.Object) | |
| } | |
| sc.nodeQueue.Forget(nodeWrapper) | |
| return true | |
| } |
| err := sc.SyncHyperNode(hnWrapper.Object) | ||
| if hnWrapper.IsInInitialList { | ||
| sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object) | ||
| } | ||
| if err == nil { | ||
| sc.hyperNodesQueue.Forget(name) | ||
| sc.hyperNodesQueue.Forget(hnWrapper) | ||
| return true | ||
| } |
There was a problem hiding this comment.
Move the Done call after the error check to ensure the initial sync actually succeeded before marking it as complete in the tracker. Also, add a nil check for sc.hyperNodesInitialEventTracker to prevent potential panics.
| err := sc.SyncHyperNode(hnWrapper.Object) | |
| if hnWrapper.IsInInitialList { | |
| sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object) | |
| } | |
| if err == nil { | |
| sc.hyperNodesQueue.Forget(name) | |
| sc.hyperNodesQueue.Forget(hnWrapper) | |
| return true | |
| } | |
| err := sc.SyncHyperNode(hnWrapper.Object) | |
| if err == nil { | |
| if hnWrapper.IsInInitialList && sc.hyperNodesInitialEventTracker != nil { | |
| sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object) | |
| } | |
| sc.hyperNodesQueue.Forget(hnWrapper) | |
| return true | |
| } |
| if isInInitialList { | ||
| sc.nodeInitialEventTracker.Add(node.Name) | ||
| sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name) | ||
| } |
There was a problem hiding this comment.
Add nil checks for sc.nodeInitialEventTracker and sc.hyperNodesInitialEventTracker before calling Add() to prevent potential panics if they were not initialized.
| if isInInitialList { | |
| sc.nodeInitialEventTracker.Add(node.Name) | |
| sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name) | |
| } | |
| if isInInitialList { | |
| if sc.nodeInitialEventTracker != nil { | |
| sc.nodeInitialEventTracker.Add(node.Name) | |
| } | |
| if sc.hyperNodesInitialEventTracker != nil { | |
| sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name) | |
| } | |
| } |
| if isInInitialList { | ||
| //track the obj handling from initial list | ||
| sc.nodeInitialEventTracker.Add(csiNode.Name) | ||
| } |
There was a problem hiding this comment.
Add a nil check for sc.nodeInitialEventTracker before calling Add() to prevent potential panics.
| if isInInitialList { | |
| //track the obj handling from initial list | |
| sc.nodeInitialEventTracker.Add(csiNode.Name) | |
| } | |
| if isInInitialList && sc.nodeInitialEventTracker != nil { | |
| //track the obj handling from initial list | |
| sc.nodeInitialEventTracker.Add(csiNode.Name) | |
| } |
| if isInInitialList { | ||
| //tack the obj handling from inital list | ||
| sc.hyperNodesInitialEventTracker.Add(object) | ||
| } |
There was a problem hiding this comment.
Add a nil check for sc.hyperNodesInitialEventTracker before calling Add() to prevent potential panics.
| if isInInitialList { | |
| //tack the obj handling from inital list | |
| sc.hyperNodesInitialEventTracker.Add(object) | |
| } | |
| if isInInitialList && sc.hyperNodesInitialEventTracker != nil { | |
| //tack the obj handling from inital list | |
| sc.hyperNodesInitialEventTracker.Add(object) | |
| } |
|
/approve |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: JesseStutler The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
cherry-pick for #5172 and #5184