Skip to content

[release-1.12] wait event handler completed before start scheduling#5201

Merged
JesseStutler merged 2 commits intovolcano-sh:release-1.12from
qi-min:release-1.12
Apr 15, 2026
Merged

[release-1.12] wait event handler completed before start scheduling#5201
JesseStutler merged 2 commits intovolcano-sh:release-1.12from
qi-min:release-1.12

Conversation

@qi-min
Copy link
Copy Markdown
Contributor

@qi-min qi-min commented Apr 14, 2026

cherry-pick for #5172 and #5184

qi-min added 2 commits April 14, 2026 21:02
Signed-off-by: qi-min <qim_34@163.com>
Signed-off-by: qi-min <qim_34@163.com>
@volcano-sh-bot volcano-sh-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 14, 2026
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a synchronization mechanism to ensure the Volcano scheduler waits for initial resource data to be fully processed before starting the scheduling cycle. Key changes include the addition of a ResourceSyncTimeout configuration, the introduction of an InitialEventAsyncHandlerTracker to monitor synchronization status, and the transition of internal queues to use a QueueObjectWrapper. The review feedback identifies several opportunities to improve the robustness of this implementation, specifically by adding nil checks for the new trackers to prevent potential panics during initialization and ensuring that initial events are only marked as complete in the tracker if the synchronization logic succeeds without errors.

Comment on lines +835 to +839
for _, handler := range sc.registeredHandlers {
if !handler.HasSynced() {
return false, nil
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a nil check for handler before calling HasSynced() to avoid potential panics if a handler registration failed during initialization.

Suggested change
for _, handler := range sc.registeredHandlers {
if !handler.HasSynced() {
return false, nil
}
}
for _, handler := range sc.registeredHandlers {
if handler != nil && !handler.HasSynced() {
return false, nil
}
}

Comment on lines +844 to +848
for name, handler := range sc.registeredHandlers {
if !handler.HasSynced() {
klog.Errorf("%s handler synchronization is not completed", name)
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a nil check for handler before calling HasSynced() in the error logging loop to prevent potential panics.

Suggested change
for name, handler := range sc.registeredHandlers {
if !handler.HasSynced() {
klog.Errorf("%s handler synchronization is not completed", name)
}
}
for name, handler := range sc.registeredHandlers {
if handler != nil && !handler.HasSynced() {
klog.Errorf("%s handler synchronization is not completed", name)
}
}

Comment on lines +1208 to 1215
err := sc.SyncNode(nodeWrapper.Object)
if nodeWrapper.IsInInitialList {
sc.nodeInitialEventTracker.Done(nodeWrapper.Object)
}
if err == nil {
sc.nodeQueue.Forget(nodeName)
sc.nodeQueue.Forget(nodeWrapper)
return true
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Move the Done call after the error check to ensure the initial sync actually succeeded before marking it as complete in the tracker. Also, add a nil check for sc.nodeInitialEventTracker to prevent potential panics.

Suggested change
err := sc.SyncNode(nodeWrapper.Object)
if nodeWrapper.IsInInitialList {
sc.nodeInitialEventTracker.Done(nodeWrapper.Object)
}
if err == nil {
sc.nodeQueue.Forget(nodeName)
sc.nodeQueue.Forget(nodeWrapper)
return true
}
err := sc.SyncNode(nodeWrapper.Object)
if err == nil {
if nodeWrapper.IsInInitialList && sc.nodeInitialEventTracker != nil {
sc.nodeInitialEventTracker.Done(nodeWrapper.Object)
}
sc.nodeQueue.Forget(nodeWrapper)
return true
}

Comment on lines +1231 to 1238
err := sc.SyncHyperNode(hnWrapper.Object)
if hnWrapper.IsInInitialList {
sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object)
}
if err == nil {
sc.hyperNodesQueue.Forget(name)
sc.hyperNodesQueue.Forget(hnWrapper)
return true
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Move the Done call after the error check to ensure the initial sync actually succeeded before marking it as complete in the tracker. Also, add a nil check for sc.hyperNodesInitialEventTracker to prevent potential panics.

Suggested change
err := sc.SyncHyperNode(hnWrapper.Object)
if hnWrapper.IsInInitialList {
sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object)
}
if err == nil {
sc.hyperNodesQueue.Forget(name)
sc.hyperNodesQueue.Forget(hnWrapper)
return true
}
err := sc.SyncHyperNode(hnWrapper.Object)
if err == nil {
if hnWrapper.IsInInitialList && sc.hyperNodesInitialEventTracker != nil {
sc.hyperNodesInitialEventTracker.Done(hnWrapper.Object)
}
sc.hyperNodesQueue.Forget(hnWrapper)
return true
}

Comment on lines +557 to +560
if isInInitialList {
sc.nodeInitialEventTracker.Add(node.Name)
sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add nil checks for sc.nodeInitialEventTracker and sc.hyperNodesInitialEventTracker before calling Add() to prevent potential panics if they were not initialized.

Suggested change
if isInInitialList {
sc.nodeInitialEventTracker.Add(node.Name)
sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name)
}
if isInInitialList {
if sc.nodeInitialEventTracker != nil {
sc.nodeInitialEventTracker.Add(node.Name)
}
if sc.hyperNodesInitialEventTracker != nil {
sc.hyperNodesInitialEventTracker.Add(string(hyperNodeEventSourceNode) + "/" + node.Name)
}
}

Comment on lines +667 to +670
if isInInitialList {
//track the obj handling from initial list
sc.nodeInitialEventTracker.Add(csiNode.Name)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a nil check for sc.nodeInitialEventTracker before calling Add() to prevent potential panics.

Suggested change
if isInInitialList {
//track the obj handling from initial list
sc.nodeInitialEventTracker.Add(csiNode.Name)
}
if isInInitialList && sc.nodeInitialEventTracker != nil {
//track the obj handling from initial list
sc.nodeInitialEventTracker.Add(csiNode.Name)
}

Comment on lines +1359 to +1362
if isInInitialList {
//tack the obj handling from inital list
sc.hyperNodesInitialEventTracker.Add(object)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a nil check for sc.hyperNodesInitialEventTracker before calling Add() to prevent potential panics.

Suggested change
if isInInitialList {
//tack the obj handling from inital list
sc.hyperNodesInitialEventTracker.Add(object)
}
if isInInitialList && sc.hyperNodesInitialEventTracker != nil {
//tack the obj handling from inital list
sc.hyperNodesInitialEventTracker.Add(object)
}

@JesseStutler
Copy link
Copy Markdown
Member

/approve
/lgtm

@volcano-sh-bot volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Apr 15, 2026
@volcano-sh-bot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JesseStutler
Once this PR has been reviewed and has the lgtm label, please assign wangyang0616 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@JesseStutler JesseStutler merged commit e083d02 into volcano-sh:release-1.12 Apr 15, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants