GatedPods metric negative

[ZiMengSheng]

What happened?

Kube-Scheduler currently provides a PreEnqueue plugin that prevents pods from entering ActiveQ or BackoffQ before they meet the scheduling requirements. This plugin is currently used to support the SchedulingGate feature. However, the Kube-Scheduler implementation does not seem to account for the possibility of pods becoming Gated from non-Gated state, resulting in some anomalies in GatedPods metrics.

The problematic code is as follows. When GatedBefore is false and PreEnqueue returns a failure, a pod is generated with the metric "UnschedulablePods" but the actual podInfo.Gated = true. When this pod is deleted, the Gated metric may become negative.

func (p *PriorityQueue) moveToActiveQ(logger klog.Logger, pInfo *framework.QueuedPodInfo, event string) bool {
	gatedBefore := pInfo.Gated()
	// If SchedulerPopFromBackoffQ feature gate is enabled,
	// PreEnqueue plugins were called when the pod was added to the backoffQ.
	// Don't need to repeat it here when the pod is directly moved from the backoffQ.
	if !p.isPopFromBackoffQEnabled || event != framework.BackoffComplete {
		p.runPreEnqueuePlugins(context.Background(), pInfo)
	}

	added := false
	p.activeQ.underLock(func(unlockedActiveQ unlockedActiveQueuer) {
		if pInfo.Gated() {
			// Add the Pod to unschedulablePods if it's not passing PreEnqueuePlugins.
			if unlockedActiveQ.has(pInfo) {
				return
			}
			if p.backoffQ.has(pInfo) {
				return
			}
			if p.unschedulablePods.get(pInfo.Pod) != nil {
				return
			}
			p.unschedulablePods.addOrUpdate(pInfo, event)
			logger.V(5).Info("Pod moved to an internal scheduling queue, because the pod is gated", "pod", klog.KObj(pInfo.Pod), "event", event, "queue", unschedulableQ)
			return
		}
		if pInfo.InitialAttemptTimestamp == nil {
			now := p.clock.Now()
			pInfo.InitialAttemptTimestamp = &now
		}

		unlockedActiveQ.add(pInfo, event)
		added = true

		p.unschedulablePods.delete(pInfo.Pod, gatedBefore)
		p.backoffQ.delete(pInfo)
		logger.V(5).Info("Pod moved to an internal scheduling queue", "pod", klog.KObj(pInfo.Pod), "event", event, "queue", activeQ)
		if event == framework.EventUnscheduledPodAdd.Label() || event == framework.EventUnscheduledPodUpdate.Label() {
			p.AddNominatedPod(logger, pInfo.PodInfo, nil)
		}
	})
	return added
}

What did you expect to happen?

GatedPods Metric should be correct!

How can we reproduce it (as minimally and precisely as possible)?

Implement a custom PreEnqueue Plugin, attempting to update it to make it gated from ungated, then delete it.

Anything else we need to know?

No response

Kubernetes version

Details

$ kubectl version
# paste output here

Cloud provider

Details

OS version

Details

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Details

Container runtime (CRI) and version (if applicable)

Details

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

[ZiMengSheng]

/sig scheduling

[ZiMengSheng]

/sig scheduling

[macsko]

/triage accepted

This issue looks valid. But, what's the use case for ungated -> gated scenario?

[ZiMengSheng]

/triage accepted

This issue looks valid. But, what's the use case for ungated -> gated scenario?

1. The Koordinator uses this to ensure that only one Pod in a Gang is in the ActiveQ or BackoffQ, providing gang-level queuing.
2. Some queue components use Pod Labels to allow Pods to be scheduled if their quotas are met, and to prevent Pods from being scheduled if their quotas are not met. When a pod cannot be scheduled due to resource fragmentation or other factors, the quota status may change dynamically.

[jamshidi799]

This issue looks valid

I read the mentioned code, but I didn't find a scenario that matches it. Could you please share your scenario? Thank you.

[ZiMengSheng]

/triage accepted
This issue looks valid. But, what's the use case for ungated -> gated scenario?

1. The Koordinator uses this to ensure that only one Pod in a Gang is in the ActiveQ or BackoffQ, providing gang-level queuing.
2. Some queue components use Pod Labels to allow Pods to be scheduled if their quotas are met, and to prevent Pods from being scheduled if their quotas are not met. When a pod cannot be scheduled due to resource fragmentation or other factors, the quota status may change dynamically.

@jamshidi799 here

[chengjoey]

When GatedBefore is false and PreEnqueue returns a failure, a pod is generated with the metric "UnschedulablePods" but the actual podInfo.Gated = true.

when PreEnqueue returns a failure，the podInfo.Gated == True, so gatedRecorder metric will increase, why metric UnschedulablePods is generated

kubernetes/pkg/scheduler/backend/queue/unschedulable_pods.go

Lines 48 to 59 in f278302

	func (u *unschedulablePods) addOrUpdate(pInfo *framework.QueuedPodInfo, event string) {
	podID := u.keyFunc(pInfo.Pod)
	if _, exists := u.podInfoMap[podID]; !exists {
	if pInfo.Gated() && u.gatedRecorder != nil {
	u.gatedRecorder.Inc()
	} else if !pInfo.Gated() && u.unschedulableRecorder != nil {
	u.unschedulableRecorder.Inc()
	}
	metrics.SchedulerQueueIncomingPods.WithLabelValues("unschedulable", event).Inc()
	}
	u.podInfoMap[podID] = pInfo
	}