Update queueing strategy API comments for the 2-phase Admission

[tenzen-y]

What happened:
The SecondPass scheduling mechanism takes head workloads from ClusterQueue heads and the second pass queue in

kueue/pkg/queue/manager.go

Lines 650 to 651 in 5665ed2

	workloads = append(workloads, m.secondPassQueue.takeAllReady()...)
	for cqName, cq := range m.hm.ClusterQueues() {

.

For the BestEffort FIFIO strategy, this sounds reasonable since the strategy allows Kueue to admit following workloads when the head workload is inadmissible.

However, in the StrictFIFO strategy, if there are any workloads in the second pass queue, queueManager should not take the following workloads from the clusterQueue to guarantee the order strictly.

What you expected to happen:
In the strict FIFO strategy mode, Kueue must always admit the head workload first.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[mimowo]

I'm not understanding why this is a bug.

I think BestEffortFIFO and StrictFIFO are meant to control quota reservation. In case of the second pass of scheduling the workloads already have quota reserved. Note that, even before introduction of the second pass of scheduling the admission of workload was gated by the AdmissionChecks, and workload would be admitted independently of BestEffort / StrictFIFO as soon as all checks were green. I think this is similar case here.

For example, even without the second scheduling pass, you can have a Provisioning flavor in the StrictFIFO queue, and two workloads in the queue: wl1 (queued first), wl2 (queued second).

1. wl1 is queud, then wl2
2. wl1 has quota reserved and starts to wait for AC
3. wl2 has quota reserved and starts to wait for AC
4. wl2 is admitted before wl1 because the AC provisioned first, before AC for wl1 (this is likely to happen, for example it requested fewer nodes)
5. wl1 is admitted now once its AC provisioned

So, the order of getting quota was FIFO, but not admission.

[tenzen-y]

So, the order of getting quota was FIFO, but not admission.

Yes, that's right for AC case.
Especially, in the step 5, wl1 will be admitted by workload-controller instead of scheduler since the quota reserved workload is not considered as an inadmissible workload.

I assumed that SecondPass scheduling admission (not quota reservation) still depends on FIFO. However, if we can define the queueing strategy is respected only for quota reservation, not admission, we should update the API comment and documentations:

• kueue/apis/kueue/v1beta1/clusterqueue_types.go

  Lines 81 to 94 in adee464

  	// QueueingStrategy indicates the queueing strategy of the workloads
  	// across the queues in this ClusterQueue.
  	// Current Supported Strategies:
  	//
  	// - StrictFIFO: workloads are ordered strictly by creation time.
  	// Older workloads that can't be admitted will block admitting newer
  	// workloads even if they fit available quota.
  	// - BestEffortFIFO: workloads are ordered by creation time,
  	// however older workloads that can't be admitted will not block
  	// admitting newer workloads that fit existing quota.
  	//
  	// +kubebuilder:default=BestEffortFIFO
  	// +kubebuilder:validation:Enum=StrictFIFO;BestEffortFIFO
  	QueueingStrategy QueueingStrategy `json:"queueingStrategy,omitempty"`

• https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#queueing-strategy

[tenzen-y]

/remove-kind bug
/kind cleanup

/retitle Update queueing strategy API comments for the 2-phase Admission

[tenzen-y]

/assign