ClusterQueue update events continue to fire forever

[tenzen-y]

What happened:
ClusterQueueStatus.flavorsUsage.resources with random order continues to fire ClusterQueue update events forever, and the number of reconciles will explode.
Therefore kueue-controller-manager much increases the load of kube-apiserver and etcd.

What you expected to happen:
ClusterQueues aren't updated if the old and the new ones have the same usage.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.25
• Kueue version (use git describe --tags --dirty --always): v0.3.2
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[tenzen-y]

/assign

[tenzen-y]

The problem part:

kueue/pkg/cache/cache.go

Lines 867 to 881 in b463573

	for rName, rQuota := range flvQuotas.Resources {
	used := flvUsage[rName]
	rUsage := kueue.ResourceUsage{
	Name: rName,
	Total: workload.ResourceQuantity(rName, used),
	}
	// Enforce `borrowed=0` if the clusterQueue doesn't belong to a cohort.
	if cq.Cohort != nil {
	borrowed := used - rQuota.Nominal
	if borrowed > 0 {
	rUsage.Borrowed = workload.ResourceQuantity(rName, borrowed)
	}
	}
	outFlvUsage.Resources = append(outFlvUsage.Resources, rUsage)
	}

[alculquicondor]

Are we triggering a reconcile even if the usage didn't change?

[tenzen-y]

Are we triggering a reconcile even if the usage didn't change?

Yes.
if the order of ClusterQueueStatus.flavorsUsage.resources is changed, it fires an update event.

For example, in the following diffs fire update event:

status:
...
  flavorsUsage:
  - name: foo
    resources:
    - borrowed: "0"
      name: "cpu"
      total: "4"
    - borrowed: "0"
      name: "memory"
      total: "4"

status:
...
  flavorsUsage:
  - name: foo
    resources:
    - borrowed: "0"
      name: "memory"
      total: "4"
    - borrowed: "0"
      name: "cpu"
      total: "4"

[tenzen-y]

This issue will often happen in the HPC cluster. Because in HPC cluster, pods have multiple SR-IOV Virtual functions in resources.Resources like this:

resources:
  requests:
    cpu: 256
    memory: 512Gi
    example.com/gpu: 8
    example.com/vf-0: 1
    example.com/vf-1: 1
    example.com/vf-2: 1
    example.com/vf-3: 1
    example.com/vf-4: 1
    example.com/vf-5: 1
    example.com/vf-6: 1
    example.com/vf-7: 1

[alculquicondor]

ah I see. we should probably do an internal alphabetical sort

[tenzen-y]

ah I see. we should probably do an internal alphabetical sort

Right. I will sort outFlvUsage.Resources according to resourceName in

kueue/pkg/cache/cache.go

Line 850 in b463573

func (c *Cache) Usage(cqObj *kueue.ClusterQueue) ([]kueue.FlavorUsage, int, error) {

.

Also, I will apply this way to the localQueue usage as well.