ClientConnection.QPS applies to individual API types

[gabesaba]

What happened:
When setting ClientConnection.QPS, the limit applies to each type (e.g. pod, workload, status, clusterqueue, etc)

What you expected to happen:
This should be a global limit. Otherwise, it is hard for administrators to know exactly how many queries per second Kueue will be making to the API Server.

How to reproduce it (as minimally and precisely as possible):
Set to a very low limit (e.g. 1), with burst set to same as the limit. Observe that Kueue is making more than 1 call per second to the API Server.

[trasc]

/assign

[trasc]

Set to a very low limit (e.g. 1), with burst set to same as the limit. Observe that Kueue is making more than 1 call per second to the API Server.

@gabesaba How many calls do you observe?

[gabesaba]

Set to a very low limit (e.g. 1), with burst set to same as the limit. Observe that Kueue is making more than 1 call per second to the API Server.

@gabesaba How many calls do you observe?

it should be on the O(#API types). I didn't check if this was multiplied by namespaces, for namespaced types. So in the case of 1qps, at least CQ, LQ, Workload, Pod, Event (and others I may be missing) = 5qps.

[trasc]

Set to a very low limit (e.g. 1), with burst set to same as the limit. Observe that Kueue is making more than 1 call per second to the API Server.

@gabesaba How many calls do you observe?

it should be on the O(#API types). I didn't check if this was multiplied by namespaces, for namespaced types. So in the case of 1qps, at least CQ, LQ, Workload, Pod, Event (and others I may be missing) = 5qps.

How do you "mesure" this?

[gabesaba]

I'm not sure what development environment you're using, but generally:

• access k8 apiserver logs
• filter requests from Kueue
• add some load to Kueue so that it's making enough API calls
• group by time observe that in a given window (e.g. 60 seconds), there are more than 60 calls (if qps=1)
• additionally, group by API type (Workload/Pod/CQ) - observe that qps is at most the limit you set

[trasc]

@gabesaba It took some time but I found the cause, #2462 should fix it, please let me know if you can confirm the fix on your side as well.