KEP-168-2: Pending workloads visibility

[PBundyra]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Introduces a new API to expose information about the position of pending workloads in both ClusterQueue and LocalQueue.

Which issue(s) this PR fixes:

Part of #168

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	bb3df08
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6555ed88d35acb0008c37fd6

[k8s-ci-robot]

Hi @PBundyra. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[PBundyra]

/assign @mimowo
/assign @mwielgus

[tenzen-y]

/ok-to-test

[PBundyra]

/assign @tenzen-y

[denkensk]

Will the implementation consider the ability to support paged queries? I think this is still pretty important, especially with a lot of workloads.

[PBundyra]

Since the K8s Aggregation Layer does not support pagination out of the box, we don't want to commit to it since it would require significant effort. However, it can be implemented on the client side. We also expose the endpoint to fetch information about a single workload, so there is a way to query its position without listing all the remaining pending workloads.

I've updated the non-goals and API Details sections to cover this.

[alculquicondor]

Let's clarify what we mean by pagination.
Let's say I want to list 1000 elements now, and I can only see 100 at a time. Do you keep the 1000 elements somewhere in memory, with some key id, and allow users to query that list in 100 increments? In other words, you get a consistent view if you use the same id?

I don't think we need that. Simply listing elements from position X to Y, at this time, is enough.

Can we do something simple like having 2 query parameters for the API:

• first position
• number of elements (and this is some sane value by default: probably around 1000?)

Doing it in the client side (let's say a dashboard) could be wasteful.

[mimowo]

We don't know at the moment how to have extra query parameters, so went for the simple proposal without pagination at all. In the prototype we implement it here:

kueue/pkg/visibility/api/pending_workload_CQ.go

Lines 80 to 91 in 5bbb00f

	func (m *pendingWorkloadsInCQ) Get(ctx context.Context, name string, opts *metav1.GetOptions) (runtime.Object, error) {
	var wls []v1alpha1.PendingWorkloadSummary
	for _, val := range m.kueueMgr.GetPendingWorkloadsInfo(name) {
	wls = append(wls, v1alpha1.PendingWorkloadSummary{
	ObjectMeta: metav1.ObjectMeta{
	Name: val.Name,
	Namespace: val.Namespace,
	},
	})
	}
	return &v1alpha1.PendingWorkloadSummaryList{Items: wls}, nil
	}

. However, at this layer we don't already have access to http.Request to see the query parameters. It might be possible, but it is not obvious how to get them. AFAIK metrics server also does not do pagination, so we don't have a good example of passing the query parameters.

[denkensk]

Can we do something simple like having 2 query parameters for the API:

first position
number of elements (and this is some sane value by default: probably around 1000?)

Doing it in the client side (let's say a dashboard) could be wasteful.

I also think it's better to support on the server side or not support. On the client side is not a good choice.

[alculquicondor]

Cross linking: The Pod log subresource implements query parameters https://github.com/kubernetes/kubernetes/blob/029452198566a41bc39d04a1ec5bad3f37621a1c/pkg/registry/core/pod/rest/log.go#L78

It's a different interface.

[PBundyra]

Indeed, implementing a different interface allows us to pass query parameters. Hence, following @alculquicondor proposal, we will enable users to fetch pending workloads from position X to Y.

[mimowo]

nit: The last point does not seem important enough to mention in this section.

Also, I would suggest to formulate to capture the main technical aspects of the proposal, without going too much detail like in the "Design Details" section (or something along the lines):

"Add a new Extension API server to expose on-demand endpoints for fetching information about pending workloads."

Then, it may also be worth adding something like:
"The returned information about pending workloads includes all the necessary information relevant for their position in the queue, along with the position itself. There are three such endpoints: (1) to list the pending workloads in ClusterQueue, (2) list the pending workloads in LocalQueue, and (3) get a specific pending workload."

[mimowo]

LGTM overall. Left some nits, but it is IMO good enough to:
/assign @alculquicondor

[alculquicondor]

cc @yaroslava-serdiuk

[alculquicondor]

Let's clarify what we mean by pagination.
Let's say I want to list 1000 elements now, and I can only see 100 at a time. Do you keep the 1000 elements somewhere in memory, with some key id, and allow users to query that list in 100 increments? In other words, you get a consistent view if you use the same id?

I don't think we need that. Simply listing elements from position X to Y, at this time, is enough.

Can we do something simple like having 2 query parameters for the API:

• first position
• number of elements (and this is some sane value by default: probably around 1000?)

Doing it in the client side (let's say a dashboard) could be wasteful.

[alculquicondor]

that this mean that there will be 2 endpoints for clusterqueue, 2 for localqueue, etc?

[mimowo]

IIUC there is only one endpoint for ClusterQueue, and one for LocalQueue. These translate in an implied way into endpoints for admins and users, but this may depend on RBAC. We may rephrase this sentence to clarify.
For example: "There will be separate endpoints exposing the information about pending workloads for LocalQueues, and ClusterQueues". Or something along the lines.

[PBundyra]

Clarified

[alculquicondor]

would something like this be possible?

Suggested change

	GET /apis/pending-workloads.kueue.x-k8s.io/VERSION/clusterqueues/CQ_NAME
	GET /apis/visibility.kueue.x-k8s.io/VERSION/clusterqueues/CQ_NAME/pending_workloads

[mimowo]

We tried like this, but there seems to be a limitation of the framework, that the path can only have one fixed fragment. The fragment is the key in this map in the prototype:

kueue/pkg/visibility/api/install.go

Lines 45 to 47 in 5bbb00f

	visibilityServerResources := map[string]rest.Storage{
	"clusterqueues": pendingWorkloadsInCQ,
	}

.

Then, depending on the value returned here:

kueue/pkg/visibility/api/pending_workload_CQ.go

Line 100 in 5bbb00f

return false

we may yet have the namespace fragment on the path.

However, I don't think we can have the path you suggest here within the framework.

[alculquicondor]

It seems that / is treated specially.
We do this for pods https://github.com/kubernetes/kubernetes/blob/029452198566a41bc39d04a1ec5bad3f37621a1c/pkg/registry/core/rest/storage_core.go#L231

[PBundyra]

Thanks for pointing that out @alculquicondor. Indeed, it's possible to introduce that kind of subresource so I'll change the KEP

[alculquicondor]

What if the workload is admitted or finished? would you see some information?

[mimowo]

Not at the moment. However, for admitted or finished workloads users don't need extra information that is dynamically changing, such as position in the queue.

[alculquicondor]

ok, let's clarify in the KEP.
Maybe this is an argument for keeping the name of the API group as pending-workloads.kueue.x-k8s.io.

Let's go with that and we can iterate in the future as we see the need for new "visibility APIs".

[PBundyra]

I've clarified that this endpoint refers to a single pending workload

[tenzen-y]

Maybe, I can review this KEP in the next week.

[tenzen-y]

I came back here today.

[alculquicondor]

/approve
I'll leave lgtm to @mimowo

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, PBundyra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alculquicondor]

Also please squash

[tenzen-y]

Other than #1300 (comment), LGTM.

[mimowo]

LGTM, Just some nits

[mimowo]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 2d1ff78485b8093d033a843de53e2267bc3a5ade