Ensure that the framework is available using RESTMapper instead of getting CRDs #1046

tenzen-y · 2023-08-07T18:27:11Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Using RESTMapper, we can ensure that the manager recognizes the frameworks, and we can reduce passing permissions to the manager.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

netlify · 2023-08-07T18:27:17Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`76c693d`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/64d6256524aa160008e1aa31

mimowo

Thanks for looking into this. Looks like the right direction, some questions.

mimowo · 2023-08-08T08:39:29Z

main.go

-		if isFrameworkEnabled(cfg, name) && crds.Has(name) {
-			if err := cb.NewReconciler(
+		gvk := cb.JobType.GetObjectKind().GroupVersionKind()
+		if _, err := mgr.GetRESTMapper().RESTMapping(gvk.GroupKind(), gvk.Version); err != nil && isFrameworkEnabled(cfg, name) {


isFrameworkEnabled(cfg, name) && crds.Has(name) was the happy path. So IIUC it should correspond to no error (err= nil).

Also, make sure the error we get is not lost. At least log it, but probably we should propagate it up.

I would also do any checks only if the isFrameworkEnabled responds to true. Otherwise we risk logging an error or propagating up for frameworks which are disabled.

what is the err if there is no mapping, is it NotFound? Maybe there are 3 cases:

no error -> happy path

NotFound error- > path for !crds.Has(name)

other generic error -> propagate the error

isFrameworkEnabled(cfg, name) && crds.Has(name) was the happy path. So IIUC it should correspond to no error (err= nil).

Yes, that's right. It's my bad :(

what is the err if there is no mapping,

It's expected meta.IsNoMatchError.

Since the controller-runtime v0.15, the returned error seems to be changed with discovery.ErrGroupDiscoveryFailed .

And the changing seems the unintended breaking changes:
kubernetes-sigs/controller-runtime#2425

trasc

I don't think this is doing the same thing, especially since we no longer need RBAC permissions I suspect that GetRESTMapper will only have the view of the local schema, which will always contain the types in question.

Was this tested in a live cluster?

tenzen-y · 2023-08-08T11:41:15Z

I don't think this is doing the same thing, especially since we no longer need RBAC permissions I suspect that GetRESTMapper will only have the view of the local schema, which will always contain the types in question.

Was this tested in a live cluster?

It's a great catch @trasc! You're right. We need to pass GVK in another way.

trasc · 2023-08-08T12:09:56Z

I don't think this is doing the same thing, especially since we no longer need RBAC permissions I suspect that GetRESTMapper will only have the view of the local schema, which will always contain the types in question.

Was this tested in a live cluster?

It's a great catch @trasc! You're right. We need to pass GVK in another way.

If the check is problematic in some usecase, we can make it configurable (enabled by default).

tenzen-y · 2023-08-08T12:41:06Z

main.go

+			var NoMatchingErr *discovery.ErrGroupDiscoveryFailed
+			if !errors.As(err, &NoMatchingErr) {
+				return err
+			}
+			log.Info("No matching API server for job framework, skip to create controller and webhook")


I confirmed if this check is effective using ytenzen/kueue-controller:rest-mapper in the KinD cluster:

... {"level":"info","ts":"2023-08-08T12:35:10.352203711Z","logger":"setup","caller":"workspace/main.go:233","msg":"No matching API server for job framework, skip to create controller and webhook","jobFrameworkName":"kubeflow.org/mpijob"} ...

tenzen-y · 2023-08-08T12:42:00Z

I don't think this is doing the same thing, especially since we no longer need RBAC permissions I suspect that GetRESTMapper will only have the view of the local schema, which will always contain the types in question.
Was this tested in a live cluster?

It's a great catch @trasc! You're right. We need to pass GVK in another way.

If the check is problematic in some usecase, we can make it configurable (enabled by default).

No worries. I just would like to clean up.

tenzen-y · 2023-08-08T12:42:27Z

main.go

+			// TODO: If the below PR is released, we need to change a way to check if the GVK is registered.
+			// REF: https://github.com/kubernetes-sigs/controller-runtime/pull/2425
+			// if !meta.IsNoMatchError(err) {
+			//   return err
+			// }
+			// log.Info("No matching API server for job framework, skip to create controller and webhook")


Since the controller-runtime v0.15, the returned error seems to be changed with discovery.ErrGroupDiscoveryFailed .

And the changing seems the unintended breaking changes:
kubernetes-sigs/controller-runtime#2425

mimowo · 2023-08-08T13:05:08Z

main.go

 			}
 		}
+		if err := noop.SetupWebhook(mgr, cb.JobType); err != nil {


Should think not be under else?

In the current behavior, the noop.SetupWebhook is run even when isFrameworkEnabled==true && crds.Has==false.
So I think we need to run here even when isFrameworkEnabled==true && errors.As(err, &NoMatchingErr)==true.

WDYT?

Oh, I missed that in the happy path you now do return nil. I was thinking the noop.SetupWebhook is called now in that case.

tenzen-y · 2023-08-09T16:27:44Z

@trasc @mimowo Could you recheck this PR?
/assign @trasc @mimowo

mimowo

LGTM

mimowo · 2023-08-10T08:03:43Z

main.go

+				// if !meta.IsNoMatchError(err) {
+				//   return err
+				// }
+				// log.Info("No matching API server for job framework, skip to create controller and webhook")


nit: no need for the log line, it's not changing :)

It's a good point!
Done.

k8s-ci-robot · 2023-08-10T08:12:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2023-08-10T08:24:27Z

main.go

+
+		if isFrameworkEnabled(cfg, name) {
+			if _, err := mgr.GetRESTMapper().RESTMapping(cb.GVK.GroupKind(), cb.GVK.Version); err != nil {
+				// TODO: If the below PR is released, we need to change a way to check if the GVK is registered.


Actually, one more thought. There is a risk that once the PR is released, say in controller-runtime 0.16, we may not remember to check this when bumping the version and break this. I think it is unlikely a person doing a bump of controller-runtime would look at this comment without prior knowledge.
Some ideas to mitigate this risk:

prepare the check in advance, if !meta.IsNoMatchError(err) && !errors.As(err, &NoMatchingErr) { return err }. Then leave the TODO comment to cleanup up once released

coordinate with the release of the PR and bump controller-runtime (but might be not necessarily involving)

create an issue in advance in kueue to increase visibility, giving a title like "Bump controller-runtime and adjust RESTMapper usage"

I'm leaning towards (1.) or (2.), but (2.) only if it is to happen with Kueue 0.5 release, because I don't like keeping PRs in a freezer for long :). (1.) seems safe to do.

EDIT: 4. integration tests for the scenario.
(4.) would be great, but might be overkill

WDYT?

1 sounds good, but also leave a TODO and issue.

@mimowo It's a great suggestion. I think 1 would be nice.

1 sounds good, but also leave a TODO and issue.

I agree. We should create an issue.

Changed logic and created an issue: #1054

alculquicondor · 2023-08-10T19:44:44Z

main.go

+				if !errors.As(err, &NoMatchingErr) {
+					return err
+				}
+				log.Info("No matching API server for job framework, skip to create controller and webhook")


include which framework, by adding a key and a value

The framework name is already set in

kueue/main.go

Line 219 in ed880c7

log := setupLog.WithValues("jobFrameworkName", name)

.

alculquicondor · 2023-08-10T19:47:28Z

main.go

+
+		if isFrameworkEnabled(cfg, name) {
+			if _, err := mgr.GetRESTMapper().RESTMapping(cb.GVK.GroupKind(), cb.GVK.Version); err != nil {
+				// TODO: If the below PR is released, we need to change a way to check if the GVK is registered.


1 sounds good, but also leave a TODO and issue.

trasc · 2023-08-11T10:27:51Z

main.go

-				return err
+
+		if isFrameworkEnabled(cfg, name) {
+			if _, err := mgr.GetRESTMapper().RESTMapping(cb.GVK.GroupKind(), cb.GVK.Version); err != nil {


Could we use something like

mgr.GetScheme().ObjectKinds(cb.JobType)

or apiutil.GVKForObject() insted if chaging IntegrationCallbacks

Let me check.

As a result, both ways (mgr.GetScheme().ObjectKinds(cb.JobType) and apiutil.GVKForObject()) are different from what is expected.

As I checked both ways in the KinD cluster, both ways returned the GVK even if the jobFramework CRD wasn't deployed because I guess that we registered all jobFramework schemes as an init().

Also, mgr.GetScheme().ObjectKinds returns false as a second returned value regardless of whether the jobFramework CRD is deployed in the cluster.

Sure, but what I meant is that instead of changing the IntegrationCallbacks structure and add new field in all the integrations, get the GVK with one of those two calls. something like:

Suggested change

if _, err := mgr.GetRESTMapper().RESTMapping(cb.GVK.GroupKind(), cb.GVK.Version); err != nil {

gvk := apiutil.GVKForObject(cb.JobType)

if _, err := mgr.GetRESTMapper().RESTMapping(gvk.GroupKind(), gvk.Version); err != nil {

It would work well.

…tting theCRDs Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

trasc

/lgtm

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Aug 7, 2023

k8s-ci-robot requested review from denkensk and trasc August 7, 2023 18:27

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 7, 2023

tenzen-y force-pushed the use-mapper-provider branch from d47966e to 28218b4 Compare August 7, 2023 22:16

mimowo reviewed Aug 8, 2023

View reviewed changes

trasc reviewed Aug 8, 2023

View reviewed changes

tenzen-y force-pushed the use-mapper-provider branch from 28218b4 to 06e96dc Compare August 8, 2023 12:37

tenzen-y commented Aug 8, 2023

View reviewed changes

tenzen-y force-pushed the use-mapper-provider branch from 06e96dc to d98d6fb Compare August 8, 2023 12:44

tenzen-y changed the title ~~Ensure that the framework is available using RESTMapper instead of getting CRDs~~ WIP: Ensure that the framework is available using RESTMapper instead of getting CRDs Aug 8, 2023

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 8, 2023

tenzen-y force-pushed the use-mapper-provider branch from d98d6fb to 06860fc Compare August 8, 2023 12:57

mimowo reviewed Aug 8, 2023

View reviewed changes

tenzen-y changed the title ~~WIP: Ensure that the framework is available using RESTMapper instead of getting CRDs~~ Ensure that the framework is available using RESTMapper instead of getting CRDs Aug 8, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 8, 2023

k8s-ci-robot assigned mimowo and trasc Aug 9, 2023

mimowo reviewed Aug 10, 2023

View reviewed changes

tenzen-y force-pushed the use-mapper-provider branch from 06860fc to ed880c7 Compare August 10, 2023 08:12

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2023

mimowo reviewed Aug 10, 2023

View reviewed changes

alculquicondor reviewed Aug 10, 2023

View reviewed changes

trasc reviewed Aug 11, 2023

View reviewed changes

tenzen-y force-pushed the use-mapper-provider branch from ed880c7 to b95d0fd Compare August 11, 2023 11:11

tenzen-y mentioned this pull request Aug 11, 2023

Change a way to check if the GVK is registered #1054

Closed

Ensure that the framework is available using RESTMapper instead of ge…

76c693d

…tting theCRDs Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the use-mapper-provider branch from b95d0fd to 76c693d Compare August 11, 2023 12:11

trasc reviewed Aug 11, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 11, 2023

k8s-ci-robot merged commit acec9e1 into kubernetes-sigs:main Aug 11, 2023
15 checks passed

k8s-ci-robot added this to the v0.5 milestone Aug 11, 2023

tenzen-y deleted the use-mapper-provider branch August 11, 2023 12:43

tenzen-y mentioned this pull request Aug 16, 2023

Extend make verify to fully check charts/kueue/templates #1059

Merged

	if _, err := mgr.GetRESTMapper().RESTMapping(cb.GVK.GroupKind(), cb.GVK.Version); err != nil {
	gvk := apiutil.GVKForObject(cb.JobType)
	if _, err := mgr.GetRESTMapper().RESTMapping(gvk.GroupKind(), gvk.Version); err != nil {

Ensure that the framework is available using RESTMapper instead of getting CRDs #1046

Ensure that the framework is available using RESTMapper instead of getting CRDs #1046

Conversation

tenzen-y commented Aug 7, 2023

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Aug 7, 2023 • edited

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

mimowo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trasc left a comment

Choose a reason for hiding this comment

tenzen-y commented Aug 8, 2023

trasc commented Aug 8, 2023

Choose a reason for hiding this comment

tenzen-y commented Aug 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Aug 9, 2023

mimowo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 10, 2023

mimowo Aug 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trasc Aug 11, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trasc left a comment

Choose a reason for hiding this comment

netlify bot commented Aug 7, 2023 •

edited

mimowo Aug 10, 2023 •

edited

trasc Aug 11, 2023 •

edited