⚠️ Use and provide a single dynamic rest mapper #2296

vincepri · 2023-05-02T18:37:31Z

No description provided.

k8s-ci-robot · 2023-05-02T18:37:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2023-05-02T18:57:24Z

Didn't see the test failure before but looks unrelated

/test pull-controller-runtime-test

sbueringer

one nit, up to you

pkg/client/apiutil/restmapper.go

sbueringer · 2023-05-02T19:08:48Z

/retest

vincepri · 2023-05-03T14:24:57Z

/retest

pkg/client/apiutil/restmapper.go

Signed-off-by: Vince Prignano <vincepri@redhat.com>

k8s-ci-robot · 2023-05-03T19:27:55Z

@vincepri: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-controller-runtime-apidiff	`935faeb`	link	false	`/test pull-controller-runtime-apidiff`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sbueringer · 2023-05-04T05:16:28Z

/lgtm
/approve

provider-kubernetes initialized a new kubernetes client for each reconcile. The REST mapper in controller-runtime used to fetch information about every CRD in the cluster. controller-runtime introduced a lazy restmapper which means we don't have to introduce any complex caching to get a significant performance boost in provider-kubernetes: kubernetes-sigs/controller-runtime#2116 This seems to become the default in the next release: kubernetes-sigs/controller-runtime#2296 But this is so significant that we want to update now: * CPU reduced from constant throttling at 0.4 cores to 0.04 cores * CloudWatch / EKS audit log costs reduced significantly (55% for our cluster, with a lot of provider-kubernetes resources) Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>

New changes done to the controller-runtime to make the rest mapper truly lazy in nature now means that the connection is never really validated until the first connection is made. So the current test is basically of no use anymore. Cleaning them up. xref: kubernetes-sigs/controller-runtime#2296

provider-kubernetes initialized a new kubernetes client for each reconcile. The REST mapper in controller-runtime used to fetch information about every CRD in the cluster. controller-runtime introduced a lazy restmapper which means we don't have to introduce any complex caching to get a significant performance boost in provider-kubernetes: kubernetes-sigs/controller-runtime#2116 This seems to become the default in the next release: kubernetes-sigs/controller-runtime#2296 But this is so significant that we want to update now: * CPU reduced from constant throttling at 0.4 cores to 0.04 cores * CloudWatch / EKS audit log costs reduced significantly (55% for our cluster, with a lot of provider-kubernetes resources) Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>

…RESTMapper kubernetes-sigs/controller-runtime#2296

RainbowMango · 2023-06-28T08:18:25Z

@vincepri May I ask the background of this PR?

Why removed the following options: WithCustomMapper, WithExperimentalLazyMapper, WithLazyDiscovery, and WithLimiter?

vincepri · 2023-07-05T19:55:32Z

@RainbowMango The options were either outdated, or provided no meaningful additions. Folks can always use their own RESTMapper implementation if they need. For the majority of controller-runtime users, having a single implementation is less error prone.

New changes done to the controller-runtime to make the rest mapper truly lazy in nature now means that the connection is never really validated until the first connection is made. So the current test is basically of no use anymore. Cleaning them up. xref: kubernetes-sigs/controller-runtime#2296

RainbowMango · 2023-07-18T10:21:24Z

Hi @vincepri,
We built a cached restmapper based on the DynamicRESTMapper of contriller-runtime@v0.14, the cached restmapper simply maintains a map which records the recent match result, and the map will be reset if the DiscoveryRESTMapper needs a reload.
We did that by the WithCustomMapper before, but during the bump to v0.15, we can't use it anymore.

Do you have any suggestions about how we can adopt the breaking change?
Or, is it possible to add the WithCustomMapper back to the upcoming release?

PS:
The background why we need the cahced restmapper is all about the performance, the underlying Kubernetes DiscoveryRESTMapper caches the resource with a slice, which performance is not good comparing with map.
We did a benchmark after that:

# go test -bench=. 
goos: linux
goarch: amd64
pkg: github.com/karmada-io/karmada/pkg/util/restmapper
cpu: Intel(R) Xeon(R) Gold 6278C CPU @ 2.60GHz
BenchmarkGetGroupVersionResource-4            	   52779	     20799 ns/op
BenchmarkGetGroupVersionResourceWithCache-4   	  245550	      4754 ns/op   // cached restmapper
PASS
ok  	github.com/karmada-io/karmada/pkg/util/restmapper	2.574s

The performance of cached restmapper had improved significantly.

vincepri · 2023-07-18T13:44:46Z

@RainbowMango Overriding the RESTMapper implementation in the manager is still possible with

controller-runtime/pkg/manager/manager.go

Lines 108 to 112 in 56a973c

    
           // MapperProvider provides the rest mapper used to map go types to Kubernetes APIs. 
        
           // 
        
           // If set, the RESTMapper returned by this function is used to create the RESTMapper 
        
           // used by the Client and Cache. 
        
           MapperProvider func(c *rest.Config, httpClient *http.Client) (meta.RESTMapper, error)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 2, 2023

k8s-ci-robot requested a review from alvaroaleman May 2, 2023 18:37

k8s-ci-robot requested a review from FillZpp May 2, 2023 18:37

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 2, 2023

sbueringer mentioned this pull request May 2, 2023

🐛 make clusterctl REST client throttling configurable kubernetes-sigs/cluster-api#8411

Closed

sbueringer reviewed May 2, 2023

View reviewed changes

pkg/client/apiutil/restmapper.go Outdated Show resolved Hide resolved

vincepri force-pushed the restmapper-one branch 3 times, most recently from 6ac8b0c to 98137e8 Compare May 3, 2023 14:17

vincepri force-pushed the restmapper-one branch 3 times, most recently from 651fe09 to 98d420f Compare May 3, 2023 15:14

sbueringer reviewed May 3, 2023

View reviewed changes

pkg/client/apiutil/restmapper.go Outdated Show resolved Hide resolved

pkg/client/apiutil/restmapper.go Outdated Show resolved Hide resolved

pkg/client/apiutil/restmapper.go Outdated Show resolved Hide resolved

⚠️ Use and provide a single dynamic rest mapper

935faeb

Signed-off-by: Vince Prignano <vincepri@redhat.com>

vincepri force-pushed the restmapper-one branch from 98d420f to 935faeb Compare May 3, 2023 19:26

k8s-ci-robot assigned sbueringer May 4, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 4, 2023

k8s-ci-robot merged commit 57674c8 into kubernetes-sigs:main May 4, 2023
8 of 9 checks passed

chlunde mentioned this pull request May 8, 2023

Use lazy rest mapper for performance and less audit crossplane-contrib/provider-kubernetes#116

Closed

2 tasks

ary1992 added a commit to ary1992/gardener that referenced this pull request Jun 28, 2023

Remove WithLazyDiscovery, WithLimiter options from apiutil.NewDynamic…

6c6f4ac

…RESTMapper kubernetes-sigs/controller-runtime#2296

RainbowMango mentioned this pull request Jul 15, 2023

Bump Kubernetes dependencies to v1.27.3 karmada-io/karmada#3730

Merged

xuezhaojun mentioned this pull request Sep 13, 2023

Fix: mapping failed. stolostron/multicloud-operators-foundation#660

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

⚠️ Use and provide a single dynamic rest mapper #2296

⚠️ Use and provide a single dynamic rest mapper #2296

vincepri commented May 2, 2023

k8s-ci-robot commented May 2, 2023

sbueringer commented May 2, 2023

sbueringer left a comment

sbueringer commented May 2, 2023

vincepri commented May 3, 2023

k8s-ci-robot commented May 3, 2023

sbueringer commented May 4, 2023 •

edited

RainbowMango commented Jun 28, 2023

vincepri commented Jul 5, 2023

RainbowMango commented Jul 18, 2023

vincepri commented Jul 18, 2023

⚠️ Use and provide a single dynamic rest mapper #2296

⚠️ Use and provide a single dynamic rest mapper #2296

Conversation

vincepri commented May 2, 2023

k8s-ci-robot commented May 2, 2023

sbueringer commented May 2, 2023

sbueringer left a comment

Choose a reason for hiding this comment

sbueringer commented May 2, 2023

vincepri commented May 3, 2023

k8s-ci-robot commented May 3, 2023

sbueringer commented May 4, 2023 • edited

RainbowMango commented Jun 28, 2023

vincepri commented Jul 5, 2023

RainbowMango commented Jul 18, 2023

vincepri commented Jul 18, 2023

sbueringer commented May 4, 2023 •

edited