feat: add metrics-adapter component to support centralized hpa #3578

jwcesign · 2023-05-24T12:17:52Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Part of #3379
This PR focus on karmada-metrics-adapter, the responsibility is:

Query metrics from target member clusters.
Do metrics aggregation and return to apiserver.

Which issue(s) this PR fixes:
Fixes #none

Special notes for your reviewer:
Install way

# with karmada-apiserver context
kubectl apply -f artifacts/deploy/karmada-metrics-adapter-apiservice.yaml

# with karmada-host context
docker exec karmada-host-control-plane crictl rmi docker.io/karmada/karmada-metrics-adapter:latest
make image-karmada-metrics-adapter
docker tag karmada/karmada-metrics-adapter:{replcase here the tag} docker.io/karmada/karmada-metrics-adapter:latest
kind load docker-image docker.io/karmada/karmada-metrics-adapter:latest --name karmada-host

kubectl delete -f artifacts/deploy/karmada-metrics-adapter.yaml
kubectl apply -f artifacts/deploy/karmada-metrics-adapter.yaml

Test

root@karmada [10:40:26 AM] [~/workspace/git/fix-files] [metrics-adapter-v4]
-> # kubectl --context=karmada-apiserver top pods -A
NAMESPACE            NAME                                            CPU(cores)   MEMORY(bytes)
default              nginx-748c667d99-2crv5                          0m           5Mi
default              nginx-748c667d99-vjvkx                          0m           5Mi
kube-system          coredns-787d4945fb-g4jx7                        2m           19Mi
kube-system          coredns-787d4945fb-qb4dg                        4m           46Mi
kube-system          coredns-787d4945fb-wphrp                        2m           15Mi
kube-system          coredns-787d4945fb-zxtbb                        4m           57Mi
kube-system          etcd-member1-control-plane                      35m          112Mi
kube-system          etcd-member2-control-plane                      15m          51Mi
kube-system          kindnet-kb5cv                                   0m           27Mi
kube-system          kindnet-smqxk                                   1m           14Mi
kube-system          kube-apiserver-member1-control-plane            77m          997Mi
kube-system          kube-apiserver-member2-control-plane            27m          293Mi
kube-system          kube-controller-manager-member1-control-plane   27m          100Mi
kube-system          kube-controller-manager-member2-control-plane   11m          42Mi
kube-system          kube-proxy-7866s                                0m           38Mi
kube-system          kube-proxy-vdl4g                                1m           18Mi
kube-system          kube-scheduler-member1-control-plane            5m           43Mi
kube-system          kube-scheduler-member2-control-plane            3m           20Mi
kube-system          metrics-server-6b6f9ccc7-hxrl6                  2m           20Mi
local-path-storage   local-path-provisioner-c8855d4bb-5sjlg          1m           9Mi
local-path-storage   local-path-provisioner-c8855d4bb-psh8f          0m           18Mi

Does this PR introduce a user-facing change?:

`karmada-metrics-adapter`: Add new component karmada-metrics-adapter, and support cpu/memory metrics query.

RainbowMango

The code gen is failing.

Signed-off-by: jwcesign <jiangwei115@huawei.com>

codecov-commenter · 2023-05-25T02:12:13Z

Codecov Report

Merging #3578 (8dc8c7e) into master (218a0b2) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #3578      +/-   ##
==========================================
- Coverage   55.86%   55.83%   -0.03%     
==========================================
  Files         216      216              
  Lines       20121    20129       +8     
==========================================
  Hits        11240    11240              
- Misses       8276     8284       +8     
  Partials      605      605

Flag	Coverage Δ
unittests	`55.83% <0.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/util/cluster.go	`87.50% <0.00%> (-7.30%)`	⬇️

... and 1 file with indirect coverage changes

jwcesign · 2023-05-25T02:54:43Z

cc @XiShanYongYe-Chang @Poor12 @chaunceyjiang @Garrybest

chaunceyjiang · 2023-05-25T02:59:54Z

pkg/metricsadapter/provider/resourcemetrics.go

+		return nodeMetrics, nil
+	}
+
+	// In the previous step, if query with label selector, the name will be set to empty


I have some doubts. Why are there some metrics related to nodes here when there are no node objects in the Karmada control panel?

kubectl --kubeconfig karmada.kubeconfig get --raw /apis/metrics.k8s.io/v1beta1/nodes # Some information will be returned.

kubectl --kubeconfig karmada.kubeconfig get nodes # No resources found

The HPA controller will exclusively utilize the /apis/metrics.k8s.io/v1beta1/pods API. Therefore, it is acceptable if the other API is not available. In my opinion, users should use karmadactl top to inquire about metrics.

However, if we do provide the metrics API, we cannot prevent users from using it. If we choose not to provide it, there will be a need for an explanation.
cc @RainbowMango for checking

In my opinion, users should use karmadactl top to inquire about metrics.

For some UI projects, these project only use apis and commands are not used.

I prefer to keep the node metrics as we are unsure if users will utilize them.

XiShanYongYe-Chang

To be continue.

artifacts/deploy/karmada-metrics-adapter.yaml

pkg/metricsadapter/controller.go

pkg/metricsadapter/provider/resourcemetrics.go

artifacts/deploy/karmada-metrics-adapter.yaml

cmd/metrics-adapter/app/options/validation.go

pkg/metricsadapter/provider/resourcemetrics.go

chaunceyjiang · 2023-05-25T07:54:57Z

From the test results, it seems that using kubectl top pods cannot distinguish which pods come from which member cluster.

I suggest introducing a karmadactl top command to enhance kubectl top. karmadactl top pods can return cluster information.

-> # karmadactl --context=karmada-apiserver top pods -A
CLUSTER    NAMESPACE            NAME                                            CPU(cores)   MEMORY(bytes)
cluster1    default              nginx-748c667d99-2crv5                          0m           5Mi
cluster2    default              nginx-748c667d99-vjvkx                          0m           5Mi
cluster3    kube-system          coredns-787d4945fb-g4jx7                        2m           19Mi
cluster1    kube-system          coredns-787d4945fb-qb4dg                        4m           46Mi
cluster1    kube-system          coredns-787d4945fb-wphrp                        2m           15Mi
cluster1    kube-system          coredns-787d4945fb-zxtbb                        4m           57Mi
cluster1    kube-system          etcd-member1-control-plane                      35m          112Mi
cluster2    kube-system          etcd-member2-control-plane                      15m          51Mi
cluster1    kube-system          kindnet-kb5cv                                   0m           27Mi
cluster1    kube-system          kindnet-smqxk                                   1m           14Mi
cluster1    kube-system          kube-apiserver-member1-control-plane            77m          997Mi
cluster1    kube-system          kube-apiserver-member2-control-plane            27m          293Mi
cluster1    kube-system          kube-controller-manager-member1-control-plane   27m          100Mi
cluster1    kube-system          kube-controller-manager-member2-control-plane   11m          42Mi
cluster1    kube-system          kube-proxy-7866s                                0m           38Mi
cluster1    kube-system          kube-proxy-vdl4g                                1m           18Mi
cluster1    kube-system          kube-scheduler-member1-control-plane            5m           43Mi
cluster1    kube-system          kube-scheduler-member2-control-plane            3m           20Mi
cluster1    kube-system          metrics-server-6b6f9ccc7-hxrl6                  2m           20Mi
cluster1    local-path-storage   local-path-provisioner-c8855d4bb-5sjlg          1m           9Mi
cluster1    local-path-storage   local-path-provisioner-c8855d4bb-psh8f          0m           18Mi

jwcesign · 2023-05-25T07:57:32Z

I suggest introducing a karmadactl top command to enhance kubectl top. karmadactl top pods can return cluster information.

I think it's a good idea. cc @lonelyCZ ,

pkg/metricsadapter/controller.go

pkg/metricsadapter/provider/resourcemetrics.go

pkg/metricsadapter/controller.go

pkg/metricsadapter/provider/resourcemetrics.go

Signed-off-by: jwcesign <jiangwei115@huawei.com>

Poor12

Generally LGTM

cmd/metrics-adapter/app/options/options.go

pkg/metricsadapter/controller.go

Poor12 · 2023-05-27T08:35:30Z

pkg/metricsadapter/provider/resourcemetrics.go

+}
+
+// PodLister is an internal lister for pods
+type PodLister struct {


This struct implement cache.GenericLister interface. I'd like to add the following to ensure this interface is fully implemented.

var _ cache.GenericLister = &PodLister{} var _ v1listers.NodeLister = &NodeLister{} var _ api.MetricsGetter = &ResourceMetricsProvider{}

When init the metrics server, we use these two listers, there will be type checking. So whether is it necessary?

RainbowMango

Generally looks good to me. But I didn't look carefully.
Please @chaunceyjiang @Poor12 take a look.

artifacts/deploy/karmada-metrics-adapter-apiservice.yaml

artifacts/deploy/karmada-metrics-adapter.yaml

pkg/util/cluster.go

pkg/metricsadapter/provider/resourcemetrics.go

pkg/metricsadapter/controller.go

pkg/metricsadapter/provider/resourcemetrics.go

Signed-off-by: jwcesign <jwcesign@gmail.com>

RainbowMango

/lgtm
/approve

/hold
(I need to get the image registry ready before moving forward)

karmada-bot · 2023-05-28T15:01:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

RainbowMango · 2023-05-28T15:20:08Z

/hold cancel

karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label May 24, 2023

karmada-bot requested review from lfbear, lonelyCZ and RainbowMango May 24, 2023 12:18

karmada-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 24, 2023

RainbowMango reviewed May 24, 2023

View reviewed changes

jwcesign force-pushed the metrics-adapter-v4 branch 2 times, most recently from 4f7c279 to 79436c5 Compare May 24, 2023 12:37

Update vendor dep

a18702b

Signed-off-by: jwcesign <jiangwei115@huawei.com>

jwcesign force-pushed the metrics-adapter-v4 branch 2 times, most recently from b7ed104 to 7d712e1 Compare May 25, 2023 01:54

jwcesign force-pushed the metrics-adapter-v4 branch from 7d712e1 to 4177cb0 Compare May 25, 2023 02:19

chaunceyjiang reviewed May 25, 2023

View reviewed changes

XiShanYongYe-Chang reviewed May 25, 2023

View reviewed changes

artifacts/deploy/karmada-metrics-adapter.yaml Show resolved Hide resolved

pkg/metricsadapter/controller.go Outdated Show resolved Hide resolved

pkg/metricsadapter/controller.go Outdated Show resolved Hide resolved

pkg/metricsadapter/controller.go Outdated Show resolved Hide resolved

chaunceyjiang reviewed May 25, 2023

View reviewed changes

pkg/metricsadapter/provider/resourcemetrics.go Outdated Show resolved Hide resolved

chaunceyjiang reviewed May 25, 2023

View reviewed changes

pkg/metricsadapter/provider/resourcemetrics.go Outdated Show resolved Hide resolved

zishen reviewed May 25, 2023

View reviewed changes

chaunceyjiang reviewed May 25, 2023

View reviewed changes

pkg/metricsadapter/controller.go Show resolved Hide resolved

chaunceyjiang reviewed May 25, 2023

View reviewed changes

pkg/metricsadapter/provider/resourcemetrics.go Outdated Show resolved Hide resolved

XiShanYongYe-Chang reviewed May 25, 2023

View reviewed changes

jwcesign force-pushed the metrics-adapter-v4 branch 4 times, most recently from c6f5cb9 to e1429fb Compare May 26, 2023 03:00

feat: add metrics-adapter component to support centralized hpa

ae6c345

Signed-off-by: jwcesign <jiangwei115@huawei.com>

jwcesign force-pushed the metrics-adapter-v4 branch from e1429fb to ae6c345 Compare May 26, 2023 03:23

Poor12 reviewed May 27, 2023

View reviewed changes

RainbowMango reviewed May 27, 2023

View reviewed changes

artifacts/deploy/karmada-metrics-adapter-apiservice.yaml Show resolved Hide resolved

artifacts/deploy/karmada-metrics-adapter.yaml Show resolved Hide resolved

pkg/util/cluster.go Outdated Show resolved Hide resolved

jwcesign changed the title ~~feat: add metrics-adapter component to support centralized hpa~~ [WIP] feat: add metrics-adapter component to support centralized hpa May 27, 2023

karmada-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2023

jwcesign force-pushed the metrics-adapter-v4 branch 2 times, most recently from 4fb2ee6 to bf24272 Compare May 27, 2023 16:02

jwcesign changed the title ~~[WIP] feat: add metrics-adapter component to support centralized hpa~~ feat: add metrics-adapter component to support centralized hpa May 27, 2023

karmada-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2023

jwcesign force-pushed the metrics-adapter-v4 branch 2 times, most recently from b532f65 to 4cd3d3f Compare May 27, 2023 16:40

chaunceyjiang reviewed May 28, 2023

View reviewed changes

pkg/metricsadapter/provider/resourcemetrics.go Show resolved Hide resolved

jwcesign force-pushed the metrics-adapter-v4 branch from 4cd3d3f to f87f74f Compare May 28, 2023 02:15

chaunceyjiang mentioned this pull request May 28, 2023

feat: introducing a karmadactl top command #3593

Merged

jwcesign force-pushed the metrics-adapter-v4 branch 3 times, most recently from 4e33e8c to 8dc8c7e Compare May 28, 2023 06:14

chaunceyjiang reviewed May 28, 2023

View reviewed changes

pkg/metricsadapter/controller.go Show resolved Hide resolved

chaunceyjiang reviewed May 28, 2023

View reviewed changes

pkg/metricsadapter/provider/resourcemetrics.go Outdated Show resolved Hide resolved

feat: support fieldSelector

6dfa4d0

Signed-off-by: jwcesign <jwcesign@gmail.com>

jwcesign force-pushed the metrics-adapter-v4 branch from 8dc8c7e to 6dfa4d0 Compare May 28, 2023 08:03

RainbowMango approved these changes May 28, 2023

View reviewed changes

karmada-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 28, 2023

karmada-bot assigned RainbowMango May 28, 2023

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label May 28, 2023

karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 28, 2023

karmada-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 28, 2023

karmada-bot merged commit 4235569 into karmada-io:master May 28, 2023
12 checks passed

zishen mentioned this pull request Aug 21, 2023

REQUEST: New membership for zishen karmada-io/community#51

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add metrics-adapter component to support centralized hpa #3578

feat: add metrics-adapter component to support centralized hpa #3578

jwcesign commented May 24, 2023 •

edited

RainbowMango left a comment

codecov-commenter commented May 25, 2023 •

edited

jwcesign commented May 25, 2023

chaunceyjiang May 25, 2023 •

edited

jwcesign May 25, 2023

chaunceyjiang May 25, 2023

jwcesign May 25, 2023

XiShanYongYe-Chang left a comment

chaunceyjiang commented May 25, 2023

jwcesign commented May 25, 2023

Poor12 left a comment

Poor12 May 27, 2023

jwcesign May 27, 2023

RainbowMango left a comment

RainbowMango left a comment

karmada-bot commented May 28, 2023

RainbowMango commented May 28, 2023

feat: add metrics-adapter component to support centralized hpa #3578

feat: add metrics-adapter component to support centralized hpa #3578

Conversation

jwcesign commented May 24, 2023 • edited

RainbowMango left a comment

Choose a reason for hiding this comment

codecov-commenter commented May 25, 2023 • edited

Codecov Report

jwcesign commented May 25, 2023

chaunceyjiang May 25, 2023 • edited

Choose a reason for hiding this comment

jwcesign May 25, 2023

Choose a reason for hiding this comment

chaunceyjiang May 25, 2023

Choose a reason for hiding this comment

jwcesign May 25, 2023

Choose a reason for hiding this comment

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

chaunceyjiang commented May 25, 2023

jwcesign commented May 25, 2023

Poor12 left a comment

Choose a reason for hiding this comment

Poor12 May 27, 2023

Choose a reason for hiding this comment

jwcesign May 27, 2023

Choose a reason for hiding this comment

RainbowMango left a comment

Choose a reason for hiding this comment

RainbowMango left a comment

Choose a reason for hiding this comment

karmada-bot commented May 28, 2023

RainbowMango commented May 28, 2023

jwcesign commented May 24, 2023 •

edited

codecov-commenter commented May 25, 2023 •

edited

chaunceyjiang May 25, 2023 •

edited