Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ManagedCluster informer to improve performance #117

Merged
merged 2 commits into from
Dec 7, 2022
Merged

Add ManagedCluster informer to improve performance #117

merged 2 commits into from
Dec 7, 2022

Conversation

ivan-cai
Copy link
Contributor

@ivan-cai ivan-cai commented Dec 6, 2022

We got a higher latency when using cluster-gateway with ocm-integration=true to access managed cluster k8s resources.
And through e2e benchmark test, I found that cluster-gateway will get ManagedCluster per request, and sometimes it will cost X*100ms(sometimes X senconds in the production environment).
ManagedCluster is not got from cache，so I add ManagedCluster informer to improve performance, and according to the result of e2e benchmark, the latency is greatly reduced.

add arg --v=4 to gateway-deployment, and then run multicluster e2e benchmark test case test case, I copy the logs of some high latency requests below:

I1206 10:22:12.321915       1 request.go:597] Waited for 392.608513ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-81
I1206 10:22:12.326916       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-81/proxy/api/v1/namespaces/kube-system" latency="398.586785ms" userAgent="Go-http-client/2.0" audit-ID="94579a16-45ba-4333-8c83-7bb95dd703ea" srcIP="172.23.211.87:58464" resp=200
I1206 10:22:12.522535       1 request.go:597] Waited for 387.186395ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-82
I1206 10:22:12.526706       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-82/proxy/healthz" latency="391.656812ms" userAgent="Go-http-client/2.0" audit-ID="1e02f8f4-8bcd-4fdb-a96d-978074d2f0c5" srcIP="172.23.211.88:36280" resp=200
I1206 10:22:12.722840       1 request.go:597] Waited for 382.323062ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-77
I1206 10:22:12.727957       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-77/proxy/api/v1/namespaces/kube-system" latency="387.744393ms" userAgent="Go-http-client/2.0" audit-ID="5022a7df-d6e8-48a7-a53a-2901cc15f187" srcIP="172.23.211.87:58464" resp=200
I1206 10:22:12.922070       1 request.go:597] Waited for 387.643281ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-80
I1206 10:22:12.926795       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-80/proxy/healthz" latency="393.948768ms" userAgent="Go-http-client/2.0" audit-ID="ac30211c-7c27-45f2-97b8-c6885826022b" srcIP="172.23.211.88:36280" resp=200
I1206 10:22:13.122193       1 request.go:597] Waited for 392.955174ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-78
I1206 10:22:13.126164       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-78/proxy/api/v1/namespaces/kube-system" latency="397.184533ms" userAgent="Go-http-client/2.0" audit-ID="5a27ffae-58d2-4b63-badf-4c6d832bc8e9" srcIP="172.23.211.87:58464" resp=200

the result of e2e benchmark before adding ManagedCluster informer:

• [MEASUREMENT]
Basic RoundTrip Test
/root/cluster-gateway/e2e/benchmark/configmap.go:32
  list namespace from managed cluster
  /root/cluster-gateway/e2e/benchmark/configmap.go:143

  Ran 10 samples:
  runtime:
    Fastest Time: 0.384s
    Slowest Time: 1.212s
    Average Time: 0.921s ± 0.311s
------------------------------

Ran 1 of 1 Specs in 9.210 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
    /root/cluster-gateway/e2e/benchmark/configmap.go:143

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

--- PASS: TestE2E (9.23s)
PASS
ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	9.250s

the result of e2e benchmark after adding ManagedCluster informer:

• [MEASUREMENT]
Basic RoundTrip Test
/root/cluster-gateway/e2e/benchmark/configmap.go:32
  list namespace from managed cluster
  /root/cluster-gateway/e2e/benchmark/configmap.go:143

  Ran 10 samples:
  runtime:
    Fastest Time: 0.020s
    Slowest Time: 0.058s
    Average Time: 0.025s ± 0.011s
------------------------------

Ran 1 of 1 Specs in 0.250 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
    /root/cluster-gateway/e2e/benchmark/configmap.go:143

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

--- PASS: TestE2E (0.26s)
PASS
ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	0.286s

@ivan-cai
use informer to get ManagedCluster, and add featuregate OCMClusterCon…
cbdf485 
…trol

Signed-off-by: ivan-cai <caijing.cai@alibaba-inc.com>
@ivan-cai
Copy link
Contributor Author

ivan-cai commented Dec 6, 2022

@yue9944882 please review this pr, thx.

@ivan-cai
avoid nil pointer error before poststart hook is done
51dcb7d 
Signed-off-by: ivan-cai <caijing.cai@alibaba-inc.com>
Somefive
Somefive approved these changes Dec 7, 2022
View reviewed changes
Copy link
Collaborator

@Somefive Somefive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM /cc @yue9944882

@yue9944882 yue9944882 merged commit 3208442 into oam-dev:master Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Somefive Somefive Awaiting requested review from Somefive

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ivan-cai @Somefive @yue9944882