Facing latency issue in kubernetes dashboard in v 7.1.2

[Sandeepsac]

What happened?

We facing latency issue in latest version 7.1.2 , mainly in list of Node details taking more than 2 min or getting 504 error timeout. other than this we facing issue in list of pods, jobs , deployment taking time maximum of 30sec.
we having more than 80 nodes running in our cluster , we have increased resource value max of 1 core CPU, 1Gb of memory for Dashboard API. still we facing same issue

What did you expect to happen?

we expecting faster as in previous version v2.7.0

How can we reproduce it (as minimally and precisely as possible)?

we are using 80+ nodes, more than 200 pods,.. version of k8s 1.27.1

Anything else we need to know?

No response

What browsers are you seeing the problem on?

Chrome

Kubernetes Dashboard version

7.1.2

Kubernetes version

1.27

Dev environment

No response

[floreks]

Check resource refresh time on the Settings view. By default, it is 10 seconds. It might not be enough to fetch a big amount of data. Extend it to 1 minute or more.

[Sandeepsac]

Its 10sec.

[floreks]

Ye, that is the default value. Increase it until you will not see any requests being cancelled in the network tab.

[Sandeepsac]

@floreks Still list of pods and node getting more time.

[floreks]

You can just set it to 0 and disable auto refresh. API server heavily throttles requests with such a big amount of data.

[Sandeepsac]

Can't able to set 0 to disable auto refresh, if we save by keeping zero, it getting revert to default 10 again

[Sandeepsac]

Team, what ever resource configuration we gave list of nodes taking more time , kindly make sure test with 100+ nodes

[floreks]

Can't able to set 0 to disable auto refresh, if we save by keeping zero, it getting revert to default 10 again

That's a bug then. We will fix that.

Team, what ever resource configuration we gave list of nodes taking more time , kindly make sure test with 100+ nodes

Unfortunately, we don't have access to clusters with so many resources. There is no good way for us to test it. This is a non-profit open-source project maintained by very few people in our own free time.

The main issue here is that the Kubernetes API server does not expose advanced data selection with pagination, filtering, and sorting. That's why every request needs to get all resources.

Using informers can also be problematic due to how our authentication works. Dashboard API would essentially require full admin access to the cluster if we'd want to benefit from caching. That's something I'd want to try but didn't have time yet.

[Sandeepsac]

Thanks for Update @floreks , we have tested with full admin access RBAC policy to check.,

In previous version which we used 2.7.0, it has only dashboard metrics and Dashboard API with single replica , it worked fine, the only issue is in cronjob can't be described , other than that it absolutely fine,

but here we have web service, API service with 3 replica in default, Kong for api proxy, but here we can see lot of latency.
Is there any possible other way to reduce latency. ? we have checked by increasing API resource limit in deployment Still facing latency

[floreks]

Could you try overriding "kong.env.dns_order" value to "A,CNAME,LAST,SRV" and see if it improves anything?

I have also noticed some latency issues with kong, but it usually worked better on subsequent requests.

[floreks]

Closing in favor of #8835

/close

[k8s-ci-robot]

@floreks: Closing this issue.

In response to this:

Closing in favor of #8835

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.