Health check api spends too much time

[exiaohao]

Describe the bug
By using kiali to observe some more services(maybe 50+ services) like following APIs:
/namespaces/{namespace}/health
/namespaces/{namespace}/services/{service}/health
/api/namespaces/{namespace}/workloads/{workload}/health
...

It creates many requests to query health info, and spends lot of time to get results. Nothing will be rendering before all requests are response successfully
In my cluster, use Overview tab to see all namespaces' status. There's 190 requests created by browser and spends 1.4min to finish it.

Is there any way to cache or reduce request time? Or have pagination to split requests in every page turning?

Versions used
Kiali: v0.12.0
Istio: v1.0.4
Kubernetes version: v1.11.0

[jotak]

Hi @exiaohao
Thanks for reporting issue. I have a couple of questions:

Can you tell how many namespaces do you have?

Normally, in services/apps/workloads list pages the health is fetched asynchronously hence should not block the display of the list? Can you confirm it's the case, and what you're seeing only applies to the overview page?

In Overview page, if you pause the refresh rate and manually hit refresh, are you still waiting 1.4min? (I ask this because I suspect it could be a problem of refresh jobs summing over unfinished ones)

In any case we may improve the overview page to fetch health asynchronously

[exiaohao]

Thanks for your help, there's 178 namespaces returned by /api/namespaces

I know it's fetched asynchronously, but all async apis are response very slow. In following screen capture, i've paused auto refresh and use Chrome's network tool to observe all 190 requests' load time.

[jotak]

Ok so we could investigate some throttling and more asynchronous calling to tackle that.

[mathspanda]

@jotak Every health request is slow. Could app healths be cached in Kiali? Just start a goroutine to poll the healths.

[jotak]

@mathspanda health is something we want to get updated frequently to see any degradation when it occurs, caching would play against that. Each new health request performed at a given time may bring new information / failure detection. I would rather try to be more subtle in how we fetch it (with throttling or maybe lazy loading list)

[jotak]

Tracked here: https://issues.jboss.org/browse/KIALI-2216

[exiaohao]

Thanks for your help, kiali/kiali-ui#935 seems improved users experience a lot, but I'm still looking forward to improvement of speedup health request API.

[jmazzitelli]

@exiaohao This is not related to this issue directly, however, can you tell me about the performance you see with the graph pages (by that I mean the pages where you see the service/workload/app nodes and the lines/edges connecting them - what we call "the graph"). Do you see those pages loading OK? Is it slow? I'm just curious :-)

[exiaohao]

@jmazzitelli sorry for late response.
Form my view, generally the overview pages are slow (maybe lots of APIs are called). the detail page(as you said "the graph") is not slow.

[ghost]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.