how to explain the Response Time

[nkongkimo]

Question Description
How should I explain the Response Time displayed in the Graph?

List of Questions

1. I see there are four indicators in the response time:
   "Average"
   "Median"
   "95th Percentile"
   "99th Percentile"

2.I want to know what the statistical methods are for these.
Regarding the situation in the graph below, can I explain that the time from client to ingress-gateway is 10ms, and from nginx-proxy to passthroughCluster is 3ms? And does the 14ms time include the 5ms?

3. In the part of the graph highlighted in red, 47ms is less than 57ms. How can I interpret this?
   

Can someone in the community please give me some responses?

[jshaughn]

The statistical method is basically using Prometheus' Histogram features against Istio's istio_request_duration_milliseconds metric. See Prometheus for more on their Histogram support. Histograms are not perfect and there is a degree of error. Looking at your very low request rates (For example, 4.4 req per second from ingress to the xxx-portal), you will probably see some fluctuation. In your top diagram I would say, yes, the 10ms include the 3ms of the upstream request. In your second diagram it could be that the "latest" workload does not always forward a request out of the mesh. Also, I don't know what request time option you have chosen. P99 can vary wildly based on outliers.

[nkongkimo]

thank you for your response
Chapter 2's graph uses the average method to show.
You mean in a more complex situation, it's not recommended to use Kiali's topology to check the transport time, right?

[jshaughn]

You mean in a more complex situation, it's not recommended to use Kiali's topology to check the transport time, right?

No, I did not say that. I don't think there is a problem with response time reporting in Kiali. I believe it correctly reflects response times based on the histogram metrics reported by Istio.

[jshaughn]

I did say that, in general, larger volumes of traffic will better reflect actual response times, because small samples can produce greater variation.

[nkongkimo]

Understood, thank you for your reply.
I have another small question to ask.
According to picture 2, it shows 57ms.
Can I think that this is because it's sending packets through many points to an outside service? Some trace are faster, and some are slower, so on average, it's higher than the route from istio-ingressgateway to the portal's pod, because the route from istio-ingressgateway to the portal's pod is relatively simple.

[jshaughn]

I think interpreting your data is really something for you to try and explain but yes, if you are going to use average and you have relatively low throughput, one outlier (for example, one really slow request), can easily produce a larger result. The average of [10,10,10,10,100] is 35. In general, average is not a very good metric for response time. Median is usually better, but percentiles are the norm, and it's why Kiali's default is P95 (95% of the requests fall below the displayed response time).