Telemetry access logging: add cache

[howardjohn]

In the old style access logs, we had caching of the generated config. In Telemetry, we lost this.

In real prod clusters, we saw 50% of Istiod CPU spent just on this part. It would be good to add a cache here.

I did some initial work here but got distracted. I might be able to complete but if anyone else wants to pick it up, even better.

[zirain]

I will take a look this.

[zirain]

In real prod clusters, we saw 50% of Istiod CPU spent just on this part. It would be good to add a cache here.

can you share more details about this? what the cluster's size? how many workloads? how many telemetry resources in the clusters?

buildAccessLogFromTelemetry take 50% if istio beyond my understanding.

[howardjohn]

can you share more details about this? what the cluster's size? how many workloads? how many telemetry resources in the clusters?

Just 1 single telemetry. 5000 proxies. Access logs config are EXTREMELY expensive to generate since its so verbose and so many fields (envoyproxy/envoy#21718)

I created a WIP in https://github.com/istio/istio/compare/master...howardjohn:pilot/telemetry-cache?expand=1 but its not very good -- its hard to handle the packages the way here. I think we may not want to cache it as part of Telemetries in model...

[zirain]

if you change provider from envoy-json to envoy, still have the same cpu cost?

can you share the pprof?

[zirain]

A long-term goal is ecds: add support for access log extensions