High CPU Utilization in Istio-Ingressgateway

[soloitguy]

Bug description
Exponential CPU Utilization in Istio Proxy after 1.3.*.

When upgrading from Istio 1.2.5 to 1.3.0, I noticed a dramatic increase in CPU Utilization in the istio-ingressgateway. I admittedly didn't look directly into it after the upgrade, just assumed there were some underlying changes that required a bit more CPU.

Then when deploying a new workload to the cluster, it failed due to unavailable CPU. I then ran a top on the pods and noticed that the HPA had scaled the pods up to the 5 maximum and each reaching close to +1000m CPU rather than the 10-100m that we saw normally.

Following some advice from a post on discuss.istio.io, I downgraded the istio-proxyV2 image to 1.2.8 and saw the CPU utilization drop back to the norm of a single pod and less than 100m CPU.

I don't want to have to keep an old version of this image going forward with upgrades. Is there any explanation of the increase?

Expected behavior
CPU Utilization remains the same or only slightly increases
Steps to reproduce the bug
Upgrade from 1.2* to 1.3.*
Version (include the output of istioctl version --remote and kubectl version)
istioctl version --remote

client version: 1.3.3
citadel version: 1.3.3
egressgateway version: 1.2.5
galley version: 1.3.3
ingressgateway version: 1.3.3
pilot version: 1.3.3
policy version: 1.3.3
sidecar-injector version: 1.3.3
telemetry version: 1.3.3

kubectl version

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm (template)
Environment where bug was observed (cloud vendor, OS, etc)
Azure AKS

[howardjohn]

Likely related to #18229

[howardjohn]

Has this been resolved in newer versions? I believe this is fixed now per above issue

[soloitguy]

It has! Thanks.