Severe memory leak on mixer 0.2.7 when driving load

[jnativio]

**Is this a BUG or FEATURE REQUEST?: BUG

Did you review existing epics or issues to identify if this already being worked on? (please try to add the correct labels and epics):

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version  0.2.7
kubectl version  
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.4-1+1540c973d4ff9d", GitCommit:"1540c973d4ff9da2d6a204a7709084488a459ed4", GitTreeState:"clean", BuildDate:"2017-10-06T15:08:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ?
No. Used istio.yaml

What happened:
I installed Istio on IBM Container Service (Armada) cluster with 3 worker nodes. All Istio pods in one of the nodes. I monitor the CPU and memory usage of nodes and pods while driving load to a microservices app benchmark. In the example below the load ran for about 40 min, and since the beginning the memory usage by the mixer kept growing. If the load continues, the node runs out of memory and kubernetes would restart all pods running there. After stopping the load, the memory usage goes down very slowly, and the CPU usage by the mixer pod is still very high.

What you expected to happen:

How to reproduce it:
Drive continuous load to microservices using Istio 0.2.7 and mixer enabled.

Feature Request:
N

Describe the feature:

[douglas-reid]

@jnativio did you happen to capture any charts related to number of goroutines, etc.?

also, would you mind characterizing the load a bit (rps, etc.)?

[mandarjog]

Thanks @jnativio
How long did you leave it running? You mentioned that memory goes down slowly once load stops, does it go back all the way to pre-run state?

[mandarjog]

Potentially related to istio/old_mixer_repo#1369
However the expected behaviour in the IL memory leak case is that memory usage will not go down as the load stops in absence of config changes.

[jnativio]

The chart below shows throughput and number of jmeter clients for the previous mixer memory/cpu charts, to answer the questions about of time and tps:

[jnativio]

I don't have any data at goroutines level. It appears mixer does not go back all the way to the same memory usage as when it first starts but stabilize at some higher value. The chart below shows its memory after 2 short runs:

[mandarjog]

@jnativio Can you get a heap dump for us.
port forward to mixer port 9093
and run

curl localhost:9093/debug/pprof/heap?debug=1
Thanks,

• Mandar

[jnativio]

@mandarjog I got two heap dumps: one before any load (freshly installed) and another after some load.

hd_before_load.txt
hd_after_load.txt

[mandarjog]

Thanks @jnativio

[mandarjog]

So far I am seeing a lot of allocations via opentracing/zipkin. Mixer does not sample traces, it is producing trace info for every request, so that needs to be addressed.
I would like to do a simple repro.