High istiod/discovery memory usage since upgrade to 1.10

[Stono]

Bug description
We recently upgraded from 1.9.5 to 1.10.1, and we are finding the memory usage of istiod much higher and quite unstable, as can be seen from the graph here:

This is in mb, so we're seeing 35%ish swings on the average. The average has gone from 400mb to 1.3gb, and we're peaking as high as 1.7gb.

Zooming in to the lower time frames, and a single pod we see:

07:00 is roughly the time of day our cluster starts taking deployments from developers, so i'd say this memory usage is related to config pushes, but the times don't really marry with the istiod logs:

logs.log.tar.gz

[ ] Docs
[ ] Installation
[ ] Networking
[x] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

Expected behavior
Memory to be stable, and memory increases in istiod to be acknowledged as expected in release notes if they are indeed expected.

Steps to reproduce the bug
Unsure, obviously we have a big deployment with a lot of configuration, and welcome anyone to live debug with me.

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)
1.10.1

How was Istio installed?

Environment where the bug was observed (cloud vendor, OS, etc)

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.

[howardjohn]

If you could get a memory profile it would be helpful: https://github.com/istio/istio/wiki/Analyzing-Istio-Performance#memory.

[howardjohn]

The only change that stands out to me is go 1.16 upgrade which has changes to how memory is managed/reported: https://golang.org/doc/go1.16#runtime

[Stono]

Here you go @howardjohn

pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz

And FWIW these are kubernetes reported memory usage metrics, not golang

[Stono]

Been working with @howardjohn on slack on this and tested out a variety of things.

We correlated the memory increases to whenever a full push happened, that links to the deployments above as they were modifying the DestinationRule

One thing we've discovered is that using an istiod image built against 1.15 golang seems to solve the problem, here you can see how memory is lower, and not at all spiky after switching and forcing lots of full pushes (by editing the istiod configmap):

Johns suspicions came from golang/go#44554, albeit the environmental workaround in there GODEBUG=madvdontneed=0 doesn't resolve it.

[Stono]

@howardjohn - i'm assuming, but could you confirm that istio-proxy is also targeting 1.16 golang now? I ask because our operations teams have been dealing with numerous high memory alerts for proxies since the 1.10 upgrade.

Our upgrade went out on the 21st so it does broadly look like things are trending up in the wrong direction, albeit not by huge amounts.

NOTE This graph shows the previous 1.9.x upgrade on june the 11th as well, so you can see the differences in immediate and growth memory there.

[howardjohn]

Yes, everything is 1.16 now

…

On Mon, Jun 28, 2021 at 4:10 AM Karl Stoney ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> - i'm assuming, but could you confirm that istio-proxy is also targeting 1.16 golang now? I ask because our operations teams have been dealing with numerous high memory alerts for proxies since the 1.10 upgrade. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33569 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXK75X7FED32IL2FRPTTVBKC3ANCNFSM47DLFOXA> .

[Stono]

Hey, just wondering if there's any progression with this? Are you using anything specifically 1.16? Wondering if a rollback to 1.15 until the memory profile is understood makes sense?

I ask because i need to know if i need to go and adjust our expectations/alerts around proxy and control plane memory usage

[howardjohn]

My suspicion is still that this is a metric reporting discrepancy, rather than a real regression in memory usage. But we need to prove that. Until we do that it would be preferred to not revert, as we do depend on 1.16 for some features now, our dependencies (ie k8s) are using it so it has the most testing, and it has been shown to have concrete performance benefits in many other areas. We should leave the option open of course, but i am hoping it does not come to that

One thing I have noticed:

All memory stats are ~the same between versions except the one you are looking at (working_set_bytes). With 1.16 (v10 on the graph), it is identical to container_memory_usage_bytes

[howardjohn]

working_set_bytes is https://github.com/google/cadvisor/blob/50b23f4ed9bc53cf068316b67bee04c4145f1e73/container/libcontainer/handler.go#L862
Usage-total_inactive_file

Which for different versions is seen as:

v10: 447827968-0
v9: 690978816-574328832

So golang 1.16 is actually using less memory total, but is reporting none of it as inactive.

Interestingly, here is the two after disconnecting all proxies:

The go 1.15 pod never actually returns the memory to the OS

[howardjohn]

From kubernetes-sigs/metrics-server#359 it seems like looking at rss may be more appropriate. I have seen suggestions of the opposite (https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66, https://faun.pub/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d) but those seem to be prior to that PR.

However, looking at kubectl top it does seem to be working_set, not RSS, that is reported. I believe that PR is in v0.4.0 metric server, while GKE (v1.20.6) has v0.3.6, which doesn't have that PR.

If I am reading this right:

The OOMKill happens when rss/memory_usage exceeds limit (2GB here); working_set is still far below.

[Stono]

They all report pretty much the same for me...