-
Notifications
You must be signed in to change notification settings - Fork 7.7k
-
Notifications
You must be signed in to change notification settings - Fork 7.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Approx. 20% increase in istio-proxy memory usage on 1.7 vs 1.6 #28652
Comments
I'm going to document the things I check here as I go. There are some new metrics in the 1.7 envoy:
It looks as if locality metrics are now enabled by default, which is a bit annoying as we don't need them, and also, they're still in a strange broken statsd format (the Either way, i wouldn't consider this a large amount of additional metrics so wouldn't account for the 10mb base jump in proxies. |
Seems to show the increase is in |
Associated: #26232 |
istio-agent increased in memory because xDS proxy was added, which requires importing a whole bunch of protobufs from envoy. I think the question is whether the value added by the proxy justifies the cost. |
Is xds proxy already present in 1.7? |
multiple questions
|
On 3 month old memory: when looking into this, its extremely challenging to reduce the binary size in 1.7 In 1.9 we will drop the k8s imports, which I have done locally, and its 40mb, so we definitely have a long term path forward. Also the |
There's a budget of how much users are going to tolerate per-pod overhead. Keep in mind Wasm overhead and whether it's better to use the budget for xDS proxy or Wasm. |
We can reduce the overhead of XDS proxy. We don't unmarhsal Any so we don't need to import all the filters. The bare cost of just importing core XDS is not that high |
I'm not sure the point here, you can equally consider the other extreme - where you've got people building golang microservices running @ 10-20mb, adding istio @ 50-60mb to their application architecture is a significant increase. The other key word there is microservices. Larger proxies will favour people with larger/monolithic services, which is counter-productive when one of istios biggest selling points is the observability it gives you when you're building large microservice architectures!
"25% increase in memory usage of the data plane" rightly paints a very different picture in peoples heads than "15mb increase in memory usage per proxy". So that is why the percentage matters, because it more accurately depicts the impact to the users regardless of their number of proxies. In real terms for me, that's over a 15gb increase cluster wide which isn't something to be sniffed at. Especially considering we've just took 10gb increase just to facilitate SDS. So in the course of two releases we've not far off doubled the operating overhead (memory) of Istio. This means proxy memory usage now accounts for 8-10% of all our memory usage. |
I'm going to close this in favour of #26232 which seems to be tracking the regression and all associated fixes. @howardjohn has also back ported #28670 to 1.7 which is awaiting merge which will help a bit with the bloat on 1.7 (believe to land roughly half way between what I've observed here and 1.6). I think; for those coming into this issue, we'll have to accept some increase in 1.7, with many improvements coming in 1.8 and 1.9. |
Bug description
Hi,
I've been chatting to @howardjohn about this, and felt it warrants a github issue to track.
We've recently upgrade a staging cluster which has circa 500 istio-proxies on it from 1.6.13 to 1.7 and once we'd updated all sidecars we observed about a 20-25% increase in istio-proxy memory cluster wide:
On top of the 10% added going from 1.5 -> 1.6 due to SDS, these increases are getting hard to stomach.
We can compare this cluster with our production cluster. They're identical other than traffic patterns and number of endpoints (the 1.6 cluster having twice as many endpoints and significantly more load - so if anything, it should be higher).
Looking at the
min
across the board gives a decent indication of proxy memory usage before they start taking load, you can see themin
on 1.7 is around the 50mb mark:Whereas on 1.6 it's more like 40mb:
For comparison purposes, we run a service on all of our clusters call
istio-test
, as you can see here on the 1.7 cluster we're around 50mb average usage:Vs. the 1.6 cluster which is around 40mb:
Both are configured identically as their purpose is to compare istio releases.
[ ] Docs
[ ] Installation
[ ] Networking
[x] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure
Expected behavior
Not a 20% increase in memory between releases.
Steps to reproduce the bug
Version (include the output of
istioctl version --remote
andkubectl version --short
andhelm version
if you used Helm)1.7.5-4d93d71598da6e07cf9c68e32aaa0c4eadc308a4
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
GKE
The text was updated successfully, but these errors were encountered: