Setting istio sidecar proxy resource request/limit #16126

sebarys · 2019-08-08T07:25:29Z

Hello :)

Describe the feature request

Is there an option to set istio proxy sidecar request/limits per application (Pod)? For some of our applications we would like to use Guaranteed QoS (https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) that require setting limit == request for all Pod containers. Without such functionality we're loosing this feature.

Describe alternatives you've considered

We've considered using global config value (https://istio.io/docs/reference/config/installation-options/#global-options) but if I understood correctly it could be used only during Istio installation, not applied for each Pod separately what don't match described case.
Some of our applications have bigger traffic so need higher requested resource values, other lower. As istio-proxy resources are somehow proportional to application container resources (e.g. one application is handling much bigger traffic than another -> it will require more resources also on istio-proxy sidecar layer) I would like to have an option to align resources assigned to isito-proxy per Pod.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[X] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

The text was updated successfully, but these errors were encountered:

howardjohn · 2019-08-08T14:37:53Z

You can set requests per proxy with sidecar.istio.io/proxyCPU and sidecar.istio.io/proxyMemory annotations, but doesn't look like we have one for limits.

sebarys · 2019-08-09T06:43:41Z

Thanks for information @howardjohn !
Is there any chance to add similar annotations for limits or it is already some work toward this way?

mafarinacci · 2019-08-09T22:34:46Z

I would also like this feature. I know what needs to be done to get this to work. Should I open a PR?

howardjohn · 2019-08-09T22:50:26Z

I remember @ostromart mentioning this, I forget if it was "we should do this" or "we should not do this".

If you do, you should submit the PR against https://github.com/istio/installer

mafarinacci · 2019-08-14T23:37:53Z

I think this is absolutely a needed feature. I want to get guaranteed cores on kubernetes nodes so that my low latency applications don't get throttled by kubernetes. It looks like me and another user have the same concern. I think we could talk about it at the next community meeting perhaps?

As far as implementation, that's actually not what I was thinking. this is what I think we should modify to do it on a per pod basis: https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/files/injection-template.yaml

Please lmk if I am incorrect. I will open a PR as soon as I get confirmation.

mafarinacci · 2019-08-15T23:23:09Z

@howardjohn howardjohn ^

anxolerd · 2019-08-28T07:36:01Z

This feature (per-pod resource limits for sidecar) would be useful for me as well. My concern is that when I override requests with annotation, sidecar limits are getting lost, which may lead to uncontrolled resource usage by sidecar, which I'd like to avoid.

ZackButcher · 2019-08-28T23:38:02Z

We would love to have a PR adding this. We should approach this by adding explicit flags to set resource limits, in addition to the current flags that sets resource requests. We should be able to both set a global default limit and a per-pod limit, like we do with resource requests today.

@mafarinacci we'll be happy to review a PR as soon as you've got it ready.

rakeshjain1213 · 2019-09-20T06:33:59Z

Hi,

Please let me know, if this feature is released in ISTIO 1.3.0 or still its in progress.

howardjohn · 2019-09-20T13:59:54Z

It did not make 1.3, you can set requests but not limits

pavelzhurov · 2019-10-03T11:42:10Z

@sebarys As far as I understood this annotation is set in the istio-sidecar-injector template. So, you can add any annotation that you want there. For example, I changed my template in this way:

  containers:
  - name: istio-proxy
…
    resources:
      limits:
        [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsCPU`) -]]
        cpu: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsCPU` ]]"
        [[ else -]]
        cpu: 200m
        [[ end ]]
        [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsMemory`) -]]
        memory: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsMemory` ]]"
        [[ else -]]
        memory: 256Mi
        [[ end ]]
      requests:
        [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyRequestsCPU`) -]]
        cpu: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyRequestsCPU` ]]"
        [[ else -]]
        cpu: 100m
        [[ end ]]
        [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyRequestsMemory`) -]]
        memory: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyRequestsMemory` ]]"
        [[ else -]]
       memory: 128Mi
        [[ end ]]

Thus, if I want fully custom memory/cpu request limits, I add following annotations to the deployment:

sidecar.istio.io/proxyRequestsCPU: "200m"
sidecar.istio.io/proxyRequestsMemory: "512Mi"
sidecar.istio.io/proxyLimitsCPU: "300m"
sidecar.istio.io/proxyLimitsMemory: "1Gi"

Warning: This example doesn't use global-options!!! Be careful, when you change your side-car template.

kebe7jun · 2019-10-17T09:17:11Z

emmm, can we set limits same as requests?

hobbytp · 2019-11-29T08:58:48Z

I have tried to modify istio-sidecar-injector.yaml as below, and then add annotation in application deployment, but the resource can work, and limits can not be added in istio-proxy, any comment?

    resources:
      {{ if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}}
      requests:
        {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}}
        cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}"
        {{ end}}
        {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}}
        memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}"
        {{ end }}
    {{ else -}}
  {{- if .Values.global.proxy.resources }}
      {{ toYaml .Values.global.proxy.resources | indent 4 }}
  {{- end }}
    {{  end -}}
      {{ if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsMemory`) -}}
      limits:
        {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsCPU`) -}}
        cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsCPU` }}"
        {{ end}}
        {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsMemory`) -}}
        memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyLimitsMemory` }}"
        {{ end }}
    {{ else -}}
  {{- if .Values.global.proxy.resources }}
      {{ toYaml .Values.global.proxy.resources | indent 4 }}
  {{- end }}
    {{  end -}}

sjmiller609 · 2019-12-12T14:10:42Z

I would also love for an annotation that helps to set the limit per-pod, but what about adding resource requests and limits to the Sidecar custom resource? It seems useful to be able to control the requests and limits using a workload selector.

howardjohn · 2019-12-12T16:18:00Z

@sjmiller609 that seems like it could be a pretty good UX but would be a pretty invasive change; right now the sidecar injection doesn't know about and CRs, so it doesn't read Sidecar resource. It could and maybe should, but doesn't today

howardjohn · 2019-12-12T16:18:11Z

I actually like that idea more as I think about it

howardjohn · 2019-12-12T20:37:20Z

Just so this doesn't get buried, I opened #19555 to track that idea. Its unlikely to be done in the short term though

sjmiller609 · 2019-12-13T18:11:30Z

Just so this doesn't get buried, I opened #19555 to track that idea. Its unlikely to be done in the short term though

That's ok, I really appreciate you taking the time to consider it!

prageethw · 2020-01-21T08:21:36Z

to set global restrictions... on sidecars

--set values.global.proxy.resources.limits.memory="300Mi"
--set values.global.proxy.resources.limits.cpu="500m" \

Arnold1 · 2020-02-12T14:15:57Z

@prageethw are there any default values for memory request/limit with Istio 4.1? if so what are those?
what about cpu?
and what happens if there is no memory limit set?

InsOpDe · 2020-02-12T20:45:00Z

I think its actually

--set global.proxy.resources.limits.memory="300Mi"
--set global.proxy.resources.limits.cpu="500m" \

and btw - we do not have the same problem with sidecars whose pods and services have no accompanied virtualservice/gateway.
maybe that helps with debugging.
Edit:
just to clarify, we do have a gateway per (virtual)service - which is as far as i know an antipattern we will get rid of soon.

sjmiller609 · 2020-02-12T20:51:43Z

@prageethw are there any default values for memory request/limit with Istio 4.1? if so what are those?
what about cpu?
and what happens if there is no memory limit set?

@Arnold1 you can find all defaults here: https://istio.io/docs/reference/config/installation-options/#global-options

rolandkool · 2020-03-12T17:00:16Z

Having the option to set the limits via an annotation would help us as well. If there is a limitrange and quota applied on the namespace and you override the CPU and memory requests, the limits are not set. This will cause pods to not start with errors such as

maximum cpu usage per Container is XXXX. No limit is specified., maximum memory usage per Container is YYYY. No limit is specified.

Usually when not setting requests/limits the defaults are applied, but perhaps there's an ordering issue with all the controllers modifying the spec that is causing this.

Patching the sidecar injector template ourselves is not ideal as this templates can change between versions.

* Add annotations for setting cpu/memory limits on sidecar When a limitrange is active, not setting the limits will result in an error. This patch will allow setting limits for the sidecar. Fixes #16126 Change-Id: I031d510812a867c2790eaa9af2f51145b4e5f006 * update test Change-Id: Ic610fc6e9c9a6d814083bd7434704592a5c8f92c

* Add annotations for setting cpu/memory limits on sidecar When a limitrange is active, not setting the limits will result in an error. This patch will allow setting limits for the sidecar. Fixes istio#16126 Change-Id: I031d510812a867c2790eaa9af2f51145b4e5f006 * update test Change-Id: Ic610fc6e9c9a6d814083bd7434704592a5c8f92c

* Add annotations for setting cpu/memory limits on sidecar When a limitrange is active, not setting the limits will result in an error. This patch will allow setting limits for the sidecar. Fixes #16126 * add auto generated files from make gen

nicktrav · 2020-04-15T16:16:36Z

@howardjohn - any chance of getting #22395 into 1.4 too?

webercoder · 2020-05-21T00:36:34Z

Any idea when PR #23053 might land in 1.4?

peiniliu · 2022-07-31T15:31:16Z

@sebarys
Hi Sebarys,
Did you finally have a Guaranteed Pod after injecting istio? I have followed those comments to set annotations for istio-proxy container, but the istio-init initcontainer is still a burstable container which will still cause the whole pod to be a burstable pod.
Is there any way to set the resource requests/limits for the istio-init container?
Best,
Peini

kumarganesh2814 · 2022-08-16T17:14:21Z

Do we have anything yet for istio-proxy, I tried adding annotation my particular deployment but limits remained same also requests

RicHincapie · 2022-09-08T22:34:33Z

As of Istio 1.10, the solution is already implemented and its pretty similar to @pavelzhurov post. The istio-sidecar-inejctor template ConfigMap has the limits and resources parameters that allows us to set these parameters on a per-deployment basis. Those params are annotations at the deployment manifest level and look like this:

.spec.template.metadata.annotations:
sidecar.istio.io/proxyCPU: 600m
sidecar.istio.io/proxyCPULimit: 600m
sidecar.istio.io/proxyMemory: 256Mi
sidecar.istio.io/proxyMemoryLimit: 256Mi

With this config, you may be able to have QoS Guaranteed class for your pods.

peiniliu · 2022-09-09T06:58:34Z

Hi ricarhincapie,

Thanks for your suggestion, you are right after setting these annotations, the istio-proxy container will have the same cpu/memory request and limit. But note that the pod is still not a QOS Guaranteed pod, because the istio-init has burstable settings which will let this pod finally becomes a QOS Burstable pod.

Also some discussion about the initcontainer here k8s-initcontainer-discussion.

I would like to find a way to set the resource requests/limits for the istio-init container as well :)

Best,
Peini

howardjohn · 2022-09-09T13:32:01Z

I thought the init container uses identical resources as sidecar?

…

On Thu, Sep 8, 2022, 11:58 PM Peini Liu ***@***.***> wrote: Hi ricarhincapie, Thanks for your suggestion, you are right after setting these annotations, the istio-proxy container will have the same cpu/memory request and limit. But note that the pod is still *not a QOS Guaranteed* pod, because the istio-init has burstable settings which will let this pod finally becomes a *QOS Burstable* pod. Also some discussion about the initcontainer here k8s-initcontainer-discussion <kubernetes/kubernetes#93282>. I would like to find a way to set the resource requests/limits for the istio-init container as well :) Best, Peini — Reply to this email directly, view it on GitHub <#16126 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXPXEOI36HWWBZECMILV5LN2LANCNFSM4IKHG5UQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

peiniliu · 2022-09-12T12:57:11Z

I thought the init container uses identical resources as sidecar?
…
On Thu, Sep 8, 2022, 11:58 PM Peini Liu @.> wrote: Hi ricarhincapie, Thanks for your suggestion, you are right after setting these annotations, the istio-proxy container will have the same cpu/memory request and limit. But note that the pod is still not a QOS Guaranteed pod, because the istio-init has burstable settings which will let this pod finally becomes a QOS Burstable pod. Also some discussion about the initcontainer here k8s-initcontainer-discussion <kubernetes/kubernetes#93282>. I would like to find a way to set the resource requests/limits for the istio-init container as well :) Best, Peini — Reply to this email directly, view it on GitHub <#16126 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXPXEOI36HWWBZECMILV5LN2LANCNFSM4IKHG5UQ . You are receiving this because you were mentioned.Message ID: @.>

Yes, I confirm with the setting the pod can be QoS Guaranteed. initcontainer has the same resources as sidecar. Thanks all.

When a limitrange is active, not setting the limits will result in an error. This patch will allow setting limits for the sidecar. Fixes istio/istio#16126

istio-policy-bot added area/perf and scalability kind/enhancement labels Aug 8, 2019

kebe7jun mentioned this issue Oct 17, 2019

sidecar.istio.io/proxyCPU annotation only for requests #17974

Closed

istio-policy-bot added the lifecycle/needs-triage label Oct 30, 2019

rolandkool mentioned this issue Mar 13, 2020

Add annotations for setting cpu/memory limits on sidecar #22155

Closed

This was referenced Mar 23, 2020

Add annotations for setting cpu/memory limits on sidecar #22395

Merged

Document new sidecar annotations for setting resource limits istio/api#1346

Merged

istio-testing closed this as completed in #22395 Mar 24, 2020

rolandkool mentioned this issue Mar 25, 2020

[release-1.5] Add annotations for setting cpu/memory limits on sidecar #22468

Closed

rolandkool mentioned this issue Apr 7, 2020

Add annotations for setting cpu/memory limits on sidecar #22787

Merged

rolandkool mentioned this issue Apr 18, 2020

[release 1.4] Add annotations for setting cpu/memory limits on sidecar #23053

Closed

kfhanjie mentioned this issue Mar 31, 2022

sidecarset support different resource config for diffrent app openkruise/kruise#942

Closed

skonto mentioned this issue Apr 10, 2023

Allow a more flexible configuration of queue proxy resources knative/serving#13861

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setting istio sidecar proxy resource request/limit #16126

Setting istio sidecar proxy resource request/limit #16126

sebarys commented Aug 8, 2019 •

edited

Loading

howardjohn commented Aug 8, 2019

sebarys commented Aug 9, 2019

mafarinacci commented Aug 9, 2019

howardjohn commented Aug 9, 2019

mafarinacci commented Aug 14, 2019

mafarinacci commented Aug 15, 2019

anxolerd commented Aug 28, 2019

ZackButcher commented Aug 28, 2019

rakeshjain1213 commented Sep 20, 2019

howardjohn commented Sep 20, 2019

pavelzhurov commented Oct 3, 2019 •

edited

Loading

kebe7jun commented Oct 17, 2019

hobbytp commented Nov 29, 2019 •

edited

Loading

sjmiller609 commented Dec 12, 2019 •

edited

Loading

howardjohn commented Dec 12, 2019

howardjohn commented Dec 12, 2019

howardjohn commented Dec 12, 2019

sjmiller609 commented Dec 13, 2019

prageethw commented Jan 21, 2020

Arnold1 commented Feb 12, 2020 •

edited

Loading

InsOpDe commented Feb 12, 2020 •

edited

Loading

sjmiller609 commented Feb 12, 2020

rolandkool commented Mar 12, 2020 •

edited

Loading

nicktrav commented Apr 15, 2020 •

edited

Loading

webercoder commented May 21, 2020

peiniliu commented Jul 31, 2022

kumarganesh2814 commented Aug 16, 2022

RicHincapie commented Sep 8, 2022

peiniliu commented Sep 9, 2022

howardjohn commented Sep 9, 2022 via email

peiniliu commented Sep 12, 2022

Setting istio sidecar proxy resource request/limit #16126

Setting istio sidecar proxy resource request/limit #16126

Comments

sebarys commented Aug 8, 2019 • edited Loading

howardjohn commented Aug 8, 2019

sebarys commented Aug 9, 2019

mafarinacci commented Aug 9, 2019

howardjohn commented Aug 9, 2019

mafarinacci commented Aug 14, 2019

mafarinacci commented Aug 15, 2019

anxolerd commented Aug 28, 2019

ZackButcher commented Aug 28, 2019

rakeshjain1213 commented Sep 20, 2019

howardjohn commented Sep 20, 2019

pavelzhurov commented Oct 3, 2019 • edited Loading

kebe7jun commented Oct 17, 2019

hobbytp commented Nov 29, 2019 • edited Loading

sjmiller609 commented Dec 12, 2019 • edited Loading

howardjohn commented Dec 12, 2019

howardjohn commented Dec 12, 2019

howardjohn commented Dec 12, 2019

sjmiller609 commented Dec 13, 2019

prageethw commented Jan 21, 2020

Arnold1 commented Feb 12, 2020 • edited Loading

InsOpDe commented Feb 12, 2020 • edited Loading

sjmiller609 commented Feb 12, 2020

rolandkool commented Mar 12, 2020 • edited Loading

nicktrav commented Apr 15, 2020 • edited Loading

webercoder commented May 21, 2020

peiniliu commented Jul 31, 2022

kumarganesh2814 commented Aug 16, 2022

RicHincapie commented Sep 8, 2022

peiniliu commented Sep 9, 2022

howardjohn commented Sep 9, 2022 via email

peiniliu commented Sep 12, 2022

sebarys commented Aug 8, 2019 •

edited

Loading

pavelzhurov commented Oct 3, 2019 •

edited

Loading

hobbytp commented Nov 29, 2019 •

edited

Loading

sjmiller609 commented Dec 12, 2019 •

edited

Loading

Arnold1 commented Feb 12, 2020 •

edited

Loading

InsOpDe commented Feb 12, 2020 •

edited

Loading

rolandkool commented Mar 12, 2020 •

edited

Loading

nicktrav commented Apr 15, 2020 •

edited

Loading