No way to add a custom bootstrap configuration in ingress gateway

[ankitpatel96]

Bug description
There's no way to add a custom bootstrap configuration to an ingress gateway. The recommended way (for sidecars apparently) is by adding an annotation to a pod with key sidecar.istio.io/bootstrapOverride. If you add try to add this to an ingress gateway, no bootstrap configuration is injected. This annotation should either work or another interface for gateways should be exposed.

Affected product area (please put an X in all that apply)

[ x] Docs
[x ] Installation
[ ] Networking
[ ] Performance and Scalability
[x] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Expected behavior
The bootstrap configuration should be built into a volume, mounted, and used by envoy.

Steps to reproduce the bug
Install Istio via Operator - two IstioOperator objects so I can control the ingressgateways in different IstioOperator objects.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istiocontrolplane
spec:
  profile: default
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: blah-ingressgateway
spec:
  profile: empty
  components:
    ingressGateways:
    - enabled: true
      name: blah-ingressgateway
      k8s:
        podAnnotations:
          sidecar.istio.io/bootstrapOverride: "blah-custom-bootstrap-config"

Then, make a bootstrap configuration with this configmap name:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    component: istio
  name: blah-custom-bootstrap-config
  namespace: istio-system
data:
  custom_bootstrap.json: |
    {
       "overload_manager":{
          "refresh_interval":{
             "seconds":1
          },
          "actions":[
             {
                "name":"envoy.overload_actions.shrink_heap",
                "triggers":[
                   {
                      "threshold":{
                         "value":0.90
                      },
                      "name":"envoy.resource_monitors.fixed_heap"
                   }
                ]
             },
             {
                "name":"envoy.overload_actions.stop_accepting_requests",
                "triggers":[
                   {
                      "threshold":{
                         "value":0.99
                      },
                      "name":"envoy.resource_monitors.fixed_heap"
                   }
                ]
             }
          ],
          "resource_monitors":[
             {
                "config":{
                   "max_heap_size_bytes":2147483648
                },
                "name":"envoy.resource_monitors.fixed_heap"
             }
          ]
       }
    }

Then, either examine the deployment or run istioctl proxy-config bootstrap pod-asdf to examine the bootstrap configuration. The bootstrap isn't injected in.

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)

$ istioctl version --remote
client version: 1.6.5
control plane version: 1.6.5
data plane version: 1.6.5 (12 proxies)
$ kubectl version --short
Client Version: v1.18.8
Server Version: v1.16.13-eks-2ba888

How was Istio installed?
IstioOperators as above

Environment where bug was observed (cloud vendor, OS, etc)
EKS v 1.16.13

[howardjohn]

This is documented in https://istio.io/latest/news/security/istio-security-2020-007/#mitigation but should probably be in samples/custom-bootstrap/ or improved

[ankitpatel96]

We operate with a GitOp style workflow with a whole bunch of different clusters so actually doing kubectl patch isn't a great solution for me - I managed to get this working by doing something like this horrifying blob:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: blah-ingressgateway
spec:
  profile: empty
  components:
    ingressGateways:
    - enabled: true
      name: blah-ingressgateway
      k8s:
        podAnnotations:
          sidecar.istio.io/bootstrapOverride: "blah-custom-bootstrap-config"
        overlays:
          - apiVersion: apps/v1
            kind: Deployment
            name: blah-ingressgateway
            patches:
              - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts
                value:
                  - name: istio-envoy
                    mountPath: /etc/istio/proxy
                  - name: config-volume
                    mountPath: /etc/istio/config
                  - name: istiod-ca-cert
                    mountPath: /var/run/secrets/istio
                  - name: istio-token
                    readOnly: true
                    mountPath: /var/run/secrets/tokens
                  - name: ingressgatewaysdsudspath
                    mountPath: /var/run/ingress_gateway
                  - name: podinfo
                    mountPath: /etc/istio/pod
                  - name: ingressgateway-certs
                    readOnly: true
                    mountPath: /etc/istio/ingressgateway-certs
                  - name: ingressgateway-ca-certs
                    readOnly: true
                    mountPath: /etc/istio/ingressgateway-ca-certs
                  - name: custom-bootstrap-volume
                    mountPath: /etc/istio/custom-bootstrap
              - path: spec.template.spec.volumes
                value:
                  - name: istiod-ca-cert
                    configMap:
                      name: istio-ca-root-cert
                      defaultMode: 420
                  - name: podinfo
                    downwardAPI:
                      items:
                        - path: labels
                          fieldRef:
                            apiVersion: v1
                            fieldPath: metadata.labels
                        - path: annotations
                          fieldRef:
                            apiVersion: v1
                            fieldPath: metadata.annotations
                      defaultMode: 420
                  - name: istio-envoy
                    emptyDir: {}
                  - name: ingressgatewaysdsudspath
                    emptyDir: {}
                  - name: istio-token
                    projected:
                      sources:
                        - serviceAccountToken:
                            audience: istio-ca
                            expirationSeconds: 43200
                            path: istio-token
                      defaultMode: 420
                  - name: config-volume
                    configMap:
                      name: istio
                      defaultMode: 420
                      optional: true
                  - name: ingressgateway-certs
                    secret:
                      secretName: istio-ingressgateway-certs
                      defaultMode: 420
                      optional: true
                  - name: ingressgateway-ca-certs
                    secret:
                      secretName: istio-ingressgateway-ca-certs
                      defaultMode: 420
                      optional: true
                  - name: custom-bootstrap-volume
                    configMap:
                      name: blah-custom-bootstrap-config
      label:
        app: blah-ingressgateway
  # Copy settings from istio-ingressgateway as needed.
  # these helm values are shared by all ingress/egress gateways
  values:
    gateways:
      istio-ingressgateway:
        env:
          ISTIO_BOOTSTRAP_OVERRIDE: /etc/istio/custom-bootstrap/custom_bootstrap.json
        podAnnotations:
          sidecar.istio.io/bootstrapOverride: "logstash-custom-bootstrap-config"

This might be improved by #26289? Trying to upgrade to 1.7 soon to find out
#27188 says I can use -1 to append which is really what I want.

Honestly it would be nice if istio just provided first class support for this...

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2020-10-27. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[ngtuna]

Should Istio support this via EnvoyFilter ?

[mattva01]

That would really help my usecase!

[marcbachmann]

This is supported since #33456 got merged. There are examples on this page: https://istio.io/latest/docs/reference/config/networking/envoy-filter/

At the moment only the MERGE operation is supported.

[oxxenix]

The issue should be reopened as the bootstrap config is not being updated when applying configuration with histogram buckets.
The applied Envoy Filter is: apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: stats namespace: istio-system spec: configPatches: applyTo: BOOTSTRAP patch: operation: MERGE value: stats_config: histogram_bucket_settings: match: contains: "xxx" - buckets: [2,3,4,5,6,7,8,9]
Istioctl versions are:
client version: 1.12.2
control plane version: 1.13
data plane version: 1.13-dev (2 proxies)

I would like to know how it can be fixed? Thanks.

[howardjohn]

You need BOOTSTRAP_XDS_AGENT env var set if you do not already

…

On Wed, Feb 16, 2022 at 12:36 AM Oksana Baranova ***@***.***> wrote: The issue should be reopened as the bootstrap config is not being updated when applying configuration with histogram buckets. The applied Envoy Filter is: `apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: stats namespace: istio-system spec: configPatches: - applyTo: BOOTSTRAP patch: operation: MERGE value: stats_config: histogram_bucket_settings: match: contains: "xxx" - buckets: [2,3,4,5,6,7,8,9] ` Istioctl versions are: client version: 1.12.2 control plane version: 1.13 data plane version: 1.13-dev (2 proxies) I would like to know how it can be fixed? Thanks. — Reply to this email directly, view it on GitHub <#28302 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXKHOXHGXBYTEZTQCGDU3NOZVANCNFSM4TAJIWOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>