Allow compression for the Envoy stats endpoint

[svenwltr]

Describe the feature request

It should be possible to enable gzip compression for the Envoy stats endpoint (http://127.0.0.1:15090/stats/prometheus) to save some bandwidth. Currently it does not seem to be the case:

$ kubectl exec -it my-pod-0 -c server -- curl -o /dev/null -vsS --compressed http://127.0.0.1:15090/stats/prometheus
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15090 (#0)
> GET /stats/prometheus HTTP/1.1
> Host: 127.0.0.1:15090
> User-Agent: curl/7.61.1
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< cache-control: no-cache, max-age=0
< x-content-type-options: nosniff
< date: Fri, 19 Feb 2021 10:42:25 GMT
< server: envoy
< x-envoy-upstream-service-time: 2
< transfer-encoding: chunked
< 
{ [26267 bytes data]
* Connection #0 to host 127.0.0.1 left intact

Describe alternatives you've considered

I tried configuring an EnvoyFilter to enable compression, but I lack knowledge of the Envoy internals and the filter concept.

Details

I tried crafting a filter filter, but it does not work. Disclaimer: This is also a Stackoverflow question. This is my latest attempt:

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gzip
spec:
  workloadSelector:
    labels:
      app: my-pod
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
              subFilter:
                name: envoy.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.compressor
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
            compressor_library:
              name: text_optimized
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
            remove_accept_encoding_header: true

[ ] Docs
[ ] Installation
[x] Networking
[x] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Additional context

Currently, we have around 100 Pods with an Istio sidecar and we are aiming to increase that number. For scaling reasons we did split our Prometheus instances into two parts: One for the Envoy Metrics and one for the actual application metrics. This means we disabled metric merging and are pointing one Prometheus instance directly to :15090/stats/prometheus endpoint of each pod.

[bianpengyuan]

I don't think Envoy supports compressed stats. The gzip HTTP filter you are trying to configure does not compress stats. It is used for request compression. It would probably be better to open an issue at upstream Envoy first. Istio should be able to adopt this with no problem.

[howardjohn]

The envoyfilter doesn't work since the stats listener is static configuration.

@bianpengyuan what is the difference between request compression and stats compression?

[bianpengyuan]

The envoyfilter doesn't work since the stats listener is static configuration.

Ah I see, the intent is to inject compression into the stats listener. I was thinking supporting compression directly at Envoy's prometheus endpoint, based on content type requested by the client. That way there is no extra filter needed.

[svenwltr]

For clarification: We want to scrape compressed metrics directly from the istio-proxy pod. I don't know what possible options we have to achieve this.

So according to your comments I have to open an Envoy issue, right?

[howardjohn]

You just need to add the gzip filter directly to the 15090 listener in the bootstrap. You can see an example of this in samples/custom-bootstrap

[svenwltr]

This means we have to:

1. render the built-in bootstrap template (not yet sure how to do this),
2. patch the result,
3. add the annotation to every pod
4. and repeat that on every Istio upgrade to make sure it does not diverge too much?

I tried to use the proxyBootstrapTemplatePath for the proxy config, but I do not see an easy way to mount the patched template to the istiod pods.

Would it be okay to create a PR that adds this to the original bootstrap template?

[bianpengyuan]

This means we have to:

1. render the built-in bootstrap template (not yet sure how to do this),
2. patch the result,
3. add the annotation to every pod
4. and repeat that on every Istio upgrade to make sure it does not diverge too much?

I tried to use the proxyBootstrapTemplatePath for the proxy config, but I do not see an easy way to mount the patched template to the istiod pods.

Would it be okay to create a PR that adds this to the original bootstrap template?

You only need to provide the patch itself with custom bootstrap. No need to provide the whole patched Envoy bootstrap. And the patch needs to be mounted to the workload pod instead of istiod.

[svenwltr]

Thanks @bianpengyuan. I completely missed that those values would be merged. Thank you.

I got a working example and our network usage went down from ~20MBytes/s to ~30KBytes/s (yes, from Mega to Kilo 🔥). First I thought there was any error, but the data was complete and I did a short check with my CLI:

$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail --compressed -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1
7763

$ kubectl exec elasticsearch-0 -c istio-proxy -- timeout 1 curl -Ss --fail -w '%{size_download}' -i http://localhost:14090/stats/prometheus | tail -n 1          
330315

It is only 2.35% of its original size and both have the same mount of lines!

The problem is now solved for us, but it might make sense to add the compression feature to the upstream bootstrap templates.

Here is our configuration for completeness.

apiVersion: v1
kind: ConfigMap

metadata:
  annotations:
  name: istio-custom-bootstrap-config
  namespace: default

data:
  custom_bootstrap.json: |-
    {
        "staticResources": {
            "listeners": [
                {
                    "address": {
                        "socketAddress": {
                            "address": "0.0.0.0",
                            "portValue": 14090
                        }
                    },
                    "filterChains": [
                        {
                            "filters": [
                                {
                                    "name": "envoy.filters.network.http_connection_manager",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                                        "httpFilters": [
                                            {
                                                "name": "envoy.filters.http.compressor",
                                                "typed_config": {
                                                    "@type": "type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor",
                                                    "compressor_library": {
                                                        "name": "text_optimized",
                                                        "typed_config": {
                                                            "@type": "type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip"
                                                        }
                                                    },
                                                    "remove_accept_encoding_header": true
                                                }
                                            },
                                            {
                                                "name": "envoy.filters.http.router",
                                                "typedConfig": {
                                                    "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                                                }
                                            }
                                        ],
                                        "routeConfig": {
                                            "virtualHosts": [
                                                {
                                                    "domains": [
                                                        "*"
                                                    ],
                                                    "name": "backend",
                                                    "routes": [
                                                        {
                                                            "match": {
                                                                "prefix": "/stats/prometheus"
                                                            },
                                                            "route": {
                                                                "cluster": "prometheus_stats"
                                                            }
                                                        }
                                                    ]
                                                }
                                            ]
                                        },
                                        "statPrefix": "stats"
                                    }
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    }

[bianpengyuan]

@svenwltr Thanks for providing the configuration! It will be useful for future referencing.

We will consider have this as an option when more people want this. Thanks!

[DimiDr0l]

GZIP in sidecar metrics is cool

[taisph]

FWIW, the cumbersome workaround using a secondary metrics port with a gzip compression filter, greatly reduced interzone traffic in our cluster resulting in about €300 per month savings in cloud spending.

[aranair]

gzip support really needs to be the default 💦

[msvticket]

I'm trying to get compression working for merged metrics. I started with the configuration described in the comment above. This made it possible to scrape compressed metrics on port 14090.

What I don't understand is how to route the traffic for port 15020 to this listener. Can anybody give me a hint?

[czenker]

Just to let everyone know, that this feature was released with Istio 1.21.0 (#47997) and it is really easy now. Thanks @zirain et al!

Add the annotation sidecar.istio.io/statsCompression: gzip to your pods and that's it. The istio-sidecar-injector can add this annotation automatically, if you configure it. It currently does not work with merged metrics and Prometheus does not support zstd compression, only gzip at the moment.