Envoy Stat Filter removed from Istio version 1.21.2

[RishabhKumarGupta]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

With the new version of istio 1.21.2 that deprecated EnvoyFilter implementation for Telemetry v2 and that removed stat Filter as well.
Trying to migrate the below envoyfilter to use telemetryAPI

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: stats-filter-1.20
  namespace: istio-system
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
        proxy:
          proxyVersion: ^1\.20.*
      patch:
        operation: INSERT_BEFORE
        value:
          name: istio.stats
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/stats.PluginConfig
            value: {}
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
        proxy:
          proxyVersion: ^1\.20.*
      patch:
        operation: INSERT_BEFORE
        value:
          name: istio.stats
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/stats.PluginConfig
            value:
              debug: false
              disable_host_header_fallback: true
              metrics:
                - dimensions:
                    request_operation: istio_operationId
              stat_prefix: istio
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
        proxy:
          proxyVersion: ^1\.20.*
      patch:
        operation: INSERT_BEFORE
        value:
          name: istio.stats
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/stats.PluginConfig
            value:
              disable_host_header_fallback: true
  priority: -1

---------

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: REQUEST_COUNT
      tagOverrides:
        request_operation:
          value: "istio_operationId"

Not sure if there a way to use telemetryAPI for istio.stats having value disable_host_header_fallback: true and how to use this for Gateway ?
If this right I am approaching i need to migrate the envoyfilter to Telemetry API because when the stat-filter-1.* are removed , application is not able to scrape metrics from some endpoints and also i dont see any metrics istio_request_total {}

Stats not present using this command: istioctl pc l -oyaml <your_pod_name> | grep "istio.stats"

Or

I can still use these envoyfilter (stat-filter-1.* , tcp-stat-filter-1.) by changing the proxyversion to 1.21. ?

Version

istio version 1.21.2 in ubuntu
minikube version: v1.32.0
Client Version: v1.28.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Additional Information

No response

[RishabhKumarGupta]

@howardjohn @zirain Can you take a look please

[RishabhKumarGupta]

Can someone check on this please.

[zirain]

you can remove this default providers and still use the EnvoyFilter way.

[RishabhKumarGupta]

Thanks @zirain for the reply , by default provider you mean that we can disable v2 from the telemetry configuration that will remove stats filter , tcp-stats-filter but I can create envoyfilter using the same configuration of stats filter, tcp-stats filter by changing the proxy version. Correct ?

[zirain]

https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-DefaultProviders

[RishabhKumarGupta]

@zirain I am using defaultprovider as tracing in the istio configuration

meshConfig:
defaultProviders:
tracing:
- opentelemetry
extensionProviders:
- name: "opentelemetry"
opentelemetry:
service: "opentelemetry-collector.lightops-obs-agent.svc.cluster.local"
port: 4317
maxTagLength: 256

Why do i need to remove this ?

I have tested to remove that Telemetry API "default" which i created manually and work with envoyfilter using the same configuration of stats filter, tcp-stats filter it is working for me. Can i go with this ?

[zirain]

I just confirmed that disableHostHeaderFallback is default true on Inbound and Gateway context.

[RishabhKumarGupta]

@zirain, Let me test by removing configpatches for disableHostHeaderFallback is true in EnvoyFilter.

[nicole-lihui]

I have verified that after deprecating the built-in stats EnvoyFilter in Istio, configuring only Telemetry config is sufficient to collect metrics data.

If both the stats EnvoyFilter and Telemetry are present simultaneously, it will result in duplicated metric data. 👀

I suspect that the lack of metric data in your current environment is not due to the deprecation of the EnvoyFilter, and suggest checking the sidecar XDS configuration to ensure that the metrics Telemetry settings are effective.

[nicole-lihui]

Additionally, meshConfig.tracing is a configuration unrelated to metrics and should not be removed.

[RishabhKumarGupta]

Hi @nicole-lihui , Thanks for the inputs.

When I am trying to use below Telemetry getting an error in the application proxy logs: "failed scraping application metrics: error scraping http://localhost:8080/actuator/prometheus: Get "http://localhost:8080/actuator/prometheus\": dial tcp [::1]:8080: connect: connection refused"} .

I need to add request_operation tag in metrics: REQUEST_COUNT

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: REQUEST_COUNT
      tagOverrides:
        request_operation:
          value: "istio_operationId"

And isito_request_total logs are not generating ; Below is configuration of istio that i am overiding in main helm values:

meshConfig:
    accessLogEncoding: JSON
    accessLogFile: /dev/stdout
    accessLogFormat: |
      {
        "authority": "%REQ(:AUTHORITY)%",
        "bytes_received": "%BYTES_RECEIVED%",
        "bytes_sent": "%BYTES_SENT%",
        "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
        "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
        "duration": "%DURATION%",
        "method": "%REQ(:METHOD)%",
        "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        "protocol": "%PROTOCOL%",
        "request_id": "%REQ(X-REQUEST-ID)%",
        "requested_server_name": "%REQUESTED_SERVER_NAME%",
        "response_code": "%RESPONSE_CODE%",
        "response_flags": "%RESPONSE_FLAGS%",
        "route_name": "%ROUTE_NAME%",
        "start_time": "%START_TIME%",
        "trace_id": "%REQ(X-B3-TRACEID)%",
        "traceparent": "%REQ(TRACEPARENT)%",
        "upstream_cluster": "%UPSTREAM_CLUSTER%",
        "upstream_host": "%UPSTREAM_HOST%",
        "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
        "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
        "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
        "user_agent": "%REQ(USER-AGENT)%",
        "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
      }
    enableTracing: true
    defaultConfig:
      holdApplicationUntilProxyStarts: true
      extraStatTags:
        - request_operation
      tracing:
        zipkin:
          address: opentelemetry-collector.lightops-obs-agent:9411
    defaultProviders:
      tracing:
        - opentelemetry
    extensionProviders:
      - name: "opentelemetry"
        opentelemetry:
          service: "opentelemetry-collector.lightops-obs-agent.svc.cluster.local"
          port: 4317
          maxTagLength: 256
      - envoyExtAuthzHttp:
          headersToDownstreamOnDeny:
            - content-type
            - set-cookie
          headersToUpstreamOnAllow:
            - authorization
            - path
            - x-auth-request-user
            - x-auth-request-email
            - x-auth-request-access-token
            - x-auth-request-groups
          includeRequestHeadersInCheck:
            - authorization
            - cookie
            - x-forwarded-proto
          includeAdditionalHeadersInCheck:
            x-forwarded-host: '%REQ(:authority)%'
            x-auth-request-redirect: '%REQ(x-forwarded-proto)%://%REQ(:authority)%%REQ(:path)%'
          port: 4180
          service: oauth2-proxy.oauth2-proxy.svc.cluster.local
        name: oauth2-proxy
  telemetry:
    enabled: true
    v2:
      enabled: false
      prometheus:
        enabled: false

Can you give me some idea why it is not working with Telemetry and working with below EnvoyFilter using istio.stats

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
 name: envoy-stats-filter-1.21
 namespace: {{ .Release.Namespace }}
 labels:
   istio.io/rev: default
spec:
 configPatches:
   - applyTo: HTTP_FILTER
     match:
       context: SIDECAR_OUTBOUND
       listener:
         filterChain:
           filter:
             name: envoy.filters.network.http_connection_manager
             subFilter:
               name: envoy.filters.http.router
       proxy:
         proxyVersion: ^1\.21.*
     patch:
       operation: INSERT_BEFORE
       value:
         name: istio.stats
         typed_config:
           '@type': type.googleapis.com/udpa.type.v1.TypedStruct
           type_url: type.googleapis.com/stats.PluginConfig
           value: {}
   - applyTo: HTTP_FILTER
     match:
       context: SIDECAR_INBOUND
       listener:
         filterChain:
           filter:
             name: envoy.filters.network.http_connection_manager
             subFilter:
               name: envoy.filters.http.router
       proxy:
         proxyVersion: ^1\.21.*
     patch:
       operation: INSERT_BEFORE
       value:
         name: istio.stats
         typed_config:
           '@type': type.googleapis.com/udpa.type.v1.TypedStruct
           type_url: type.googleapis.com/stats.PluginConfig
           value:
             debug: false
             metrics:
               - dimensions:
                   request_operation: istio_operationId
             stat_prefix: istio
 priority: -1

[zirain]

please check the configuration first: istioctl pc l <podname>.<pod_namespace> -oyaml | grep -C 5 istio.stats

BTW: the mertic will be generated after real traffic happen.

[RishabhKumarGupta]

@zirain I am not getting any output using the above istioctl pc command since it doesnot have istio.stats using Telemetry . Yes i am testing in dev environment where i have some clients working on it.

[RishabhKumarGupta]

@zirain @nicole-lihui Can anyone provide your inputs here

[RishabhKumarGupta]

@zirain @nicole-lihui Can you please provide some outputs why I am facing issue with Telemetry.

[nicole-lihui]

I'm a little confused, do you want to use telemetry or envoyfilter? IMO, you shouldn't mix the two parts.

if you want only use telemetry, you shouldn't do any things. only just need a Telemetery CRD.

for https://istio.io/v1.21/docs/tasks/observability/metrics/telemetry-api/ .

[RishabhKumarGupta]

Thanks @nicole-lihui , I am able to use Telemetry API now.