First class API support for Envoy custom tags in tracing spans

[jacob-delgado]

Describe the feature request

Envoy supports custom tags to be defined on the server side proxy for adding additional metadata to the span.

To enable this feature in istio you must use a an EnvoyFilter. However, people are not comfortable using a potentially break glass feature and need a simplified API.

The API should support both environmental variables as well as extraction of custom headers.

Examples are shown below.

Environment variables

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: env-span
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      # context omitted so that this applies to both sidecars and gateways
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          tracing:
            custom_tags:
            - tag: "istio-env"
              environment:
                name: ISTIO_ENV
                default_value: "istio-env-default"

Where the tag "istio-env" is added to each request using the value from ISTIO_ENV; if none are present it will use "istio-env-default"

Custom headers

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: custom-header-span
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      # context omitted so that this applies to both sidecars and gateways
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          tracing:
            custom_tags:
            - tag: "from-custom-header"
              request_header:
                  name: "istio-header"
                  default_value: "istio-header-value"

Where the tag "from-custom-header" is added to each request using the value from the HTTP header "istio-header"; if "istio-header" is not present it will use "istio-header-value" string as its default.

Describe alternatives you've considered

None

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[X] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Additional context

Based on direct customer feedback.

[jacob-delgado]

CC @nrjpoddar @objectiser @douglas-reid

This is affected by the work defined for istio 1.6 regarding "Better API for controlling trace collection".

Other work that has been done in this area:
https://docs.google.com/document/d/1s7ScttcAfKL7g4ch5goWQYymUEghtTFiMvGPcdbTnpk/edit#heading=h.qex63c29z2to

https://docs.google.com/document/d/1FKGOionCwy8TR5OoMVzutTNwqYDJKQ_lrMIDaH6TJc4/edit#heading=h.qex63c29z2to

[kyessenov]

Please keep an eye on envoyproxy/envoy#9998 since we want to pick it up once it is available.

[jacob-delgado]

Please keep an eye on envoyproxy/envoy#9998 since we want to pick it up once it is available.

This issue is now closed. Planning is starting soon for 1.7

[jacob-delgado]

While not a first class API (that'll be part of Telemetry/Extensions) the deliverable is done for 1.6. Closing.

[jacob-delgado]

Verified in 1.6.0-beta.0