Support customized tags from request header for tracing

[morvencao]

Describe the feature request
Envoy v2 API supports request_headers_for_tags, which can be used to create tags for the active span from request headers, but looks look like Istio currently doesn't support to generate such envoy configuration.

However, this feature is useful if the operators expect to differentiate tracing data from different application.

Describe alternatives you've considered
Expect Istio to support new API or enhance current tracing API to support envoy request_headers_for_tags configuration.

/cc @linsun @gyliu513 @clyang82

[objectiser]

@douglas-reid This is another area to consider for the tracing API - as it would tend to be service specific.

[morvencao]

@douglas-reid Any thought?

[lucasponce]

+1
I've found just a similar request for this issue within Kiali.
From Kiali, I want to create a matching rule based in a header that might be or not be part of a span.
(As developer could not add it).
But as it's used inside a VirtualService I'd like to see it in Jaeger, to also correlate it in Kiali with other metrics.

So, yeah, something where headers can be added to a span via configuration would be a great feature.

[morvencao]

Is there any possible we have this in istio 1.1.x?

[clyang82]

@morvencao Can this feature meet your requirements? https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/#Headers
Here is example vs configuration for productpage:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  creationTimestamp: 2019-04-24T12:56:37Z
  generation: 1
  name: bookinfo
  namespace: default
  resourceVersion: "8291608"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/bookinfo
  uid: 65639705-6690-11e9-9063-00163e019bbe
spec:
  gateways:
  - bookinfo-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
      headers:
        request:
          add:
            x-team: develop

from the envoy log, you can see 'x-team', 'develop' was added.

[2019-04-26 03:40:54.556][24][debug][router] [external/envoy/source/common/router/router.cc:381] [C2][S17715858181031491984] router decoding headers:
':authority', 'cyangadd1.fyre.ibm.com:31380'
':path', '/productpage'
':method', 'GET'
':scheme', 'http'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'
'x-forwarded-for', '10.1.131.128'
'x-forwarded-proto', 'http'
'x-request-id', '4d7f5c28-75b9-4c4d-971b-7bd02c1d0f1d'
'x-team', 'develop'
'content-length', '0'
'x-envoy-internal', 'true'
'x-b3-traceid', '850e57ea05403b2af0130b2de960ff30'
'x-b3-spanid', 'fd59bf17855398c3'
'x-b3-parentspanid', 'f0130b2de960ff30'
'x-b3-sampled', '0'

[morvencao]

@clyang82 Header manipulation rules help us to manipulate headers in http request/response to/from destination service.

We need something beyond that, more specifically, we need to customize jaeger tags from headers in http request/response to/from destination service.

We need to update the Istio api to implement this feature.

[morvencao]

There is already a proposal by @douglas-reid to add tracing control(including tagsFromHeaders) in Sidecar API, hopefully we will have this feature in next release.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[objectiser]

Work is currently underway in Envoy to enable custom tags to be created from a variety of sources, including from request headers (which was supported before) but now having the ability to use a different name for the tag. Once this lands, it can be used via the EnvoyFilter initially, and then eventually through an appropriate Istio API.

[douglas-reid]

Not stale.

[jacob-delgado]

Related to #22457

@morvencao for the time being you can use the envoy capability until we have a better way of supporting it in istio (see the link above).

[morvencao]

thanks @jacob-delgado

[jacob-delgado]

Verified in 1.6.0-beta.0 build