BlackHoleCluster telemetry for HTTP istio_requests_total is missing destination_service

[nrjpoddar]

Bug description

When Istio is configured with

outboundTrafficPolicy:
  mode: REGISTRY_ONLY

and an HTTP request is made to http://httpbin.org the reported metric istio_requests_total has the label destination_service set to unknown.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[x] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
The label destination_service should be set to BlackHoleCluster.

Steps to reproduce the bug

• Fetch Istio 1.3.0-rc0
• Change outboundTrafficPolicy in install/kubernetes/istio-demo.yaml from mode: ALLOW_ANY to mode: REGISTRY_ONLY
• Install the updated manifest: kubectl apply -f install/kubernetes/istio-demo.yaml
• Install sleep app: kubectl appy -f samples/sleep/sleep.yaml
• Curl from the sleep container:

    PODNAME=$(kubectl get pods -l app=sleep --no-headers | awk '{print $1}')
    kubectl exec -ti $PODNAME -c sleep -- /bin/sh
    curl -v http://httpbin.org/get // should return 502

• Port-forward to Prometheus in istio-system namespace:

    PODNAME=$(kubectl get pod -n istio-system -l app=prometheus --no-headers | awk '{print $1}')
    kubectl -n istio-system port-forward $PODNAME 9090:9090

• curl -sS http://localhost:9090/api/v1/query?query=istio_requests_total | jq . should give the following:

         {
        "metric": {
          "__name__": "istio_requests_total",
          "connection_security_policy": "unknown",
          "destination_app": "unknown",
          "destination_principal": "unknown",
          "destination_service": "unknown",
          "destination_service_name": "unknown",
          "destination_service_namespace": "unknown",
          "destination_version": "unknown",
          "destination_workload": "unknown",
          "destination_workload_namespace": "unknown",
          "instance": "100.96.2.183:42422",
          "job": "istio-mesh",
          "permissive_response_code": "none",
          "permissive_response_policyid": "none",
          "reporter": "source",
          "request_protocol": "http",
          "response_code": "502",
          "response_flags": "-",
          "source_app": "sleep",
          "source_principal": "unknown",
          "source_version": "unknown",
          "source_workload": "sleep",
          "source_workload_namespace": "default"
        },
        "value": [
          1567023275.748,
          "1"
        ]
      }

Additional Info
The istio-telemetry pod logs the following:

{
  "level": "info",
  "time": "2019-08-28T20:20:08.205416Z",
  "instance": "accesslog.instance.istio-system",
  "apiClaims": "",
  "apiKey": "",
  "clientTraceId": "",
  "connection_security_policy": "unknown",
  "destinationApp": "",
  "destinationIp": "0.0.0.0",
  "destinationName": "unknown",
  "destinationNamespace": "default",
  "destinationOwner": "unknown",
  "destinationPrincipal": "",
  "destinationServiceHost": "",
  "destinationWorkload": "unknown",
  "grpcMessage": "",
  "grpcStatus": "",
  "httpAuthority": "httpbin.org",
  "latency": "207.656µs",
  "method": "GET",
  "permissiveResponseCode": "none",
  "permissiveResponsePolicyID": "none",
  "protocol": "http",
  "receivedBytes": 139,
  "referer": "",
  "reporter": "source",
  "requestId": "b13941f0-80d6-4bd3-ae54-2b10c49dde6f",
  "requestSize": 0,
  "requestedServerName": "",
  "responseCode": 502,
  "responseFlags": "-",
  "responseSize": 0,
  "responseTimestamp": "2019-08-28T20:20:08.205558Z",
  "sentBytes": 84,
  "sourceApp": "sleep",
  "sourceIp": "100.96.3.248",
  "sourceName": "sleep-66c8d79ff5-ptqjv",
  "sourceNamespace": "default",
  "sourceOwner": "kubernetes://apis/apps/v1/namespaces/default/deployments/sleep",
  "sourcePrincipal": "",
  "sourceWorkload": "sleep",
  "url": "/get",
  "userAgent": "curl/7.64.0",
  "xForwardedFor": "0.0.0.0"
}

The sleep POD route config for port 80 has the following block_all route:

{
        "name": "block_all",
        "domains": [
          "*"
        ],
        "routes": [
          {
            "match": {
              "prefix": "/"
            },
            "directResponse": {
              "status": 502
            },
            "typedPerFilterConfig": {
              "mixer": {
                "@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
                "disableCheckCalls": true,
                "mixerAttributes": {
                  "attributes": {
                    "destination.service.host": {
                      "stringValue": "BlackHoleCluster"
                    },
                    "destination.service.name": {
                      "stringValue": "BlackHoleCluster"
                    }
                  }
                },
                "forwardAttributes": {
                  "attributes": {
                    "destination.service.host": {
                      "stringValue": "BlackHoleCluster"
                    },
                    "destination.service.name": {
                      "stringValue": "BlackHoleCluster"
                    }
                  }
                }
              }
            }
}

[nrjpoddar]

@mandarjog & @douglas-reid FYI

[nrjpoddar]

Might be related to #15255

[bianpengyuan]

I dug a bit and looks like routeEntry is empty, which causes report missing destination service name:
https://github.com/istio/proxy/blob/06647a2aa53879cd78414131f256647fbea5e29f/src/envoy/http/mixer/filter.cc#L76

instead, directResponseEntry is populated, which is actually expected:
https://github.com/envoyproxy/envoy/blob/b28edcae3da369ed9969796f89118c1f898a27d9/include/envoy/router/router.h#L802-L810

Not sure if there is a good way to get metadata from route even it is a direct response.