Document metrics monitoring tooling

[youngbupark]

In what area(s)?

/area observability

Related issue

dapr/dapr#969

Describe the feature

When we improve metrics (#969), it is good to provide monitoring images and deployment yaml with monitoring configuration, which allows users to enable monitoring system easily

• metrics : prometheus + operators + grafana

[youngbupark]

@jjcollinge Can you please take a look at this issue? The purpose of this issue is to set up metric backend(prometheus + prom-operators + grafana) in a easy way. For example, Knative serving example.

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.12.0/monitoring-metrics-prometheus.yaml

@jjcollinge @yaron2 @amanbha how do you think?

[youngbupark]

Working with @LaurentLesle

[LaurentLesle]

+1

[youngbupark]

I'm validating metrics from dapr runtime. I found two problems:

1. emitting metrics are stopped intermittently
2. http request latency is growing. (when I print out the latency inside python app, it is correct)

Once @LaurentLesle completes prometheus+grafana setup, we will compare the result between azmonitor and grafana.

Test environment:
0. Test app: helloworld k8s sample app

1. AKS 1.15
2. Monitoring backend: Azure Monitor
3. Config map for AKS omsagent

kind: ConfigMap
apiVersion: v1
data:
  schema-version:
    v1
  config-version:
    ver1
  log-data-collection-settings: |-
    [log_collection_settings]
       [log_collection_settings.stdout]
          enabled = true
          exclude_namespaces = ["kube-system"]
       [log_collection_settings.stderr]
          # Default value for enabled is true
          enabled = true
          exclude_namespaces = ["kube-system"]
       [log_collection_settings.env_var]
          enabled = true
       [log_collection_settings.enrich_container_logs]
          enabled = true
       [log_collection_settings.collect_all_kube_events]
          enabled = false
  prometheus-data-collection-settings: |-
    # Custom Prometheus metrics data collection settings
    [prometheus_data_collection_settings.cluster]
        interval = "1m"
        monitor_kubernetes_pods = true
    [prometheus_data_collection_settings.node]
        interval = "1m"
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system

cc/ @yaron2 @amanbha @jjcollinge

[jjcollinge]

Are we sure this isn’t the http latency sum metric?

[youngbupark]

Are we sure this isn’t the http latency sum metric?

Yes, this is not sum. I am using the below query now.

InsightsMetrics
| where Namespace == "prometheus" and Name == "http_handler_completed_latency_seconds"
| project TimeGenerated, Name, Val, tags=parse_json(Tags)
| extend app_id=tags['dapr.io/id'], pod_name=tags['pod_name'], le=tags['le'], http_method=tags['method'], http_host=tags['host'], http_path=tags['path'], http_status=tags['status'], grpc_method=tags['grpc_method'], grpc_service=tags['gprc_service'], grpc_type=tags['grpc_type']
| summarize avg(Val) by bin(TimeGenerated, 1m), api=strcat(tostring(app_id)," ", tostring(http_path))

[youngbupark]

@jjcollinge I exported query result. please look at this.

2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 100 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"100","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 1 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,791 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"1","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 0.03 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,791 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"0.03","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 0.03 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"0.03","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 0.3 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,791 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"0.3","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 30 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"30","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 10 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"10","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 0.01 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,788 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"0.01","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 300 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"300","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 3 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"3","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 0.1 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,791 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"0.1","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 0.01 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,657 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"0.01","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 10 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"10","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | pythonapp | pythonapp-56559dfbbb-98jg5 | 3 | POST | 127.0.0.1:3500 | /v1.0/invoke/nodeapp/method/neworder | 200 |   |   |   | http_handler_completed_latency_seconds | 67,791 | {"address":"10.244.2.34","app":"python","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"pythonapp","dapr.io/log-as-json":"true","handler":"","host":"127.0.0.1:3500","le":"3","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/invoke/nodeapp/method/neworder","pod-template-hash":"56559dfbbb","pod_name":"pythonapp-56559dfbbb-98jg5","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.2.34:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 30 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/subscriptions/38875a89-0178-4f27-a141-dc6fc01f183d/resourceGroups/dapr-test/providers/Microsoft.ContainerService/managedClusters/dapr-synthetic","container.azm.ms/clusterName":"dapr-synthetic","dapr.io/enabled":"true","dapr.io/id":"nodeapp","dapr.io/log-as-json":"true","dapr.io/port":"3000","handler":"","host":"127.0.0.1:3500","le":"30","method":"POST","name":"daprd","namespace":"default","path":"/v1.0/state/statestore","pod-template-hash":"58c4544586","pod_name":"nodeapp-58c4544586-wz69b","prometheus.io/path":"/","prometheus.io/port":"9090","prometheus.io/scrape":"true","scrapeUrl":"http://10.244.1.31:9090/","status":"200"} |  
  | 2/27/2020, 10:21:00.000 PM | nodeapp | nodeapp-58c4544586-wz69b | 0.3 | POST | 127.0.0.1:3500 | /v1.0/state/statestore | 200 |   |   |   | http_handler_completed_latency_seconds | 67,792 | {"address":"10.244.1.31","app":"node","container.azm.ms/clusterId":"/s