Metrics disable not working

[mukundansundar]

In what area(s)?

/area runtime

/area operator
/area placement
/area docs
/area test-and-release

What version of Dapr?

0.11.0-rc.4

Expected Behavior

Metrics to be disabled if the configuration is added to disable it.

Actual Behavior

Metrics still visible.

Steps to Reproduce the Problem

Add configuration in either self hosted or kubernetes mode.

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: nodeconfig
  namespace: default
spec:
  tracing:
    samplingRate: "1"
  metric:
    enabled: false

Run any of the quick starts with this configuration enabled dapr.io/config annotation for kubernetes or --config flag for self hosted.

For self hosted --- see logs for metrics server port and visit that port.

For kubernetes --- port-forward 9090 and visit localhost:9090.

RELEASE NOTE: FIX disabling metrics configuration.

[shalabhms]

@mukundansundar , it is to be noted that this metric disable option flag is only for dapr runtime metrics. This flag will not affect dapr system metrics. So if the metrics are still emitted for runtime then this need to be fixed otherwise not.

[beiwei30]

@mukundansundar, I verified v0.11.0-62-g8e718001 with the standalone mode. I confirmed that the behavior of metrics does honor the global configuration.

You may need to watch the logger output to check if

enabled metrics http middleware

appears or not.

[halspang]

There may also be some clarification required as to what qualifies as a runtime/system metric. Running the hello-world quickstart with metrics disabled, I can confirm that I don't see enabled metrics http middleware, but I do see these metrics come through:

# HELP dapr_runtime_actor_status_report_total The number of the successful status reports to placement service.
# TYPE dapr_runtime_actor_status_report_total counter
 dapr_runtime_actor_status_report_total{actor_type="",app_id="nodeapp",operation="recv"} 3
# HELP dapr_runtime_actor_table_operation_recv_total The number of the received actor placement table operations.
# TYPE dapr_runtime_actor_table_operation_recv_total counter
 dapr_runtime_actor_table_operation_recv_total{actor_type="",app_id="nodeapp",operation="lock"} 1
 dapr_runtime_actor_table_operation_recv_total{actor_type="",app_id="nodeapp",operation="unlock"} 1
 dapr_runtime_actor_table_operation_recv_total{actor_type="",app_id="nodeapp",operation="update"} 1
# HELP dapr_runtime_component_init_total The number of initialized components.
# TYPE dapr_runtime_component_init_total counter
 dapr_runtime_component_init_total{app_id="nodeapp",component="exporters.zipkin"} 1
 dapr_runtime_component_init_total{app_id="nodeapp",component="pubsub.redis"} 1
 dapr_runtime_component_init_total{app_id="nodeapp",component="state.redis"} 1
# HELP dapr_runtime_component_loaded The number of successfully loaded components.
# TYPE dapr_runtime_component_loaded counter
 dapr_runtime_component_loaded{app_id="nodeapp"} 3

In addition to what appear to be more system level metrics:

# HELP dapr_http_client_completed_count Count of completed requests
# TYPE dapr_http_client_completed_count counter
 dapr_http_client_completed_count{app_id="nodeapp",method="GET",path="dapr/config",status="404"} 1
 dapr_http_client_completed_count{app_id="nodeapp",method="GET",path="dapr/subscribe",status="404"} 1
---
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
 go_memstats_alloc_bytes 6.54784e+06

It's unclear to me whether the _runtime_ metrics are system or runtime given that they appear to be around init. Should they also be excluded when we disable metrics? What about the base level http metrics?

[mukundansundar]

@shalabhms From the previous comment it is seen that runtime metrics is not disabled as expected.
Is that a correct conclusion?

[youngbupark]

@mukundansundar, I verified v0.11.0-62-g8e718001 with the standalone mode. I confirmed that the behavior of metrics does honor the global configuration.

You may need to watch the logger output to check if

enabled metrics http middleware

appears or not.

The current implementation is not adding middleware for http and grpc. I found that Dapr still emits appchannel metrics.

I made the fix to disable entire metrics by delaying metrics intiailization. in #2295