Turn off customMetrics, performanceCounters and metrics collected by default

[lidkowiak]

Is your feature request related to a problem? Please describe.
By default agent collects lots of metrics/data:

customMetric Loaded Class Count
customMetric % Of Max Heap Memory Used
customMetric Heap Memory Used (MB)
customMetric Suspected Deadlocked Threads
customMetric Current Thread Count
customMetric GC Total Time
customMetric GC Total Count
customMetric HeartbeatState
performanceCounter Available Bytes
performanceCounter IO Data Bytes/sec
performanceCounter % Processor Time
performanceCounter Private Bytes
metric Loaded Class Count
metric % Of Max Heap Memory Used
metric Heap Memory Used (MB)
metric Suspected Deadlocked Threads
metric Current Thread Count
metric GC Total Time
metric GC Total Count
metric HeartbeatState
metric Available Bytes
metric % Processor Time
metric Private Bytes
metric IO Data Bytes/sec

I prefer to use standard provided by Spring Boot/Micrometer. Is there a possibility to turn off collecting of the above telemetries?

Describe the solution you would like

By default no telemetries should be collected by default. Adding a config entry like collectStandardJVMMetric should cause collecting the data.

Describe alternatives you have considered

Additional context

[heyams]

{
  "instrumentation": {
        "jms": {
               "enabled": false
        }
}

have you tried this? this will disable all jms metrics, perf counters.

[heyams]

another option you can try is using metrics filtering. this allows you to filter out unwanted metrics.

[lidkowiak]

Well, I'm asking about JVM metrics plus I'm not using JMS atm so "jms enabled" will not solve it.

I've also found this

ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/TelemetryClientInitializer.java

Line 91 in 9d1ec8c

telemetryClient.addNonFilterableMetricNames(

It suggests that it's not possible to disable them as they are treated as non-filtered metrics/perf counters.

[heyams]

default metrics and perf counter collected by 3.x agent. you can use metrics filtering to filter both perf counter and custom metrics. i don't believe we use jvm to define any metrics we collect.

[heyams]

fyi:

some metrics specific to jvm are like the following:
none of these matched what you had above.

| jvm_memory_used |  
  | jvm_buffer_memory_used |  
  | jvm_threads_states |  
  | jvm_memory_committed |  
  | jvm_gc_pause |  
  | jvm_memory_max |  
  | jvm_buffer_total_capacity |  
  | jvm_threads_live |  
  | jvm_classes_unloaded |  
  | jvm_classes_loaded |  
  | jvm_threads_peak |  
  | jvm_threads_daemon |  
  | jvm_buffer_count |  
  | jvm_gc_memory_promoted |  
  | jvm_gc_memory_allocated |  
  | jvm_gc_max_data_size |  
  | jvm_gc_live_data_size

[heyams]

here is an example how to use metric filter:

{
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "a_test_metric",
            "another_test_metric"
          ]
        }
      }
    ]
  }
}

[lidkowiak]

I added following config:

"preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "Loaded Class Count",
            "% Of Max Heap Memory Used",
            "Heap Memory Used (MB)",
            "Suspected Deadlocked Threads",
            "Current Thread Count",
            "GC Total Time",
            "GC Total Count",
            "HeartbeatState",
            "\\Memory\\Available Bytes",
            "\\Process\\Private Bytes",
            "\\Process\\IO Data Bytes/sec",
            "\\Process\\% Processor Time",
            "\\Processor\\% Processor Time",
            "Available Bytes",
            "Private Bytes",
            "IO Data Bytes/sec",
            "% Processor Time"
          ]
        }
      }
    ]
}

and still I got in AI following custom metrics/perf counters

  | performanceCounter | Available Bytes | Memory |  
  | performanceCounter | Private Bytes | Process |  
  | performanceCounter | IO Data Bytes/sec | Process |  
  | performanceCounter | % Processor Time | Processor |  
  | performanceCounter | % Processor Time | Process |  
  | metric | Available Bytes |   |  
  | metric | Private Bytes |   |  
  | metric | IO Data Bytes/sec |   |  
  | metric | % Processor Time  |   |

I believe they match following code

ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/TelemetryClientInitializer.java

Line 91 in 9d1ec8c

telemetryClient.addNonFilterableMetricNames(

The fact that the agent collects some metrics/perf counters out-of-the box plus it is not documented at all is confusing. Additionally some telemetries can not be filtered.

I think metrics with prefix jvm_ comes from micrometer :) Magically when I put agent jar on the classpath they were collected

[heyams]

You can disable micrometer by doing the following:

{
  "instrumentation": {
    "micrometer": {
      "enabled": false
    }
  }
}

do you want to turn off metrics and perf counter completely?
If yes, you can use sampling:

{
  "sampling": {
    "percentage": 0
  }
}

this will turn off all telemetries though.

alternatively, you can use metric interval:

{
  "preview": {
    "metricIntervalSeconds": 31536000   // captured once a year
  }
}

this includes perf counters and metrics. but it doesn't impact custom metrics via 2.x SDK trackMetrics.

Hope these help.

[heyams]

https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#auto-collected-metrics
https://github.com/microsoft/ApplicationInsights-Java/wiki/Default-Metrics-Captured-by-Java-3.0-Agent

we do documented auto-collected metrics. hope this will clear your confusion.

[lidkowiak]

Thanks, what I want to achieve is to turn off collecting of metrics/perf counters that are enabled by default.
I partially achieved the goal with metric filtering but there are some leftovers.
Obviously I still want to collect other telemetries, especially these that come from micrometer.

I think the scenario I want to achieve is pretty standard. I don't understand why there is something enabled by default that can not be completely turn off. Better approach would be to explicitly enable collecting of default metrics with some config.

I haven't found a link (https://github.com/microsoft/ApplicationInsights-Java/wiki/Default-Metrics-Captured-by-Java-3.0-Agent ) on docs.microsoft.com so I thought it's not documented at all :)

[heyams]

Thanks, a good point. We will make that link available in the public azure docs. I've created a work item to track it #12103176.

I think the scenario I want to achieve is pretty standard. I don't understand why there is something enabled by default that can not be completely turn off. Better approach would be to explicitly enable collecting of default metrics with some config.

i think that's a valid ask. I will circle back if we will extend this feature to let customers have the ability to turn off default collected custom metrics, perf counters and metrics.

[heyams]

@lidkowiak confirmed. we will add that feature to turn off custom metrics, metrics and perf counters collected by default. work item

[trask]

hi @lidkowiak @emmansun can you confirm your reason for wanting to turn of the default metrics below? is it for cost, or for other reasons? thx!

ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/telemetry/MetricFilter.java

Lines 37 to 45 in 53ea6e7

	private static final Set<String> NON_FILTERABLE_METRIC_NAMES =
	new HashSet<>(
	asList(
	MetricNames.TOTAL_CPU_PERCENTAGE,
	MetricNames.PROCESS_CPU_PERCENTAGE,
	MetricNames.PROCESS_CPU_PERCENTAGE_NORMALIZED,
	MetricNames.PROCESS_MEMORY,
	MetricNames.TOTAL_MEMORY,
	MetricNames.PROCESS_IO));

[emmansun]

Hi @trask , in our use case, we use ApplicationInsights-Java for e2e trace, e.g. Request/Dependency/Trace log/..., for metric monitor, we have other solution. So we hope end user can have options to enable/disable those metrics collection, include default metrics. Yes, cost is also our concern. Thx!