Can I exclude default metrics in java.lang.*?

[takus]

Hi, I have a question.

I defined /etc/dd-agent/conf.d/jmx.yaml like the following. exclude is set for ignoring bean java.lang.*.

---
instances:
- name: jdbc_master
  host: 127.0.0.1
  port: 8086
  tags:
    pool: master
  conf:
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*master.*
    exclude:
      bean_regex:
        - java.lang.*
        - .*PoolConfig.*
- name: jdbc_slave
  host: 127.0.0.1
  port: 8086
  tags:
    pool: slave
  conf:
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*slave.*
    exclude:
      bean_regex:
        - java.lang.*
        - .*PoolConfig.*

# Nothing to configure here
init_config:

But the config seems to be ignored.

service datadog-agent jmx list_collected_attributes

#####################################
Instance: 127.0.0.1:8086
#####################################

       Matching: 1/350. Bean name: java.lang:type=Threading - Attribute name: ThreadCount  - Attribute type: int
       Matching: 2/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 3/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 7/350. Bean name: java.lang:type=Memory - Attribute name: HeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 11/350. Bean name: java.lang:type=Memory - Attribute name: NonHeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 12/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 13/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 14/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: TotalConnections  - Attribute type: int
       Matching: 15/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: IdleConnections  - Attribute type: int
       Matching: 16/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: ActiveConnections  - Attribute type: int
       Matching: 17/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: ThreadsAwaitingConnection  - Attribute type: int

#####################################
Instance: 127.0.0.1:8086
#####################################

       Matching: 1/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: TotalConnections  - Attribute type: int
       Matching: 2/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: IdleConnections  - Attribute type: int
       Matching: 3/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: ActiveConnections  - Attribute type: int
       Matching: 4/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: ThreadsAwaitingConnection  - Attribute type: int
       Matching: 5/350. Bean name: java.lang:type=Threading - Attribute name: ThreadCount  - Attribute type: int
       Matching: 6/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 7/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 11/350. Bean name: java.lang:type=Memory - Attribute name: HeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 15/350. Bean name: java.lang:type=Memory - Attribute name: NonHeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 16/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 17/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionTime  - Attribute type: long

Any way to ignore java.lang.* metrics?

[olivielpeau]

Hi @takus, thanks for opening this issue!

These metrics on the java.lang domain are basic JVM metrics that JMXFetch collects by default, regardless of what you define in your own jmx.yaml file. There is currently no option in JMXFetch to disable the collection of these default metrics.

Could you give us a sense of why you want to avoid collecting these metrics? Any details on your use case could help here.

Thanks again!

[takus]

Hi @olivielpeau, thanks for the reply.

I'd like to add different tags into different metrics. Ideal configuration is like this:

---
instances:
- name: webapp
  host: 127.0.0.1
  port: 8086
  conf:
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*master.*
    exclude:
      bean_regex:
        - .*PoolConfig.*
    tags:
        pool: master
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*slave.*
    exclude:
      bean_regex:
        - .*PoolConfig.*
    tags:
        pool: slave

In this case, both conf record different JDBC pool's metrics with the same name. Thus, I'd like to identify each metrics with tags.

• jmx.com.zaxxer.hikari.total_connections (tag = pool:master)
• jmx.com.zaxxer.hikari.total_connections (tag = pool:slave)

But, it seems that tag should be set for each instance, not each conf. For adding different tags into different metrics, I defined multiple instances of the same JVM.

Do you have any ideas to do that?

[olivielpeau]

@takus There is currently no way to add tags on each element of the conf section. Currently custom tags can be set:

• on a per-instance basis as you noticed
• on a per-attribute basis (see Allow yaml to include a list of tags that will be applied to a specific include. #35 for an example)

That said, looking at the beans you're collecting, the metrics would be tagged automatically with type: tags, i.e type:pool_slave and type:pool_master. I do realize it may not be as useful as actual pool: tags though.

Hope this helps! Adding tags on each element of the conf section could be useful, but it would take some time to implement it. Feel free to open a PR adding this if you can :)

Thanks!

[takus]

@olivielpeau

on a per-attribute basis (see #35 for an example)

Oh, I didn't know about this setting. I'm trying to define all attributes I need with tags for now. Thanks to helpful information!!