Collect metrics from JMX

[felixbarny]

Introduce a configuration option which enables the collection of arbitrary JMX metrics. It should be possible to get a collection of metrics and to convert properties to metric tags.

For example java.lang:type=GarbageCollector,name=*

A consideration is whether the collection of these metrics should be done in a separate metrics reporter thread vs the apm-reporter thread so that the gathering of metrics does not block sending other events.

[beckjim]

👍

Great. This would make the installation of Jolokia obsolete ...
No more Jolokia requests which "polute" access logs, ...

[danielfariati]

This would be awesome.
We are using Logstash + JMX Input Plugin to send our Kafka Producer / Consumer JMX metrics to Elasticsearch.
With that feature, we would be able to use APM to do it and simplify our architecture.

[felixbarny]

Thanks for the feedback. We're planning to tackle this soon.

[felixbarny]

I have a first sort-of working draft.

One remaining question is how users can actually configure this. Unfortunately, the configuration mechanism doesn't work well when lists of objects should be configured. The underlying primitive are key/value String-String pairs so that it can also be configured via environment variable, system properties and property files, for example. Admittedly, for this use-case, yaml would be a prime candidate though. Nevertheless, I still think the String-String key/value pairs are the right primitive for the overall agent configuration use case.

One syntax I have come up with is this:

object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionCount] metric_name[collection_count], 
object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionTime]

Another option would be just to use JSON:

{"object_name": "java.lang:type=GarbageCollector,name=*", "attribute": "CollectionCount", "metric_name": "collection_count"}, 
{"object_name": "java.lang:type=GarbageCollector,name=*", "attribute": "CollectionTime"}

This will generate these metric sets (this is the intake v2 format which is not the same as the documents stored in ES):

{"metricset":{"timestamp":1566284936313000,"tags":{"name":"G1 Old Generation",  "type":"GarbageCollector"},"samples":{"jvm.jmx.collection_count":{"value":0.0},"jvm.jmx.CollectionTime":{"value":0.0}}}}
{"metricset":{"timestamp":1566284936313000,"tags":{"name":"G1 Young Generation","type":"GarbageCollector"},"samples":{"jvm.jmx.collection_count":{"value":2.0},"jvm.jmx.CollectionTime":{"value":11.0}}}}

That looks pretty similar to the GC metrics the Java agent reports by default:

{"metricset":{"timestamp":1566284729581000,"tags":{"name":"G1 Old Generation"},  "samples":{"jvm.gc.time": {"value":0.0},  "jvm.gc.count":{"value":0.0}}}}
{"metricset":{"timestamp":1566284729581000,"tags":{"name":"G1 Young Generation"},"samples":{"jvm.gc.time": {"value":10.0}, "jvm.gc.count":{"value":2.0}}}}

Note that the agent will create tags/labels for each key property such as the JMX type and name. Again, this is similar to the behavior of what the agent already does for the GC metrics, for example. The disadvantage is that it means more metricsets will be created. The advantage is that you don't have to know the name (in this example the memory pool name) in advance. Instead, you can just do a terms aggregation on labels.name and query for the jvm.jmx.collection_count metric, for example. This yields one graph per memory pool, no matter how many there are and what their exact names are.
This is quite different from how the metricbeat Jolokia plugin would handle this case. It would add the full metric name to the same metric set. See also the More advanced configurations example in https://www.elastic.co/blog/monitoring-java-applications-with-metricbeat-and-jolokia.

It's also possible to capture composite JMX data structures. In this example, the HeapMemoryUsage attribute is a composite, consisting of the init, max, used and committed values.

object_name[java.lang:type=Memory] attribute[HeapMemoryUsage] metric_name[heap]

This is the resulting metric set:

{
    "metricset": {
        "timestamp":1566285899886000,
        "tags":    {"type":"Memory"},
        "samples": {
            "jvm.jmx.heap.init":      {"value":268435456.0},
            "jvm.jmx.heap.max":       {"value":4294967296.0},
            "jvm.jmx.heap.used":      {"value":22404496.0},
            "jvm.jmx.heap.committed": {"value":268435456.0}
        }
    }
}

Currently, only Numbers can be captured.

/cc @ruflin @jsoriano who are the maintainers of the Jolokia Metricbeat module (I think)

[eyalkoren]

Looks good!
JSON is much better formatted of course, but it doesn't fit the other configs we have, so I prefer comma-separated strings within the elasticapm.config options.
It doesn't mean we cannot introduce yaml configuration, but maybe as a separate file (with a predefined name). Meaning- support both String-String through "regular" config and yaml as file, and make a superset of them. This way, users can configure complicated (and sharable for community) files, or use the normal config if they need something very small/basic. If you make the String-String config dynamic, you can also provide a way to test stuff without restarts and get it for free through the stagemonitor mechanism.
However, no need to cover all use cases in first go- better release a MVP and make enhancements based on feedback.

[felixbarny]

JSON is much better formatted of course, but it doesn't fit the other configs we have, so I prefer comma-separated strings within the elasticapm.config options.

Not sure I understand correctly. We could use the JSON as the value in the elasticapm.properties file like this:

capture_jmx_metrics=[ \
  {"object_name": "java.lang:type=GarbageCollector,name=*", "attribute": "CollectionCount", "metric_name": "collection_count"}, \
  {"object_name": "java.lang:type=GarbageCollector,name=*", "attribute": "CollectionTime"} \
]
application_packages=com.foo

[eyalkoren]

Sorry for being unclear- I would prefer not using JSON as the string value of config options in elasticapm.properties, but simpler formatted string, like the one you proposed that fits only and exactly what you need.

However, this is just a matter of preference, I wouldn't object to using JSONs.

[ruflin]

@felixbarny I haven't touched jmx in quite some time but one thing I'm curious about is if it would make sense to align the Metricbeat jmx format and this one here? Could you share an example on what the "end format" is that is stored in Elasticsearch in contrast to what we get with Metricbeat? BTW I think @jsoriano is the expert on this one.

[jsoriano]

The main difference I see with metricbeat regarding configuration (apart of options naming) is that in the Jolokia module, for each object name (mbean), there can be multiple attribute mappings. So the JSON example here would be something like:

  {"object_name": "java.lang:type=GarbageCollector,name=*", "attributes": [
    {"attribute": "CollectionCount", "metric_name": "collection_count"},
    {"attribute": "CollectionTime"}
  ]}

And not sure how this could be done with the other format.

In metricbeat yaml-based config this mapping would be defined with something like this:

- mbean: 'java.lang:type=GarbageCollector,name=*'
  attributes:
    - attr: CollectionCount
      field: collection_count
    - attr: CollectionTime
      field: CollectionTime

I'm curious about is if it would make sense to align the Metricbeat jmx format and this one here?

Metrics collected by metricbeat are stored in events under jolokia.<namespace>.<configured field>, what is not ideal because we cannot provide a mapping.
We are considering changing it to follow what we are doing in other "generic" modules: storing all metrics under the same object. For example for prometheus collector all metrics are stored under prometheus.metrics.*. For JMX metrics we could consider using something as jmx.metrics.*. Or jmx.<type>.metrics.* if we want to support non-numeric attributes.

I guess that to align the format with APM, the metrics should be stored under metricset.samples.jvm.jmx..., what can be too different to how Metricbeat metrics are stored. In metricbeat metricset is used only for metadata, and there is no concept of samples.

[felixbarny]

Could you share an example on what the "end format" is that is stored in Elasticsearch in contrast to what we get with Metricbeat?

See the examples in http://apm-agent-java_801.docs-preview.app.elstc.co/guide/en/apm/agent/java/master/config-jmx.html

there can be multiple attribute mappings.

Yes, that's some kind of limitation of the syntax. We could extend it to something like

object_name[java.lang:type=GarbageCollector,name=*] attributes[CollectionCount, CollectionTime] metric_name[collection_count]

or even

object_name[java.lang:type=GarbageCollector,name=*] attributes[CollectionCount[collection_count], CollectionTime]

But it comes with its own challenges. In the end it's the same as having multiple declarations with the same object name, just a bit more concise. If we would get rid of the name to customize the metric name and just use the attribute name, I guess this syntax would be best:

object_name[java.lang:type=GarbageCollector,name=*] attributes[CollectionCount, CollectionTime]

[felixbarny]

another variant:

object_name[java.lang:type=GarbageCollector,name=*] attributes[CollectionCount as collection_count, CollectionTime]

[felixbarny]

To continue the brainstorming we could start out with this syntax:

object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionCount]

To specify additional optional properties, like the metric_name, we can leverage the syntax used by javax.management.ObjectName. We can then even rely on that class for parsing the String.

object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionCount:metric_name=collection_count]

That would also allow adding additional properties later on, for example metric_type

object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionCount:metric_name=collection_count,metric_type=delta]

ObjectName parsing test

ObjectName objectName = new ObjectName("CollectionCount:metric_name=collection_count,metric_type=delta");
System.out.println(objectName);
System.out.println(objectName.getDomain());
System.out.println(objectName.getKeyPropertyList());

Output

CollectionCount:metric_name=collection_count,metric_type=delta
CollectionCount
{metric_name=collection_count, metric_type=delta}

---

To configure multiple attributes, just repeat the attribute part:

object_name[java.lang:type=GarbageCollector,name=*] attribute[CollectionCount] attribute[CollectionTime]

[ruflin]

Talking about the event structure: It's basically jolokia.* with metricbeat vs. jvm.jmx.* for the Java agent. As we might change jolokia in the future, we could align on jvm.jmx.*? @felixbarny How would a memory example look in your case? https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-jolokia-jmx.html#_fields_57

[felixbarny]

A heap example is also available in the docs: http://apm-agent-java_801.docs-preview.app.elstc.co/guide/en/apm/agent/java/master/config-jmx.html

EDIT:
(just noticed the example is wrong, there is no value object, will remove

EDIT 2:
fixed

[jsoriano]

Talking about the event structure: It's basically jolokia.* with metricbeat vs. jvm.jmx.* for the Java agent. As we might change jolokia in the future, we could align on jvm.jmx.*?

Sounds good to me, I thought this was the event sent by the Java agent:

{
    "metricset": {
        "timestamp":1566285899886000,
        "tags":    {"type":"Memory"},
        "samples": {
            "jvm.jmx.heap.init":      {"value":268435456.0},
            "jvm.jmx.heap.max":       {"value":4294967296.0},
            "jvm.jmx.heap.used":      {"value":22404496.0},
            "jvm.jmx.heap.committed": {"value":268435456.0}
        }
    }
}

[felixbarny]

That’s correct but the apm server will transform that before sending to es

[ruflin]

It seems the two formats are not too far apart from each other. We should probably also make this part of the ECS metrics effort to unify on the naming for the most common metrics?

[mdeshmu]

Using APM Java agent can we collect thread list at host level and some basic info (cpu time/user time) for each thread like java melody provides?