Have some example rules of generic JVM metrics? #111

EdSchouten · 2017-02-14T10:31:18Z

Hi there,

We're making use of the JMX exporter in combination with Tomcat and Cassandra. While using it, we've noticed that none of the examples contain any generic rules for extracting properties out of the JVM, like memory usage. Right now we're planning on adding something like this to our configs:

- pattern: 'java.lang<type=Memory><(\w+)MemoryUsage>(\w+):'
  name: java_memory_usage_$2_bytes
  labels:
    area: "$1"
  help: Java $2 $1 memory usage
  type: GAUGE

Resulting in metrics like these:

java_memory_usage_committed_bytes{area="Heap",} 1.489731584E9
java_memory_usage_committed_bytes{area="NonHeap",} 3.9223296E8
java_memory_usage_max_bytes{area="Heap",} 4.294770688E9
java_memory_usage_max_bytes{area="NonHeap",} 4.17333248E8
java_memory_usage_init_bytes{area="Heap",} 2.73888048E8
java_memory_usage_init_bytes{area="NonHeap",} 3.69557504E8
java_memory_usage_used_bytes{area="Heap",} 9.61119192E8
java_memory_usage_used_bytes{area="NonHeap",} 2.09771528E8

Question: would it make sense to have some kind of examples file containing rules like these?

The text was updated successfully, but these errors were encountered:

brian-brazil · 2017-02-14T10:59:19Z

These shouldn't be in the examples, as the standard exports of the agent already provide these metrics in a better format.

EdSchouten · 2017-02-14T11:05:56Z

Oh, wow. Running it separately or as an agent determines which metrics are exported by default? Interesting. Is this documented somewhere? What kind of metrics are exported by default when running as an agent?

brian-brazil · 2017-02-14T11:56:37Z

Those metrics should always be exported, but they're not enabled for the http server as they'd be confusing.

EdSchouten · 2017-02-14T17:04:09Z

Should we document this somewhere?

wfindley · 2017-05-18T01:13:58Z

Oh. I just ran into this. How can I induce the http server to do things like jvm_memory_bytes_used?

brian-brazil · 2017-05-18T07:33:59Z

Run it as a java agent.

hejix · 2017-05-24T18:48:24Z

Hi Brian, There are many good reasons to run the jmx_exporter separately as a server- the main one being able to change the metrics captured without having to restart the underlying process. Would appreciate either being able to turn on the standard jvm metrics or knowing the rules to enable them as mentioned by earlier posters.

brian-brazil · 2017-05-24T18:58:49Z

the main one being able to change the metrics captured without having to restart the underlying process.

That doesn't require a restart.

michaelbv · 2017-05-25T14:51:11Z

How do I drop values scraped by default i.e. not controlled by the exporter config?
Stuff like:
jmx_config_reload_success_total 0.0
jvm_memory_pool_bytes_committed{pool="Code Cache",}
Do I need to drop those in the relabeling phase?
I am running it as an agent.

brian-brazil · 2017-05-25T14:57:48Z

Why do you want to drop them? They're all fairly important.

michaelbv · 2017-05-25T15:26:20Z

I want to spend my bytes on other things.

A couple of items that would dramatically reduce the engineering cost of prometheus for us

can you elaborate on

That doesn't require a restart.

How do I get it to reload the conf without restarting the process?

I can't get the logging config to work so I get zero log output and haven't any idea why it fails. In most cases the apps I'm scraping are using several of log4j and logback and java.util.logging.

brian-brazil · 2017-05-25T15:29:40Z

I want to spend my bytes on other things.

If you're so short on resource that this matters (which seems unlikely), you can drop them on the Prometheus end.

How do I get it to reload the conf without restarting the process?

Change the config file on disk.

In most cases the apps I'm scraping are using several of log4j and logback and java.util.logging.

If it's stats on those you want look at the simpleclient modules for those. Neither expose jmx metrics as far as I'm aware.

This has gotten way off-topic, so closing. Please ask usage questions on https://groups.google.com/forum/#!forum/prometheus-users

agolomoodysaada · 2017-07-17T18:54:53Z

I would like to re-open this issue @brian-brazil. All the process metrics for Kafka are available through JMX. Since we're running the httpserver against the kafka JMX metrics, it makes sense to have an example for the JVM metrics since they're already available.

Also, I think your logic here is extremely confusing

they're not enabled for the http server as they'd be confusing.

Having inconsistent and undocumented defaults is even more confusing.

brian-brazil · 2017-07-17T20:02:46Z

All the process metrics for Kafka are available through JMX.

Not all of them.

Use of the agent is recommended over the httpserver, and I do not wish to have to maintain two versions of the same bit of code/configuration.

agolomoodysaada · 2017-07-17T20:24:45Z

Thank you for your response.
Is it possible to run the agent with Kafka? Do I have to restart kafka to attach the agent?

brian-brazil · 2017-07-17T20:31:19Z

Yes, you need to restart kafka to attach it. https://www.robustperception.io/monitoring-kafka-with-prometheus/ has instructions.

agolomoodysaada · 2017-07-17T20:34:27Z

Thank you, last question: are there plans to deprecate the httpserver?

brian-brazil · 2017-07-17T20:39:57Z

No, there's no plans to do so.

alanorth · 2017-11-05T15:52:35Z

Those metrics should always be exported, but they're not enabled for the http server as they'd be confusing.

@brian-brazil can you add some sort of tip to the readme that jvm_* metrics are only exposed when using the Java agent? It took me an hour or two of troubleshooting and searching old issues to figure this out, after playing only with the HTTP server version. Thanks!

brian-brazil · 2017-11-05T21:11:08Z

We recommend always using the java agent, if you go against that and use the http server this is one of the issues you'll run into.

alanorth · 2017-11-06T08:42:05Z

We recommend always using the java agent, if you go against that and use the http server this is one of the issues you'll run into.

@brian-brazil yes I understand that now, but only after an hour or two of troubleshooting and reading the GitHub issues. My comment above was a suggestion that you should make it more obvious in the readme.

The readme isn't very helpful for new users, in my opinion. For example: I'm a sysadmin, not a Java developer. I don't know what JMX or a Java agent is, I'm just trying to monitor some application server! It took me a few hours to figure out how to get this thing running first of all, and then it's not obvious that the JVM metrics aren't exposed via the http server.

brian-brazil · 2017-11-06T14:46:55Z

The first section is https://github.com/prometheus/jmx_exporter#running which gives you a full example command line, I'm not sure it's possible to make it any simpler.

EdSchouten · 2017-11-06T14:59:16Z

Brian,

I don't think the issue necessarily is that people don't know how to run this as an agent. It's more that the README doesn't explain the difference between running it as an agent or as a separate process. This is why this issue still gets some traffic once every couple of weeks/months of people who run into the same problems as I did.

The very top of the README currently has these two sentences right after each other:

It meant to be run as a Java Agent, exposing an HTTP server and scraping the local JVM.
This can be also run as an independent HTTP server and scrape remote JMX targets.

Would it make sense to extend this?

brian-brazil · 2017-11-06T15:28:11Z

I've tried to keep the readme light on the http server, so that users would follow the easy path and use the agent as that's what's most clearly documented and works best. At the end of the day users often over-complicate things for themselves, and I can't stop all of them.

Do you have wording you'd suggest?

EdSchouten · 2017-11-06T19:11:50Z

@brian-brazil Any thoughts on #204?

As discussed in issue prometheus#111, it is not intended that the JMX exporter extracts generic JVM metrics when running as a separate process. These values are only exposed when running the exporter as an agent as part of the process to be monitored.

#204) As discussed in issue #111, it is not intended that the JMX exporter extracts generic JVM metrics when running as a separate process. These values are only exposed when running the exporter as an agent as part of the process to be monitored.

ppatierno · 2017-11-20T16:22:07Z

@brian-brazil Hi ... reading this issue I don't see any real reason why the HTTP server mode doesn't support JVM metrics and what do you mean saying "they'd be confusing". I'm thinking about using it within a "sidecar" approach on Kubernetes/OpenShift but it seems that you really discourage the HTTP server mode. Can you explain better why ? It's something related to maintaining the source code because it will be deprecated in a near future or there are some technical reasons ?

ppatierno · 2017-11-27T08:30:17Z

@brian-brazil any updated on the above question ?

DivsDibs · 2018-07-05T19:34:29Z

@brian-brazil Hi Brian, might be too naive to ask, but I'm using the JMX Exporter as an agent as described in the robutsperception link, and I want to customize the name of the metric generated for jvm.* or process.* related metrics.. Reason being, the same jvm/process related metrics are generated by multiple targets and is fed into prometheus, which doesn't allow me to choose the metric from kafka node specifically. Couldn't figure out a way. Would be really helpful, if you please guide me in the right direction? Thanks

akamensky · 2019-12-06T07:16:55Z

Oh wow, doing design choice because believing that other developers and users don't understand things about JVM?

There is no performance nor implementation limitations of why JVM metrics cannot be available over HTTP server, while there are number of reasons to not run this as a java agent. It is just developers of this tool own decision to not do it because they want so. Amazing.

mdione-cloudian · 2020-06-22T13:52:59Z

@EdSchouten I fixed one of your patterns:

- pattern: 'java.lang<type=Memory><(\w+)MemoryUsage>(\w+): (\d+)'
  name: java_memory_usage_$2_bytes
  labels:
    area: "$1"
  value: $3
  # help: Java $2 $1 memory usage
  type: GAUGE

This returns:

# HELP java_memory_usage_used_bytes java.lang.management.MemoryUsage (java.lang<type=Memory><HeapMemoryUsage>used)
# TYPE java_memory_usage_used_bytes gauge
java_memory_usage_used_bytes{area="Heap",} 5.47159024E8
java_memory_usage_used_bytes{area="NonHeap",} 7.235664E7

You can change the help line as you want.

mdione-cloudian · 2020-07-08T14:20:04Z

This is the final set of patterns I'm using:

- pattern: 'java.lang<type=Memory><(\w+)MemoryUsage>(\w+): (\d+)'
  name: jvm_memory_usage_$2_bytes
  labels:
    area: "$1"  # Heap/NonHeap
  value: $3
  type: GAUGE

# name is always the same, the name of the GC
- pattern: 'java.lang<type=GarbageCollector, name=[^,]+, key=([^>]+)><LastGcInfo, memoryUsageAfterGc>(used|commited): (\d+)'
  name: jvm_memory_after_gc_$2_bytes
  value: $3
  labels:
    space: $1
  type: GAUGE

- pattern: 'java.lang<type=GarbageCollector, name=[^>]+><LastGcInfo>duration: (\d+)'
  name: jvm_gc_duration_seconds
  value: $1
  type: GAUGE
  # Convert microseconds to seconds
  valueFactor: 0.000001

# java.lang<type=GarbageCollector, name=G1 Young Generation><>CollectionCount
- pattern: 'java.lang<type=GarbageCollector, name=([^>]+)><>CollectionCount: (\d+)'
  name: jvm_gc_collection_count
  value: $2
  labels:
    name: $1
  type: GAUGE

prometheus#204) As discussed in issue prometheus#111, it is not intended that the JMX exporter extracts generic JVM metrics when running as a separate process. These values are only exposed when running the exporter as an agent as part of the process to be monitored.

mcastelino · 2020-12-21T00:53:05Z

@mdione-cloudian we had to use a slightly modified version of your patterns to get java metrics out of kafka. /cc @sraghav1

    - pattern: 'java.lang<type=Memory><(\w+)MemoryUsage>(\w+): (\d+)'
      name: jvm_memory_usage_$2_bytes
      labels:
        area: "$1"  # Heap/NonHeap
      value: $3
      type: GAUGE
    - pattern: 'java.lang<name=([\s\w]+), type=GarbageCollector, key=(\w+)>(.*): (\d+)'
      name: jvm_gc_$3
      labels:
        name: $1
        key: $2
      value: $4
      type: GAUGE
    - pattern: 'java.lang<name=([\s\w]+), type=MemoryPool, key=(\w+)>(.*): (\d+)'
      name: jvm_mempool_$3
      labels:
        name: $1
        key: $2
      value: $4
      type: GAUGE
    - pattern: 'java.lang<name=([\s\w]+), type=GarbageCollector>(.*): (\d+)'
      name: jvm_gc_$2
      labels:
        name: $1
      value: $3
      type: GAUGE
    - pattern: 'java.lang<name=([\s\w]+), type=MemoryPool>(.*): (\d+)'
      name: jvm_mempool_$2
      labels:
        name: $1
      value: $3
      type: GAUGE

matiasba · 2021-02-25T17:55:20Z

Does anyone have patters to collect thread and cpu metrics?

fstab · 2021-02-25T20:40:37Z

Maybe you don't need the jmx_exporter for that. You could just use the simpleclient_hotspot and call

new StandardExports().register();
new ThreadExports().register();

https://mvnrepository.com/artifact/io.prometheus/simpleclient_hotspot
https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot

matiasba · 2021-03-01T13:05:51Z

Sadly I'm trying to use this with a very legacy app that runs only on java 1.5. I need to use the jmx http agent as is. Does anyone knows how to get cpu a thread metrics using the http agent?

nareshb123 · 2021-03-17T04:48:11Z

jmx_exporter agent is not exporting CPU metrics, can anyone suggest what need to be done.

mdione-cloudian · 2021-03-17T08:29:43Z

@nareshb123 @matiasba asked for the same thing. Maybe he figured it out?

matiasba · 2021-03-17T12:21:46Z

@nareshb123 @matiasba asked for the same thing. Maybe he figured it out?

Sadly I just gave up on this, there is very little documentation. I just told the dev team that they need to upgrade their java version of they want to use Prometheus.

ndimitry · 2021-04-28T12:44:03Z

@brian-brazil Just try to start java agent with JBoss/WildFly on Java11 and you will have much fun (spoiler: it just won't run, JBoss/WildFly won't start; and even running on Java8 is a quite a pain), so the http server and examples are useful in some cases.
You may state that there's a bug in JBoss/WildFly (and that might be a fairly right statement), but nevertheless we need monitoring data before bugs are fixed...

brian-brazil closed this as completed May 25, 2017

EdSchouten mentioned this issue Nov 6, 2017

Document that running as an agent is required for generic JVM metrics. #204

Merged

ppatierno mentioned this issue Nov 20, 2017

Explore options for metrics & monitoring strimzi/strimzi-kafka-operator#42

Closed

sidps mentioned this issue Jun 23, 2020

Run JMX exporter as a Java Agent (how to?) Yolean/kubernetes-kafka#322

Open

Have some example rules of generic JVM metrics? #111

Have some example rules of generic JVM metrics? #111

Comments

EdSchouten commented Feb 14, 2017 • edited Loading

brian-brazil commented Feb 14, 2017

EdSchouten commented Feb 14, 2017

brian-brazil commented Feb 14, 2017

EdSchouten commented Feb 14, 2017

wfindley commented May 18, 2017 • edited Loading

brian-brazil commented May 18, 2017

hejix commented May 24, 2017

brian-brazil commented May 24, 2017

michaelbv commented May 25, 2017

brian-brazil commented May 25, 2017

michaelbv commented May 25, 2017 • edited Loading

brian-brazil commented May 25, 2017

agolomoodysaada commented Jul 17, 2017 • edited Loading

brian-brazil commented Jul 17, 2017

agolomoodysaada commented Jul 17, 2017 • edited Loading

brian-brazil commented Jul 17, 2017

agolomoodysaada commented Jul 17, 2017

brian-brazil commented Jul 17, 2017

alanorth commented Nov 5, 2017

brian-brazil commented Nov 5, 2017 • edited Loading

alanorth commented Nov 6, 2017 • edited Loading

brian-brazil commented Nov 6, 2017

EdSchouten commented Nov 6, 2017

brian-brazil commented Nov 6, 2017

EdSchouten commented Nov 6, 2017

ppatierno commented Nov 20, 2017

ppatierno commented Nov 27, 2017

DivsDibs commented Jul 5, 2018

akamensky commented Dec 6, 2019

mdione-cloudian commented Jun 22, 2020

mdione-cloudian commented Jul 8, 2020

mcastelino commented Dec 21, 2020 • edited Loading

matiasba commented Feb 25, 2021

fstab commented Feb 25, 2021 • edited Loading

matiasba commented Mar 1, 2021

nareshb123 commented Mar 17, 2021

mdione-cloudian commented Mar 17, 2021

matiasba commented Mar 17, 2021

ndimitry commented Apr 28, 2021

EdSchouten commented Feb 14, 2017 •

edited

Loading

wfindley commented May 18, 2017 •

edited

Loading

michaelbv commented May 25, 2017 •

edited

Loading

agolomoodysaada commented Jul 17, 2017 •

edited

Loading

agolomoodysaada commented Jul 17, 2017 •

edited

Loading

brian-brazil commented Nov 5, 2017 •

edited

Loading

alanorth commented Nov 6, 2017 •

edited

Loading

mcastelino commented Dec 21, 2020 •

edited

Loading

fstab commented Feb 25, 2021 •

edited

Loading