Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't get JVM metrix from application's JVM #76

Closed
amradk opened this issue Aug 26, 2016 · 9 comments
Closed

Can't get JVM metrix from application's JVM #76

amradk opened this issue Aug 26, 2016 · 9 comments

Comments

@amradk
Copy link

amradk commented Aug 26, 2016

Hello!
I'm trying to use jmx_exporter to get some metrics from copy of JVM that run our company's application. I succesfully got metrics from appplication's MBeams but I can't get metric from JMV (e.g. GC CollectionCount). jmx_exporter return jmx_scrape_error 1.0 in metrics page. I use jmx_exporter in http server mode, also I tried version 0.6 and 0.7 with the same result. I get the next error in stdout:

Aug 25, 2016 10:43:09 PM io.prometheus.jmx.JmxCollector collect SEVERE: JMX scrape failed: java.lang.NullPointerException at io.prometheus.jmx.JmxScraper.processBeanValue(JmxScraper.java:242) at io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:145) at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:107) at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:325) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:72) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:64) at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:55) at io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:41) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)
I use simple config:
hostPort: 127.0.0.1:9010 lowercaseOutputName: false lowercaseOutputLabelNames: false
Also I can get JVM metrics with jconsole and other tools.

@brian-brazil
Copy link
Contributor

Can you indicate exactly which version generated that error?

@amradk
Copy link
Author

amradk commented Aug 26, 2016

This was jmx_exporter version 0.7

@brian-brazil
Copy link
Contributor

Okay, looks like an issue with a composite. What are you monitoring?

@amradk
Copy link
Author

amradk commented Aug 26, 2016

It's our company's inner application.

@brian-brazil
Copy link
Contributor

So you've custom mBeans implemented? Can you turn on the debugging and see which mBean it is and what it's doing? See http://www.robustperception.io/viewing-logs-for-the-jmx-exporter/

@amradk
Copy link
Author

amradk commented Aug 26, 2016

I turned on debugging and jmx_exporter got some scrapr failed:
FINE: scrape: 'app_default-test-default-Recos:class=com.company.model.dao.SQLQueryService$$EnhancerByGuice$$5aaf4108'_'SQLQueryStatistics': Fail: javax.management.MBeanException: java.lang.reflect.InvocationTargetException
and
FINE: scrape: 'com.company.mxcache{service=CacheControl}CachesByGroup': tabulardata Aug 26, 2016 2:48:35 AM io.prometheus.jmx.JmxCollector collect SEVERE: JMX scrape failed: java.lang.NullPointerException at io.prometheus.jmx.JmxScraper.processBeanValue(JmxScraper.java:242) at io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:145) at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:107) at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:320) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:73) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:65) at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:56) at io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:41) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)
and I can't see any metrics from JMV (such as java_lang_GarbageCollector_LastGcInfo_memoryUsageAfterGc_max{name="PS MarkSweep",key="Compressed Class Space",} 1.073741824E9) in debug output.

@brian-brazil
Copy link
Contributor

If you could debug your custom mBean and submit a unittest emulating how it's breaking we can get this fixed, as without your code I can't do much.

@hlpdlpu
Copy link

hlpdlpu commented Jan 25, 2019

Hi ,this issue also happen in my project , do you resolve it?

@amradk
Copy link
Author

amradk commented Jan 25, 2019

Hi, no the problem is not resolved. At the end our developers is implement metric exporter direct into application.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants