Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Targets UNHEALTHY problem #1182

Closed
sundy-li opened this Issue Oct 26, 2015 · 8 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@sundy-li @grobie @brian-brazil
@sundy-li
Copy link

sundy-li commented Oct 26, 2015

Endpoint State:UNHEALTHY
http://192.168.10.51:8075/metrics
server returned HTTP status 500 Internal Server Error

But I am sure that http://192.168.10.51:8075/metrics is fine.It happens by accident, but when I change the port 8075 to others and restart the prometheus it will be ok. But it only works for a while.
@grobie

This comment has been minimized.

Sign in to view
Copy link
Member

grobie commented Oct 26, 2015

What is the target, an exporter or a server using a client library? Which client library / language? Anything in the logs of the target?
@sundy-li

This comment has been minimized.

Sign in to view
Copy link
Author

sundy-li commented Oct 26, 2015

This target url from my monitor written by java, and I prodive a http route GET "/metrics" for prometheus, for most time it works fine, but will be State:UNHEALTHY by accident, and I am sure the target url works fine in any time.
@grobie

This comment has been minimized.

Sign in to view
Copy link
Member

grobie commented Oct 26, 2015

Well, the response code 500 is a strong indicator that something is off on the target side.

I assume you use the prometheus/client_java in the newest version in your service?

Is there absolutely nothing in your server logs for these requests? Next time you see a 500, can you run this immediately afterwards and check the response code?

curl -H 'Accept: application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited' http://192.168.10.51:8075/metrics
@sundy-li

This comment has been minimized.

Sign in to view
Copy link
Author

sundy-li commented Oct 26, 2015

Hi, I think I got the problem by your advice.

curl -H 'Accept: application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited' http://192.168.10.51:8075/metrics

results to this error log

   <html><head><title>Apache Tomcat/7.0.52 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Request processing failed; nested exception is java.lang.IllegalStateException: getOutputStream() has already been called for this response</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Request processing failed; nested exception is java.lang.IllegalStateException: getOutputStream() has already been called for this response</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: getOutputStream() has already been called for this response
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:973)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
</pre></p><p><b>root cause</b> <pre>java.lang.IllegalStateException: getOutputStream() has already been called for this response
    org.apache.catalina.connector.Response.getWriter(Response.java:636)
    org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:213)
    io.prometheus.client.utility.servlet.MetricsServlet.dumpJson(MetricsServlet.java:35)
    io.prometheus.client.utility.servlet.MetricsServlet.doGet(MetricsServlet.java:65)
    com.sf.monitor.controllers.MetricsController.metrics(MetricsController.java:18)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:497)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.52 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.52</h3></body></html>

But curl http://192.168.10.51:8075/metrics get normal http 200 code

While my code is as fllows:

 package com.sf.monitor.controllers;

import io.prometheus.client.utility.servlet.MetricsServlet;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

@Controller
@RequestMapping("/metrics")
public class MetricsController extends MetricsServlet {

  @RequestMapping()
  public void metrics(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    super.doGet(req, resp);
  }
}

When I change the port to 8076, it got ok. I didnot know why
@sundy-li

This comment has been minimized.

Sign in to view
Copy link
Author

sundy-li commented Oct 26, 2015

The maven dependency is

  <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>client</artifactId>
      <version>0.0.10</version>
    </dependency>
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Oct 26, 2015

This looks like an error in whatever filters you have configured. You should also use the java simpleclient at https://github.com/prometheus/client_java, as the original client you are using is deprecated.
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Dec 16, 2015

If this is still a problem when using the simpleclient, please let us know.

@brian-brazil brian-brazil closed this Dec 16, 2015

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.