Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to connect to Command Metric Stream #1566

Open
AshutoshSharma10 opened this issue May 5, 2017 · 32 comments
Open

Unable to connect to Command Metric Stream #1566

AshutoshSharma10 opened this issue May 5, 2017 · 32 comments

Comments

@AshutoshSharma10
Copy link

@AshutoshSharma10 AshutoshSharma10 commented May 5, 2017

Hi,

We are using the Spring boot starter parent 1.5.2.RELEASE and hystrix 1.5.6 dependencies.
Fortunately it works once in hundred tries. We are really puzzled, why it works once and then never. All the services are running on the local machine.
The log details are as below:
Proxy opening connection to: http://localhost:6001/hystrix.stream
2017-05-05 12:29:16.951 INFO 4876 --- [nio-6001-exec-5] ashboardConfiguration$ProxyStreamServlet :
Proxy opening connection to: http://localhost:6001/hystrix.stream
2017-05-05 12:31:36.163 INFO 4876 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration

We are using the actuator also and tried the management.port once but the result is same. So we removed it. The same thing we tried with turbine and it also behaves the same way.

Please suggest.

Below is the resulting screen:
image

@mohanaraosv

This comment has been minimized.

Copy link

@mohanaraosv mohanaraosv commented May 8, 2017

@AshutoshSharma10 Could you please create sample project which simulates the above problem. Is the server polling the data?

@mattrjacobs

This comment has been minimized.

Copy link
Contributor

@mattrjacobs mattrjacobs commented May 9, 2017

@AshutoshSharma10 It's very difficult to provide an answer with those details. One possible way of getting to the bottom of it would be for you to look at the sample application that properly serves the metrics stream, and find out where yours differs.

https://github.com/Netflix/Hystrix/tree/master/hystrix-examples-webapp

@michelbetancourt

This comment has been minimized.

Copy link

@michelbetancourt michelbetancourt commented May 11, 2017

I want to plus this issue -- we are seeing the same issue with Spring-boot 1.4.x and not just 1.5.x.

If we try to curl the end point instead of getting back a snapshot of data, we get

Previous versions of Spring-boot 1.3.x did not have an issue. At first sight, this seems to be an incompatibility with Spring-boot or one of it's dependencies since our team had not had issues until a recent upgrade with Hystrix.

Noteworthy comment is that all other parts of Hystrix are working just fine. RequestLog, and at least Circuit Breaking are working as designed. For some reason the Hystrix Streaming Servlet is no longer functional for us.

Is there any way to troubleshoot this issue?

The symptoms we receive for this issue are as follows:

  • Unable to connect to Command Stream (per description of ticket)
  • Access log receives 503 http status entry for hystrix.stream end point
  • The curl below hangs:
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /debug/hystrix.stream HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
>```
@michelbetancourt

This comment has been minimized.

Copy link

@michelbetancourt michelbetancourt commented May 11, 2017

If time permits we will try to put together a sandbox sample app, if one is not already provided.

@michelbetancourt

This comment has been minimized.

Copy link

@michelbetancourt michelbetancourt commented May 11, 2017

Found the issue on our end. We added a delay in the writing of the response so that we can propagate headers in our response the way we needed them. Hystrix Servlet does not play well with our Response-delaying mechanism.

@615Manu

This comment has been minimized.

Copy link

@615Manu 615Manu commented Jun 3, 2017

Hello,

We are also having a spring boot app and trying to see the metrics on the hystrix dashboard,
But getting exactly the same issue.
Is there any update on this issue. We are using hystrix 1.5.3.
Any fix or workaround would be helpful
Thanks in Advance

@mattrjacobs

This comment has been minimized.

Copy link
Contributor

@mattrjacobs mattrjacobs commented Jun 11, 2017

@615Manu The original issue was resolved. Every circumstance is potentially different, and you haven't listed any details about your problem.

I would start by looking at the sample application that properly serves the metrics stream, and find out where yours differs.

https://github.com/Netflix/Hystrix/tree/master/hystrix-examples-webapp

@asgs

This comment has been minimized.

Copy link

@asgs asgs commented Aug 24, 2017

@615Manu @AshutoshSharma10 For spring boot apps to show Hystrix metrics, the spring starter spring-boot-starter-actuator should be added to your dependency. This will apparently enable the metrics filter.

Also, make sure you open the http://<host>:<port>/hystrix/monitor URL and type http://<host>:<port>/hystrix.stream in the textbox. Then hit the Monitor Stream button to see the live stats.

@sunday2

This comment has been minimized.

Copy link

@sunday2 sunday2 commented Feb 28, 2018

i also encountered the same problem ,and the version is 1.4.3

@Credo-Zhao

This comment has been minimized.

Copy link

@Credo-Zhao Credo-Zhao commented Mar 7, 2018

I have the same question...version is 2.0

@tomfrans

This comment has been minimized.

Copy link

@tomfrans tomfrans commented Mar 17, 2018

management.endpoints.web.exposure.include=* adding in application.properties solved for me

@spencergibb

This comment has been minimized.

Copy link
Contributor

@spencergibb spencergibb commented Mar 17, 2018

This isn't the spring cloud Netflix repository

@ezhong0812

This comment has been minimized.

Copy link

@ezhong0812 ezhong0812 commented Mar 20, 2018

same question , version 2.0.0 ,
http://localhost:8764/hystrix.stream get the same page with http://localhost:8764/hystrix

@spencergibb

This comment has been minimized.

Copy link
Contributor

@spencergibb spencergibb commented Mar 20, 2018

This isn't the spring cloud Netflix repository

@cion-J

This comment has been minimized.

Copy link

@cion-J cion-J commented Mar 26, 2018

version 2.0.0.
"management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties;
and the url was change to "http://host:port/actuator/hystrix.stream", it is ok.
also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

@moviewang

This comment has been minimized.

Copy link

@moviewang moviewang commented Apr 3, 2018

@cion-J Yours answer solved for me! thks!

@ww530220264

This comment has been minimized.

Copy link

@ww530220264 ww530220264 commented Apr 4, 2018

hi,i just occured this problem,after one day, i may have solved this,please add a class in your project

version:2.0

package com.sc.consumerribbon.config;

import com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@configuration
public class HystrixServletDefinitions {

@Bean(name = "hystrixRegistrationBean")
public ServletRegistrationBean servletRegistrationBean() {
    ServletRegistrationBean registration = new ServletRegistrationBean(
            new HystrixMetricsStreamServlet(), "/hystrix.stream");
    registration.setName("hystrixServlet");
    registration.setLoadOnStartup(1);
    return registration;
}

}
@AshutoshSharma10
可以看下我记录的
https://blog.csdn.net/qq_22772259/article/details/79812026

@1029574707

This comment has been minimized.

Copy link

@1029574707 1029574707 commented Apr 20, 2018

@ww530220264
Thanks guy, I have solved this problem in my project with your method.

easonn added a commit to easonn/Snoax-m that referenced this issue Apr 24, 2018
@playerlilin

This comment has been minimized.

Copy link

@playerlilin playerlilin commented Apr 26, 2018

@ww530220264 Thanks for a lot,your method is great.

@rajeshrengarajulu

This comment has been minimized.

Copy link

@rajeshrengarajulu rajeshrengarajulu commented Apr 27, 2018

The above solution is for 2.0.x. However it's still not working with Spring-boot 1.4.x and 1.5.x. Any updates on that?

@spencergibb

This comment has been minimized.

Copy link
Contributor

@spencergibb spencergibb commented Apr 27, 2018

This isn't the spring cloud Netflix repository

@duanhaowei

This comment has been minimized.

Copy link

@duanhaowei duanhaowei commented May 5, 2018

this solution is great , so may be is a miss part of cloud document? thanks @ww530220264

@kennyk65

This comment has been minimized.

Copy link

@kennyk65 kennyk65 commented Jun 22, 2018

Solution from @cion-J is great! Spring Cloud docs require minor adjustment to reflect the change. My suggested PR: kennyk65/spring-cloud-netflix#1

@weiwunb

This comment has been minimized.

Copy link

@weiwunb weiwunb commented Jul 12, 2018

spring cloud doc is really sucks...

@lennyxue

This comment has been minimized.

Copy link

@lennyxue lennyxue commented Aug 23, 2018

just added one class as below :
@configuration
public class HystrixServletDefinitions {
@bean(name = "hystrixRegistrationBean")
public ServletRegistrationBean servletRegistrationBean() {
ServletRegistrationBean registration = new ServletRegistrationBean(
new HystrixMetricsStreamServlet(), "/metrics/hystrix.stream");
registration.setName("hystrixServlet");
registration.setLoadOnStartup(1);
return registration;
}
}

issues resolved.

@MrSummer33

This comment has been minimized.

Copy link

@MrSummer33 MrSummer33 commented Sep 4, 2018

if there was no HystrixCommand data , the hystrix server log "java.net.SocketTimeoutException: Read timed out" . it also show you "Unable to connect to Command Metric Stream" on the dashboard .
so you can request the HystrixCommad .then it will be ok
the detail info in HystrixSampleSseServlet.handleRequest()
image

@romankh3

This comment has been minimized.

Copy link

@romankh3 romankh3 commented Sep 28, 2018

version 2.0.0.
"management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties;
and the url was change to "http://host:port/actuator/hystrix.stream", it is ok.
also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

I have the same problem and the solution above works for me!
Look very strange and not user-friendly...

@BroniusJanonis

This comment has been minimized.

Copy link

@BroniusJanonis BroniusJanonis commented Oct 1, 2018

For me on Spring-Boot version 2 helped
with dependencies

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

Added anotation to aplication
@EnableHystrixDashboard
Added bootstrap poperties (not sure why in some examples they are not using or why are they using here)

management:
  endpoints:
    web:
      exposure:
        include: hystrix.stream
      base-path: /

Run the app
Opened first http://host:port/hystrix then added in the url http://host:port/hystrix.stream and it worked
If I used http://host:port/actuator/hystrix.stream I was getting errors

@rajanikanthbhagavan

This comment has been minimized.

Copy link

@rajanikanthbhagavan rajanikanthbhagavan commented Feb 9, 2019

version 2.0.0.
"management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties;
and the url was change to "http://host:port/actuator/hystrix.stream", it is ok.
also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

This worked for me, thank you!!

@Sicmatr1x

This comment has been minimized.

Copy link

@Sicmatr1x Sicmatr1x commented Mar 20, 2019

You should add both @EnableHystrixDashboard and @EnableCircuitBreaker

like this:

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
@EnableHystrixDashboard
@EnableCircuitBreaker
public class ConsumerApplication {
	public static void main(String[] args) {
		SpringApplication.run(ConsumerApplication.class, args);
	}
}
@duansiman

This comment has been minimized.

Copy link

@duansiman duansiman commented Jun 16, 2019

微信图片_20190616232536
http://localhost:59001/actuator/hystrix.stream
Fill in the link here and change the port to your administrative port. note the 'actuator'

@hxd890724

This comment has been minimized.

Copy link

@hxd890724 hxd890724 commented Aug 13, 2019

hi,i just occured this problem,after one day, i may have solved this,please add a class in your project

version:2.0

package com.sc.consumerribbon.config;

import com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@configuration
public class HystrixServletDefinitions {

@Bean(name = "hystrixRegistrationBean")
public ServletRegistrationBean servletRegistrationBean() {
    ServletRegistrationBean registration = new ServletRegistrationBean(
            new HystrixMetricsStreamServlet(), "/hystrix.stream");
    registration.setName("hystrixServlet");
    registration.setLoadOnStartup(1);
    return registration;
}

}
@AshutoshSharma10
可以看下我记录的
https://blog.csdn.net/qq_22772259/article/details/79812026

Solved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.