Turbine is unable to aggregate the response from clusters and write it back to response stream.

[rvangapal]

Hello,

I am planning to leverage hystrix dashboard and turbine which is very useful for us.

I am trying to configure turbine to aggregate metrics from clustered hystrix service apps. For some reason, it's not working. I am not sure whether my config is wrong or is it a bug. As cluster config is failing, i tried to configure single node config. Even that is also failing. I hope i have provided required configuration for you review, hope you can guide me on how to address this. Logs are captured from catalina.out.

CONFIGURATION

---

Platform: RHEL 6.3
Tomcat version: 6.0.28
Turbine version: turbine-web-1.0.0.war (downloaded from hystrix wiki)

Cluster config:

config.properties

turbine.aggregator.clusterConfig=dev-lsds
turbine.FileBasedInstanceDiscovery.filePath=/opt/usr/apps/tomcat-6.0.28/webapps/turbine/WEB-INF/classes/hosts.csv
turbine.instanceUrlSuffix=:1776/hystrix.stream
InstanceDiscovery.impl=com.netflix.turbine.discovery.FileBasedInstanceDiscovery

hosts.csv:

dev-lsds-1.tnt13-zone1.aus1,dev-lsds,up

Single node config:

config.properties

turbine.ConfigPropertyBasedDiscovery.default.instances=dev-lsds-1.tnt13-zone1.aus1
turbine.instanceUrlSuffix=:1776/hystrix.stream

OBSERVATIONS

---

Hystrix node (SUCCESS):

URL: http://dev-lsds-1.tnt13-zone1.aus1:1776/hystrix.stream
Response: It is returning successful json response

Turbine node (FAILS)

******************** SINGLE NODE CONFIG *********************

URL: http://localhost:8080/turbine/turbine.stream

Response :

curl -v http://localhost:8080/turbine/turbine.stream

• About to connect() to localhost port 8080 (#0)
• Trying ::1... connected
• Connected to localhost (::1) port 8080 (#0)

  GET /turbine/turbine.stream HTTP/1.1
  User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
  Host: localhost:8080
  Accept: /

  < HTTP/1.1 200 OK
  < Server: Apache-Coyote/1.*< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
  < Pragma: no-cache
  < Content-Type: text/event-stream;charset=UTF-8
  < Transfer-Encoding: chunked
  < Date: Thu, 12 Dec 2013 19:27:56 GMT
  <
  : ping
  : ping
  : ping
  : ping
  : ping
  : ping
  ..
  ..
  ...
  .............

---

Here is the log output for this request.
Log: http://pastebin.com/ywUjHCz9

---

******************** CLUSTER NODE CONFIG *********************

URL: http://localhost:8080/turbine/turbine.stream?cluster=dev-lsds
RESPONSE: curl -v http://localhost:8080/turbine/turbine.stream?cluster=dev-lsds

• About to connect() to localhost port 8080 (#0)
• Trying ::1... connected
• Connected to localhost (::1) port 8080 (#0)

  GET /turbine/turbine.stream?cluster=dev-lsds HTTP/1.1
  User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
  Host: localhost:8080
  Accept: /

  < HTTP/1.1 200 OK
  < Server: Apache-Coyote/1.1
  < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
  < Pragma: no-cache
  < Content-Type: text/event-stream;charset=UTF-8
  < Transfer-Encoding: chunked
  < Date: Thu, 12 Dec 2013 19:36:04 GMT
  <
  : ping
  : ping
  : ping
  : ping
  : ping

LOG1: (Same log out put as single node confg )
http://pastebin.com/ywUjHCz9

LOG2: (After couple of attempts, reached 503 max connection requests)
http://pastebin.com/PmUWH52v

Please let me know if you need more information.

[rvangapal]

Hello turbine team,

When can you get a chance, can you please look into this and reply this?

Thanks,
Ravi

[scott-seo]

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was seeing similar response from turbine as you. I read this post. #13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery
turbine.aggregator.clusterConfig=default
turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream
turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X  <== your host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands are actively generating hystrix stream. Otherwise, you won't see much in Hystrix dashboard.

Give this a try.

[benjchristensen]

What is wrong with turbine-web-1.0.0.war?

-- 
Ben Christensen - Netflix Edge Engineering
+1.310.782.5511  @benjchristensen

On September 14, 2014 at 7:43:20 AM, Scott Seo (notifications@github.com) wrote:

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was seeing similar response from turbine as you. I read this post. #13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery
turbine.aggregator.clusterConfig=default
turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream
turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X <== your host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands are actively generating hystrix stream. Otherwise, you won't see much in Hystrix dashboard.

Give this a try.

—
Reply to this email directly or view it on GitHub.

[scott-seo]

 Sorry it's hard to tell what was exactly wrong with my environment at

that point. It's most likely my environment was wrong. All I was seeing was
pings like the person who filed this original post. I was not seeing [data]
responses from Turbine. BTW, I was connecting from my local environment to
AWS. The latency was over 3.5 seconds. Debug statement did say something
about skipping to catch up with the stream.

At this point I have decided to go with recommendation made by one of

your engineers on post #13.

On Mon, Sep 15, 2014 at 12:11 PM, Ben Christensen notifications@github.com
wrote:

What is wrong with turbine-web-1.0.0.war?

Ben Christensen - Netflix Edge Engineering
+1.310.782.5511 @benjchristensen

On September 14, 2014 at 7:43:20 AM, Scott Seo (notifications@github.com)
wrote:

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was
seeing similar response from turbine as you. I read this post. #13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first
config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery

turbine.aggregator.clusterConfig=default
turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream
turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X <== your
host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands
are actively generating hystrix stream. Otherwise, you won't see much in
Hystrix dashboard.

Give this a try.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#29 (comment).

[karthik101]

@scott-seo I am trying to build turbine-web war but after building i cant find the created war. confused on to how build this web app.

[mohan-mishra]

@scott-seo I too am unable to make the war. Could you please help