ms-dashboard on Cloud Foundry - Connection with peer reset caused in failure to retrieve nodes #51

ashwgupt · 2017-02-01T12:39:02Z

In our attempt to use the microservices-dashboard for our apps running on Cloud Foundry, we have developed a vanilla spring-boot app and configured it as microservicesDashboardServer, and also as EurekaClient to enable it discover nodes via Eureka.

The same setup works fine on local environment but fails with "connection reset by peer" error when deployed on our private PCF setup.

I understand that, this is a very raw error and has very little information to find the root cause but given there is no error reported by the microservice or the eureka server, it becomes too much of a black box to dig deep.

Below is the stack-trace for the error thrown by microservices-dashboard-server app:

2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.925 DEBUG 12 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : DispatcherServlet with name 'dispatcherServlet' processing GET request for [/icon.png]
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.926 DEBUG 12 --- [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Looking up handler method for path /icon.png
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.927 DEBUG 12 --- [nio-8080-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Did not find handler method for [/icon.png]
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.927 DEBUG 12 --- [nio-8080-exec-6] o.s.w.s.handler.SimpleUrlHandlerMapping  : Matching patterns for request [/icon.png] are [/**]
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.927 DEBUG 12 --- [nio-8080-exec-6] o.s.w.s.handler.SimpleUrlHandlerMapping  : URI Template variables for request [/icon.png] are {}
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.927 DEBUG 12 --- [nio-8080-exec-6] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapping [/icon.png] to HandlerExecutionChain with handler [ResourceHttpRequestHandler [locations=[ServletContext resource [/], class path resource [META-INF/resources/], class path resource [resources/], class path resource [static/], class path resource [public/]], resolvers=[org.springframework.web.servlet.resource.PathResourceResolver@fac80]]] and 1 interceptor
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.927 DEBUG 12 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : Last-Modified value for [/icon.png] is: -1
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.928 DEBUG 12 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
2017-02-01T17:54:12.92+0530 [APP/0]      OUT 2017-02-01 12:24:12.928 DEBUG 12 --- [nio-8080-exec-6] o.s.web.servlet.DispatcherServlet        : Successfully completed request
2017-02-01T17:54:12.93+0530 [RTR/1]      OUT microsvc-dashboard.abc.com - [01/02/2017:12:24:12.922 +0000] "GET /icon.png HTTP/1.1" 304 0 0 "https://microsvc-dashboard.abc.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36" 17.17.17.17:12009 x_forwarded_for:"10.10.10.10" x_forwarded_proto:"https" vcap_request_id:57df1a82-c0bf-4371-7b74-06080217541b response_time:0.00717554 app_id:978e8012-02e9-443b-a633-115dc9ddfa02
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.963 DEBUG 12 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : DispatcherServlet with name 'dispatcherServlet' processing GET request for [/events]
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.964 DEBUG 12 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Looking up handler method for path /events
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.965 DEBUG 12 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Returning handler method [public java.util.Collection<be.ordina.msdashboard.nodes.model.SystemEvent> be.ordina.msdashboard.controllers.EventsController.getEvents()]
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.965 DEBUG 12 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Last-Modified value for [/events] is: -1
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.966 DEBUG 12 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Written [[NodeEvent{nodeId='bootadmin-ms-alpha', message='Error retrieving node(s) for url https://bootadmin-ms-alpha.abc.com/health with headers [Host=bootadmin-ms-alpha.abc.com, User-Agent=RxNetty Client]: java.io.IOException: Connection closed by peer before sending a response.', throwable=java.io.IOException: Connection closed by peer before sending a response.}, NodeEvent{nodeId='bootadmin-ms-alpha', message='Error retrieving node(s) for url https://bootadmin-ms-alpha.abc.com/mappings with headers [Host=bootadmin-ms-alpha.abc.com, User-Agent=RxNetty Client]: java.io.IOException: Connection closed by peer before sending a response.', throwable=java.io.IOException: Connection closed by peer before sending a response.}]] as "application/json" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@69c81773]
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.967 DEBUG 12 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
2017-02-01T17:54:12.96+0530 [APP/0]      OUT 2017-02-01 12:24:12.967 DEBUG 12 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Successfully completed request
2017-02-01T17:54:12.96+0530 [RTR/1]      OUT microsvc-dashboard.abc.com - [01/02/2017:12:24:12.961 +0000] "GET /events HTTP/1.1" 200 0 7191 "https://microsvc-dashboard.abc.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36" 17.17.17.17:12011 x_forwarded_for:"10.10.10.10" x_forwarded_proto:"https" vcap_request_id:cd84e5f3-7f7c-486c-6452-0c89d2db99c3 response_time:0.006505556 app_id:978e8012-02e9-443b-a633-115dc9ddfa02
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.304 DEBUG 12 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet        : DispatcherServlet with name 'dispatcherServlet' processing GET request for [/graph]
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.305 DEBUG 12 --- [nio-8080-exec-9] s.w.s.m.m.a.RequestMappingHandlerMapping : Looking up handler method for path /graph
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.305 DEBUG 12 --- [nio-8080-exec-9] s.w.s.m.m.a.RequestMappingHandlerMapping : Returning handler method [public org.springframework.http.HttpEntity<java.util.Map<java.lang.String, java.lang.Object>> be.ordina.msdashboard.controllers.GraphController.retrieveGraph()]
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.305 DEBUG 12 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet        : Last-Modified value for [/graph] is: -1
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.306  INFO 12 --- [nio-8080-exec-9] b.o.m.n.a.h.HealthIndicatorsAggregator   : Discovering services for health
2017-02-01T17:54:13.30+0530 [APP/0]      OUT 2017-02-01 12:24:13.306  INFO 12 --- [nio-8080-exec-9] b.o.m.n.a.mappings.MappingsAggregator    : Discovering services for mappings
2017-02-01T17:54:13.31+0530 [APP/0]      OUT 2017-02-01 12:24:13.314  INFO 12 --- [RxIoScheduler-6] b.o.m.n.a.h.HealthIndicatorsAggregator   : Creating health observable: (bootadmin-ms-alpha,https://bootadmin-ms-alpha.abc.com/health)
2017-02-01T17:54:13.31+0530 [APP/0]      OUT 2017-02-01 12:24:13.315  INFO 12 --- [RxIoScheduler-7] b.o.m.n.a.mappings.MappingsAggregator    : Creating mappings observable: (bootadmin-ms-alpha,https://bootadmin-ms-alpha.abc.com/mappings)
2017-02-01T17:54:13.32+0530 [APP/0]      OUT 2017-02-01 12:24:13.328 ERROR 12 --- [o-eventloop-3-1] b.o.m.nodes.aggregators.ErrorHandler     : Error retrieving node(s) for url https://bootadmin-ms-alpha.abc.com/health with headers [Host=bootadmin-ms-alpha.abc.com, User-Agent=RxNetty Client]: java.io.IOException: Connection closed by peer before sending a response.
2017-02-01T17:54:13.32+0530 [APP/0]      OUT java.io.IOException: Connection closed by peer before sending a response.
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter.<clinit>(ClientRequestResponseConverter.java:91) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.reactivex.netty.protocol.http.client.ClientRequiredConfigurator.configureNewPipeline(ClientRequiredConfigurator.java:42) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.reactivex.netty.pipeline.PipelineConfiguratorComposite.configureNewPipeline(PipelineConfiguratorComposite.java:55) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.reactivex.netty.pipeline.PipelineConfiguratorComposite.configureNewPipeline(PipelineConfiguratorComposite.java:55) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:68) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.reactivex.netty.client.RxClientImpl$2.initChannel(RxClientImpl.java:127) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelRegisteredNow(ChannelHandlerInvokerUtil.java:32) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRegistered(DefaultChannelHandlerInvoker.java:50) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRegistered(AbstractChannelHandlerContext.java:114) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.DefaultChannelPipeline.fireChannelRegistered(DefaultChannelPipeline.java:833) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:503) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$100(AbstractChannel.java:417) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:477) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:356) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.32+0530 [APP/0]      OUT    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
2017-02-01T17:54:13.33+0530 [APP/0]      OUT 2017-02-01 12:24:13.330  INFO 12 --- [RxIoScheduler-6] b.o.m.n.a.h.HealthIndicatorsAggregator   : Completed getting all health observables
2017-02-01T17:54:13.33+0530 [APP/0]      OUT 2017-02-01 12:24:13.331  INFO 12 --- [RxIoScheduler-7] b.o.m.n.a.mappings.MappingsAggregator    : Completed getting all mappings observables
2017-02-01T17:54:13.33+0530 [APP/0]      OUT 2017-02-01 12:24:13.331  INFO 12 --- [o-eventloop-3-1] b.o.m.n.a.h.HealthIndicatorsAggregator   : Completed emission of a health node observable from url: https://bootadmin-ms-alpha.abc.com/health
2017-02-01T17:54:13.33+0530 [APP/0]      OUT 2017-02-01 12:24:13.335  INFO 12 --- [o-eventloop-3-1] b.o.m.n.a.h.HealthIndicatorsAggregator   : Completed merging all health observables
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.392 ERROR 12 --- [o-eventloop-3-2] b.o.m.nodes.aggregators.ErrorHandler     : Error retrieving node(s) for url https://bootadmin-ms-alpha.abc.com/mappings with headers [Host=bootadmin-ms-alpha.abc.com, User-Agent=RxNetty Client]: java.io.IOException: Connection closed by peer before sending a response.
2017-02-01T17:54:13.39+0530 [APP/0]      OUT java.io.IOException: Connection closed by peer before sending a response.
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter.<clinit>(ClientRequestResponseConverter.java:91) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.reactivex.netty.protocol.http.client.ClientRequiredConfigurator.configureNewPipeline(ClientRequiredConfigurator.java:42) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.reactivex.netty.pipeline.PipelineConfiguratorComposite.configureNewPipeline(PipelineConfiguratorComposite.java:55) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.reactivex.netty.pipeline.PipelineConfiguratorComposite.configureNewPipeline(PipelineConfiguratorComposite.java:55) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.reactivex.netty.client.RxClientImpl$2.initChannel(RxClientImpl.java:127) ~[rxnetty-0.5.1.jar!/:0.5.1]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:68) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelRegisteredNow(ChannelHandlerInvokerUtil.java:32) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRegistered(DefaultChannelHandlerInvoker.java:50) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRegistered(AbstractChannelHandlerContext.java:114) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.DefaultChannelPipeline.fireChannelRegistered(DefaultChannelPipeline.java:833) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:503) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$100(AbstractChannel.java:417) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:477) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:356) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.1.0.Beta7.jar!/:4.1.0.Beta7]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.392  INFO 12 --- [o-eventloop-3-2] b.o.m.n.a.mappings.MappingsAggregator    : Completed emission of a mapping node observable from url: https://bootadmin-ms-alpha.abc.com/mappings
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.393  INFO 12 --- [o-eventloop-3-2] b.o.m.n.a.mappings.MappingsAggregator    : Completed merging all mappings observables
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.394  INFO 12 --- [RxIoScheduler-5] b.o.msdashboard.graph.GraphRetriever     : Merged all emitted nodes, converting to map
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.394  INFO 12 --- [RxIoScheduler-5] b.o.msdashboard.graph.GraphRetriever     : Converted to nodes and links map
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.395  INFO 12 --- [nio-8080-exec-9] b.o.msdashboard.graph.GraphRetriever     : Graph retrieved: {nodes=[], links=[]}
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.396 DEBUG 12 --- [nio-8080-exec-9] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Written [{directed=true, types=[DB, MICROSERVICE, REST, SOAP, JMS, RESOURCE], nodes=[], lanes=[{type=UI Components, lane=0}, {type=Resources, lane=1}, {type=Microservices, lane=2}, {type=Backends, lane=3}], links=[], multigraph=false, graph=[Ljava.lang.String;@4934b630}] as "application/json" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@69c81773]
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.396 DEBUG 12 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet        : Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling
2017-02-01T17:54:13.39+0530 [APP/0]      OUT 2017-02-01 12:24:13.396 DEBUG 12 --- [nio-8080-exec-9] o.s.web.servlet.DispatcherServlet        : Successfully completed request
2017-02-01T17:54:13.39+0530 [RTR/1]      OUT microsvc-dashboard.abc.com - [01/02/2017:12:24:13.301 +0000] "GET /graph HTTP/1.1" 200 0 267 "https://microsvc-dashboard.abc.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36" 17.17.17.17:12033 x_forwarded_for:"10.10.10.10" x_forwarded_proto:"https" vcap_request_id:aadbaf82-a574-48f2-67ca-a68dfacc1cfe response_time:0.095483598 app_id:978e8012-02e9-443b-a633-115dc9ddfa02

The dashboard-server app is able to find the correct url for registered nodes from Eureka but is not able to perform the successful GET request. The same url works fine when used otherwise.

Is there any obvious config that is missed out wrt running it on Cloud Foundry?

And is there any way to avoid a Discovery server and enable microservices as client app for dashboard-server to connect directly?

The text was updated successfully, but these errors were encountered:

ashwgupt · 2017-02-10T06:58:38Z

@TYsewyn @andreasevers
Given there is no response to this issue and give the last activity in the project was way long back, can it be assumed that the project is no longer under focus and is getting obsolete slowly?

I am keen to learn if the project is being replaced with something new.

ashwgupt · 2017-02-10T07:04:05Z

This issue is happening only when we try to deploy and run the topology on Cloud Foundry.

The error indicates that, the dashboard failed to retrieve node(s) for the micro-service's url
[o-eventloop-3-1] b.o.m.nodes.aggregators.ErrorHandler : Error retrieving node(s) for url https://bootadmin-ms-alpha.paasnpoa.abc.com/health with headers [Host=bootadmin-ms-alpha.paasnpoa.bip.uk.fid-intl.com, User-Agent=RxNetty Client]: java.io.IOException: Connection closed by peer before sending a response.

Whereas the /health endpoint of the application returns a valid response when hit directly from a browser:

{"description":"Spring Cloud Eureka Discovery Client","status":"UP","discoveryComposite":{"description":"Spring Cloud Eureka Discovery Client","status":"UP","discoveryClient":{"description":"Spring Cloud Eureka Discovery Client","status":"UP","services":["bootadmin-ms-alpha","eureka-server-oak-02","eureka-server-oak-01","eureka-server-act-02","eureka-server-act-01"]},"eureka":{"description":"Remote status from Eureka server","status":"UP","applications":{"BOOTADMIN-MS-ALPHA":1,"EUREKA-SERVER-OAK-02":1,"MICROSVC-DASHBOARD":0,"EUREKA-SERVER-OAK-01":1,"EUREKA-SERVER-ACT-02":1,"EUREKA-SERVER-ACT-01":1}}},"diskSpace":{"status":"UP","total":1056858112,"free":888377344,"threshold":10485760},"refreshScope":{"status":"UP"},"hystrix":{"status":"UP"}}

Shows that the integration is going incorrect somewhere when run on CF (or a container based platform) but unsure where and what.

The same topology works fine on localhost, where also the response from micro-service's health endpoint returns similar response as from instances on CF.

Any inputs will be of help.

ashwgupt · 2017-02-10T09:08:38Z

On a closer look I believe it's an issue with the values showing up in header as [Host=bootadmin-ms-alpha.paasnpoa.bip.uk.fid-intl.com, User-Agent=RxNetty Client]. Given there is no real host name for apps on CF, am unsure if this is causing issue.

Do you have any records where the microservices dashboard has been run on Cloud Foundry? If any, the config from the project will be of high use.

TYsewyn · 2017-02-10T11:18:03Z

Hi @ashwgupt
First of all, our sincere apologies for the late respons!
Since we are developing this solution in our free time and had/have some big deadlines at our customers, we didn't have the time to work on this project.

At this moment we didn't run the ms dashboard server on CF ourselves, and we don't know if others did try this before you.
So it would be a lot of help if you could check this out and create issues where there are problems.
We do accept PR and would really appreciate your help where you can! :)

As for now, I did see some issues where RxNetty tries to connect to other apps over https.
I think this is the result of a lib clash caused by following issue ReactiveX/RxNetty#439.
This is not easy to fix, since upgrading the version will imply that we need to fix our codebase too.

ashwgupt · 2017-02-10T11:32:48Z

@TYsewyn thanks a lot for your response. I understand it's a work done at a stretch by you guys and really appreciate the value you give to community by delivering such great tools.

Wrt the issue - it's sad to hear that, connections to microservices over https give issue. We had been very hopeful use the tool in our eco-system.

Btw, is it only specific to https and not for http?

I will see what can be done, if anything, to make it working and share the findings with you. I would be grateful if you can inform back on this issue, if you have a solution/upgrade ready first.

ashwgupt · 2017-02-10T11:44:53Z

@TYsewyn can you also advise if I simply replace the RxNetty version in my fork of the project to 0.4.20 (latest of 0.4 series) and use that build instead, will it give me a workaround for the issue?

ashwgupt · 2017-02-10T12:48:40Z

FYI - I've tried by pulling the version for RxNetty to 0.4.20 and using that package for my micro-services dashboard.

That have resolved an another issue that was seeing so far i.e. incorrect status of Microservices-Dashboard on Eureka console. For all this while it was showing as DOWN for no known reason (and I had plans to look deep into it at later point). The status is showing correctly as UP there, which indicates that things have improved wrt communication for https endpoints.

However it looks like have broken the whole of dashboard app now, as the context path / is now throwing "GET / HTTP/1.1" 404 0 306.

If you know any off hand reason for it, please share or else I shall look into it on Monday.

TYsewyn · 2017-02-11T14:51:42Z

Hi @ashwgupt I'm working on a solution as we speak, but I will need your help to test it later.
Can you tell me if the certificates are signed with trusted CAs?
Unfortunately I can only test the solution locally on my machine with self signed certificates.
But if it works with self signed certificates it should definitely work with certificates signed by a trusted CA.

ashwgupt · 2017-02-12T02:20:01Z

Fortunately we also are on self sighed certs but as you said if it works for self-signed, working for CA Trusted would be obvious.

ashwgupt · 2017-02-13T03:36:20Z

And yes, I will be more than happy to try out your fix.

TYsewyn · 2017-02-13T17:01:59Z

Hi @ashwgupt
Can you create a local build of the server? You can find the solution on the fix/51-ssl-issue branch.
In your application you will need to create a bean where you can whitelist your self signed certificate.
Or you can trust all connections like the code below, but we don't recommend it.

@Bean
public CompositeHttpClient<ByteBuf, ByteBuf> rxClient() {
        return new CompositeHttpClientBuilder<ByteBuf, ByteBuf>()
                .withSslEngineFactory(DefaultFactories.trustAll()).build();
}

TYsewyn · 2017-02-13T17:17:57Z

There appears to be a build issue. I'll get back to you ASAP.

EDIT:
Some previous code was (not) committed, but this is now resolved.
You can try out the fix branch if you'd like @ashwgupt.

ashwgupt · 2017-02-14T12:17:58Z

Sure, will try it first thing tomorrow.

ashwgupt · 2017-02-15T05:19:25Z

Looks like something is still broken. The application starts okay on Cloud (PCF) but fails to host the '/' context path for the dashboard to launch. When the application url is hit on root path, HTTP-404 is received.

Any thoughts, why?

ashwgupt · 2017-02-15T05:28:47Z

All that I've added in my Dashboard app is besides using @EnableMicroservicesDashboardServer is:

Change POM to use locally built microservice-dashboard jar:

<dependency>
	<groupId>be.ordina</groupId>
	<artifactId>microservices-dashboard-server</artifactId>
	<version>1.0.2-SNAPSHOT</version>
	<scope>system,import</scope>
	<systemPath>/microservices-dashboard-server/target/microservices-dashboard-server-1.0.2-SNAPSHOT.jar</systemPath>
</dependency>

Add the new Bean config

@Configuration
public class NettyConfig {

    @Bean
    public CompositeHttpClient<ByteBuf, ByteBuf> rxClient() {
        return new CompositeHttpClientBuilder<ByteBuf, ByteBuf>()
                .withSslEngineFactory(DefaultFactories.trustAll()).build();
    }
}

ashwgupt · 2017-02-15T07:27:35Z

Removed and updated past few comments to keep it simple and easy for anyone to refer later.

TYsewyn · 2017-02-15T09:43:05Z

If I understand correctly, you only have the HTTP-404 issue now?
I'll try to check out this issue later today locally.

ashwgupt · 2017-02-15T10:05:55Z

Yes, that's correct!

None of the controller endpoints, defined in the GraphController class, are available for the Dashboard App when using the new version of microservices-dashboard.

Btw, is this testing not part of your test suite? I couldn't find any relevant test somehow.

TYsewyn · 2017-02-16T19:29:09Z

@ashwgupt I got the build working locally with self signed certificates. I will adapt one of the samples and try to deploy it on PWS.
I'll keep you posted! :)

ashwgupt · 2017-02-21T04:29:28Z

@TYsewyn - Have you got a chance to look at this issue any further?

TYsewyn · 2017-02-22T22:45:13Z

Hi @ashwgupt, I'm really sorry to keep you waiting!
Something came up at work so I didn't have the chance to deploy and test a small test setup.
If all goes well I should be able to finish this by the end of the week.

ashwgupt · 2017-02-23T03:39:30Z

Thanks for letting know @TYsewyn

No need to apologise, we completely understand your situation and value your support.

Looking forward to the fix.

ashwgupt · 2017-03-01T03:53:30Z

@TYsewyn just to simplify the investigation - you could even try it for a sample run locally. This behaviour issue is not particular to PCF and for me the tests are failing even when run locally.

MockMvc mvc = webAppContextSetup(webApplicationContext).build();

        mvc.perform(get("/"))
                .andExpect(status().isOk());

That will remove the need of PCF setup for you and make it simple to test.

TYsewyn · 2017-03-05T00:50:28Z

Hi @ashwgupt!

We currently have 2 versions running on PWS:
1.0.1: https://msdashboard-test-dashboard.cfapps.io/
1.0.2-SNAPSHOT: https://msdashboard-test-dashboard-snapshot.cfapps.io/

We can't verify the self-signed certificates on PWS, but we can see the interface and the discovered mappings, microservices & backends.
The snapshot was built using the fix/51-ssl-issue branch.

The only thing that is different is the scope of the dependency.
We didn't change this, so we use the default compile scope instead of system, import.

ashwgupt · 2017-03-06T05:14:41Z

@TYsewyn can you please upload and share the app repo's github location for our ref to compare?

ashwgupt · 2017-03-09T12:57:16Z

@TYsewyn are you able to provide me with a repo url for your code please?

andreasevers · 2017-03-11T16:52:02Z

Do you mean the microservices-dashboard-server code?
https://github.com/ordina-jworks/microservices-dashboard-server/tree/fix/51-ssl-issue

ashwgupt · 2017-03-11T16:59:42Z

Nope, that I already have and used while testing the fix. I'm interested in your app's code that is working as MS Dashboard code with the Snapshot Sever, as I am receiving 404 when trying to hit the root page of my app.

ashwgupt · 2017-03-20T05:55:26Z

@TYsewyn - It will really help us confirm if the solution has helped and also to accelerate us to move forward with its adoption if you could please share the working code repo url for running app with 1.0.2-SNAPSHOT.

TYsewyn · 2017-03-22T20:00:48Z

Hi @ashwgupt
I'm really sorry for the long delay. Things were really crazy the last 3 weeks!
You can find the 2 microservices and the dashboard application in this repository: https://github.com/ordina-jworks/microservices-dashboard-test-pws

Because we deployed the applications on PWS, we are using spring-cloud-services-dependencies to connect to the eureka "registry-as-a-service" (as you can see in manifest.yml).

ashwgupt · 2017-03-24T11:17:30Z

@TYsewyn - Thanks for your response but am still not able to make my application work and the root page gives a 404.

Can you please confirm if this bean needed any more or not to be present in the dashboard app?

@Bean
public CompositeHttpClient<ByteBuf, ByteBuf> rxClient() {
    return new CompositeHttpClientBuilder<ByteBuf, ByteBuf>()
            .withSslEngineFactory(DefaultFactories.trustAll()).build();
}

ashwgupt · 2017-03-24T11:36:22Z

I also tried using your dashboard application without the above Bean of CompositeHttpClient type.

The only change is that the app connects to Eureka service running as an app (as we don't have Spring Cloud Service tile in our private PCF installation), and we are using system scope for microservices-dashboard-server jar file (as no other scope seem to help us).

The behaviour is same as my original app, where all actuator endpoints work fine but the root path for dashboard \ throws a http-404.

No errors in logs to indicate any issue or root cause. Looks like I'm on some edge case.

Any hints?

TYsewyn self-assigned this Feb 11, 2017

TYsewyn added the type: bug A general bug label Feb 11, 2017

TYsewyn added the status: waiting-for-feedback We need additional information before we can continue label Feb 11, 2017

TYsewyn removed the status: waiting-for-feedback We need additional information before we can continue label Feb 13, 2017

TYsewyn mentioned this issue Feb 13, 2017

Update documentation with custom rxNetty client #52

Closed

TYsewyn added the status: waiting-for-feedback We need additional information before we can continue label Feb 14, 2017

TYsewyn added this to the 1.0.2 milestone Mar 22, 2017

TYsewyn mentioned this issue Mar 29, 2017

Added support for self-signed certificates #55

Merged

TYsewyn closed this as completed in #55 Apr 24, 2017

ashwgupt mentioned this issue May 2, 2017

1.0.2-SNAPSHOT results in HTTP_NOTFOUND for any of its controller endpoints #57

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ms-dashboard on Cloud Foundry - Connection with peer reset caused in failure to retrieve nodes #51

ms-dashboard on Cloud Foundry - Connection with peer reset caused in failure to retrieve nodes #51

ashwgupt commented Feb 1, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017

TYsewyn commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 •

edited

Loading

TYsewyn commented Feb 11, 2017

ashwgupt commented Feb 12, 2017

ashwgupt commented Feb 13, 2017

TYsewyn commented Feb 13, 2017

TYsewyn commented Feb 13, 2017 •

edited

Loading

ashwgupt commented Feb 14, 2017

ashwgupt commented Feb 15, 2017 •

edited

Loading

ashwgupt commented Feb 15, 2017 •

edited

Loading

ashwgupt commented Feb 15, 2017

TYsewyn commented Feb 15, 2017

ashwgupt commented Feb 15, 2017 •

edited

Loading

TYsewyn commented Feb 16, 2017

ashwgupt commented Feb 21, 2017

TYsewyn commented Feb 22, 2017

ashwgupt commented Feb 23, 2017

ashwgupt commented Mar 1, 2017

TYsewyn commented Mar 5, 2017 •

edited

Loading

ashwgupt commented Mar 6, 2017

ashwgupt commented Mar 9, 2017

andreasevers commented Mar 11, 2017 •

edited

Loading

ashwgupt commented Mar 11, 2017

ashwgupt commented Mar 20, 2017

TYsewyn commented Mar 22, 2017

ashwgupt commented Mar 24, 2017

ashwgupt commented Mar 24, 2017 •

edited

Loading

ms-dashboard on Cloud Foundry - Connection with peer reset caused in failure to retrieve nodes #51

ms-dashboard on Cloud Foundry - Connection with peer reset caused in failure to retrieve nodes #51

Comments

ashwgupt commented Feb 1, 2017 • edited Loading

ashwgupt commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 • edited Loading

ashwgupt commented Feb 10, 2017

TYsewyn commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 • edited Loading

ashwgupt commented Feb 10, 2017

ashwgupt commented Feb 10, 2017 • edited Loading

TYsewyn commented Feb 11, 2017

ashwgupt commented Feb 12, 2017

ashwgupt commented Feb 13, 2017

TYsewyn commented Feb 13, 2017

TYsewyn commented Feb 13, 2017 • edited Loading

ashwgupt commented Feb 14, 2017

ashwgupt commented Feb 15, 2017 • edited Loading

ashwgupt commented Feb 15, 2017 • edited Loading

ashwgupt commented Feb 15, 2017

TYsewyn commented Feb 15, 2017

ashwgupt commented Feb 15, 2017 • edited Loading

TYsewyn commented Feb 16, 2017

ashwgupt commented Feb 21, 2017

TYsewyn commented Feb 22, 2017

ashwgupt commented Feb 23, 2017

ashwgupt commented Mar 1, 2017

TYsewyn commented Mar 5, 2017 • edited Loading

ashwgupt commented Mar 6, 2017

ashwgupt commented Mar 9, 2017

andreasevers commented Mar 11, 2017 • edited Loading

ashwgupt commented Mar 11, 2017

ashwgupt commented Mar 20, 2017

TYsewyn commented Mar 22, 2017

ashwgupt commented Mar 24, 2017

ashwgupt commented Mar 24, 2017 • edited Loading

ashwgupt commented Feb 1, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017 •

edited

Loading

ashwgupt commented Feb 10, 2017 •

edited

Loading

TYsewyn commented Feb 13, 2017 •

edited

Loading

ashwgupt commented Feb 15, 2017 •

edited

Loading

ashwgupt commented Feb 15, 2017 •

edited

Loading

ashwgupt commented Feb 15, 2017 •

edited

Loading

TYsewyn commented Mar 5, 2017 •

edited

Loading

andreasevers commented Mar 11, 2017 •

edited

Loading

ashwgupt commented Mar 24, 2017 •

edited

Loading