Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spring boot AI and Spring Cloud Zuul problem #676

Closed
matyee opened this issue Jun 1, 2018 · 8 comments · Fixed by #680
Closed

Spring boot AI and Spring Cloud Zuul problem #676

matyee opened this issue Jun 1, 2018 · 8 comments · Fixed by #680
Assignees
Labels
Bug Spring Boot Starter Spring boot starter issues, feature asks and questions

Comments

@matyee
Copy link

matyee commented Jun 1, 2018

Hi,

First of all, the AI for JAVA is awesome. I am using it with multiple Spring Boot based applications and they work fine both the SDK and the Agent. I use Netflix's Zuul proxy through the Spring Cloud Zuul project. The SDK works well with the edge service, but when I add the agent, the calls made against the edge service throws the following error:

com.netflix.zuul.exception.ZuulException: Filter threw Exception
	at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:227) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.3.1.jar:1.3.1]
	at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:165) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44) [spring-cloud-netflix-zuul-2.0.0.RC2.jar:2.0.0.RC2]
	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) [spring-webmvc-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) [spring-boot-actuator-2.0.2.RELEASE.jar:2.0.2.RELEASE]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) [spring-boot-actuator-2.0.2.RELEASE.jar:2.0.2.RELEASE]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) [spring-boot-actuator-2.0.2.RELEASE.jar:2.0.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84) [spring-boot-actuator-2.0.2.RELEASE.jar:2.0.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at com.microsoft.applicationinsights.web.internal.WebRequestTrackingFilter.doFilter(WebRequestTrackingFilter.java:93) [applicationinsights-web-2.1.1.jar:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar:5.0.6.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar:8.5.31]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IncompatibleClassChangeError: Class org.apache.http.message.BasicHttpRequest does not implement the requested interface org.apache.http.client.methods.HttpUriRequest
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java) ~[httpclient-4.5.5.jar:4.5.5]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.5.jar:4.5.5]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:355) ~[spring-cloud-netflix-zuul-2.0.0.RC2.jar:2.0.0.RC2]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:274) ~[spring-cloud-netflix-zuul-2.0.0.RC2.jar:2.0.0.RC2]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:209) ~[spring-cloud-netflix-zuul-2.0.0.RC2.jar:2.0.0.RC2]
	at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:117) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193) ~[zuul-core-1.3.1.jar:1.3.1]
	... 65 common frames omitted

I am lost here, have no idea what the error message want to mean. Can someone help me with this? Dependency tracking would be essential on the edge server.

@dhaval24
Copy link
Contributor

dhaval24 commented Jun 1, 2018

@matyee thank you for your comment on AI. Few questions:

  1. Are you using Application Insights SpringBoot starter (released recently) for your springboot apps?
  2. Which specific dependencies are you interested to collect in edge server. The agent currently supports auto collection of outbound HTTP via apache http client, MsSQL, MySQL, OracleDB, Redis cache.

Further, it looks to me that Zuul filter is throwing some exception when they are passing filter chain. It would be much easier to investigate this, if you can provide a small repro of the application which we can work on or atleast provide steps on how we can reproduce this issue.

Please specify springboot version, netflix zuul version and any other dependencies you have in your setup.

@matyee
Copy link
Author

matyee commented Jun 1, 2018

@dhaval24 thank you for your reply. Sorry I was not infromative enough. I was in a rush.

I use Zuul as the edge service of a microservices architecture running on App Services. The microservices all offer their REST APIs. Every client requests hit the edge service first. I have routes defined in Zuul to route the traffic to the appripriate downstream microservice. Zuul uses Apache http and I would like to see the telemetry of those subsequent calls.

I have created a small repository. https://github.com/matyee/zuultest .

I defined a single route only. The /time/** endpoint calls all go to http://worldclockapi.com/api/json/est time api. Calling the /time/now endpoint of the edge service makes a subsequent call to http://worldclockapi.com/api/json/est/now and returns the result.

I included Application Insights using the Spring Boots starter. I also checked in the used agent in the repository. Without the agent the service wokrs well, it also sends the telemetry to AI. If I ad the javaagent I get the error mentioned at the beginning of the thread.

@dhaval24 dhaval24 added the Spring Boot Starter Spring boot starter issues, feature asks and questions label Jun 3, 2018
@dhaval24
Copy link
Contributor

dhaval24 commented Jun 4, 2018

@matyee thank you very much for the details and creating a small application which could be used for testing. So if I assume correctly once I run this app and hit the URL: /time/now it will throw the error correct ?when agent is being used.

@dhaval24 dhaval24 added the Bug label Jun 4, 2018
@dhaval24
Copy link
Contributor

dhaval24 commented Jun 4, 2018

@matyee it seems like there is an issue in the agent instrumentation of HTTP. Basically we instrument HttpUriRequest interface for the ApacheHttpClient but it seems like the Zuul library uses BasicHttpRequest. Due to the agent instrumentation byte code is modified and then it throws this exception of IncompitableClassChange. Most probably there needs to be a fix in the agent to make this work.

As a work around you can currently disable the HTTP tracking in agent and use it to capture other dependencies like SQL etc if you have.

You can manually wrap the outgoing HTTP request from your edge service with trackDependency() calls, sure it is some work but you might want to do it until a fix is released. Thanks for bringing the issue up here.

@grlima
Copy link
Contributor

grlima commented Jun 4, 2018

@dhaval - By looking at the exception stack, it seems the instrumentation code added to the doExecute method assumes the second argument will be org.apache.http.client.methods.HttpUriRequest. That's not always the case, as seen in this example. The doExecute method's second param is guaranteed to be org.apache.http.HttpRequest, not HttpUriRequest.

To repro, you can just do something like this in your test app (taken from what Zuul is doing)

httpRequest = new BasicHttpRequest(verb, uriWithQueryString);
httpclient.execute(httpHost, httpRequest);

@tomaszglinski
Copy link

Hi @dhaval24, when will the fix be released? It blocks the AI integration into routing services...

@dhaval24
Copy link
Contributor

dhaval24 commented Jul 5, 2018

@tomaszglinski we plan to release a new version with this anytime in next two weeks. You should be unblocked with it.

@tomaszglinski
Copy link

@dhaval24 thanks for info

@ghost ghost locked as resolved and limited conversation to collaborators Jul 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Bug Spring Boot Starter Spring boot starter issues, feature asks and questions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants