Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature kb 390 better exception logging for dispatch with exception handling #406

Conversation

holkra
Copy link
Contributor

@holkra holkra commented Oct 9, 2015

No description provided.

@holkra
Copy link
Contributor Author

holkra commented Oct 9, 2015

Example stack trace for a timeout of the Billingrecord Plugin

2015-10-09 17:43:06,605 [qtp1169015699-27] ERROR o.k.b.payment.core.ProcessorBase - TimeoutException during the execution of plugin killbill-billingrecord with requestId de1a26d6-482d-4095-b691-7d44a1b87764
java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at org.killbill.billing.payment.dispatcher.PluginDispatcher.dispatchWithTimeout(PluginDispatcher.java:59)
at org.killbill.billing.payment.dispatcher.PluginDispatcher.dispatchWithTimeout(PluginDispatcher.java:47)
at org.killbill.billing.payment.core.ProcessorBase.dispatchWithExceptionHandling(ProcessorBase.java:232)
at org.killbill.billing.payment.core.PaymentMethodProcessor.addPaymentMethod(PaymentMethodProcessor.java:106)
at org.killbill.billing.payment.api.DefaultPaymentApi.addPaymentMethod(DefaultPaymentApi.java:427)
at org.killbill.billing.payment.api.DefaultPaymentApi$$EnhancerByGuice$$5711589e.CGLIB$addPaymentMethod$25()
at org.killbill.billing.payment.api.DefaultPaymentApi$$EnhancerByGuice$$5711589e$$FastClassByGuice$$eb14824c.invoke()
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at org.killbill.billing.util.glue.KillbillApiAopModule$ProfilingMethodInterceptor$1.execute(KillbillApiAopModule.java:52)
at org.killbill.commons.profiling.Profiling.executeWithProfiling(Profiling.java:33)
at org.killbill.billing.util.glue.KillbillApiAopModule$ProfilingMethodInterceptor.invoke(KillbillApiAopModule.java:49)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at org.killbill.billing.payment.api.DefaultPaymentApi$$EnhancerByGuice$$5711589e.addPaymentMethod()
at org.killbill.billing.jaxrs.resources.ComboPaymentResource.getOrCreatePaymentMethod(ComboPaymentResource.java:116)
at org.killbill.billing.jaxrs.resources.PaymentResource.createComboPayment(PaymentResource.java:630)
at org.killbill.billing.jaxrs.resources.PaymentResource$$EnhancerByGuice$$58a79195.CGLIB$createComboPayment$15()
at org.killbill.billing.jaxrs.resources.PaymentResource$$EnhancerByGuice$$58a79195$$FastClassByGuice$$12f8c622.invoke()
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at org.killbill.billing.jaxrs.resources.PaymentResource$$EnhancerByGuice$$58a79195.createComboPayment()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)

@holkra
Copy link
Contributor Author

holkra commented Oct 9, 2015

Example logging for the routing plugin:

2015-10-09 18:04:22,808 [qtp1169015699-35] ERROR o.k.b.p.c.sm.OperationCallbackBase - TimeoutException while executing the plugin(s) killbill-psp-routing, killbill-psp-preprocessor with requestId e0be2cbc-52f4-456b-ac72-bc97fa495086
2015-10-09 18:04:23,036 [qtp1169015699-35] WARN o.k.b.p.c.s.c.OperationControlCallback - RetryOperationCallback call TIMEOUT for account 911f3f07-ffa7-4dd1-b44b-1e2fea19eeb8
2015-10-09 18:04:26,727 [qtp1169015699-35] WARN o.k.b.j.mappers.ExceptionMapperBase - Internal error
org.killbill.billing.payment.api.PaymentApiException: Internal payment error : java.util.concurrent.TimeoutException
at org.killbill.billing.payment.core.sm.PluginControlPaymentAutomatonRunner.run(PluginControlPaymentAutomatonRunner.java:130)
at org.killbill.billing.payment.core.sm.PluginControlPaymentAutomatonRunner.run(PluginControlPaymentAutomatonRunner.java:102)
at org.killbill.billing.payment.core.PluginControlPaymentProcessor.createAuthorization(PluginControlPaymentProcessor.java:83)
at org.killbill.billing.payment.api.DefaultPaymentApi.createAuthorizationWithPaymentControl(DefaultPaymentApi.java:109)
at org.killbill.billing.util.glue.KillbillApiAopModule$ProfilingMethodInterceptor$1.execute(KillbillApiAopModule.java:52)
at org.killbill.commons.profiling.Profiling.executeWithProfiling(Profiling.java:33)
at org.killbill.billing.util.glue.KillbillApiAopModule$ProfilingMethodInterceptor.invoke(KillbillApiAopModule.java:49)
at org.killbill.billing.util.security.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:45)
at org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
at org.killbill.billing.util.security.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:32)
at org.killbill.billing.jaxrs.resources.PaymentResource.createComboPayment(PaymentResource.java:643)
at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at org.killbill.billing.server.security.TenantFilter.doFilter(TenantFilter.java:110)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at org.killbill.billing.server.filters.ResponseCorsFilter.doFilter(ResponseCorsFilter.java:66)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:44)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.guice.web.SimpleFilterChain.doFilter(SimpleFilterChain.java:41)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:104)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at org.killbill.billing.payment.dispatcher.PluginDispatcher.dispatchWithTimeout(PluginDispatcher.java:59)
at org.killbill.billing.payment.dispatcher.PluginDispatcher.dispatchWithTimeout(PluginDispatcher.java:47)
at org.killbill.billing.payment.core.sm.OperationCallbackBase.dispatchWithAccountLockAndTimeout(OperationCallbackBase.java:81)
at org.killbill.billing.payment.core.sm.control.OperationControlCallback.doOperationCallback(OperationControlCallback.java:95)
at org.killbill.automaton.DefaultOperation.run(DefaultOperation.java:51)
at org.killbill.automaton.DefaultState.runOperation(DefaultState.java:80)
at org.killbill.billing.payment.core.sm.PluginControlPaymentAutomatonRunner.run(PluginControlPaymentAutomatonRunner.java:121)
... 78 common frames omitted

} else {
requestId = "NotAvailableRequestId";
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use a one-liner here:
final String requestId = Request.getPerThreadRequestData() != null ? Request.getPerThreadRequestData().getRequestId() : "NotAvailableRequestId";

Also i am confused with the following:

  1. Why don't we pass null (@nullable) and do the mapping to NotAvailableRequestId at the time we need it?
  2. Why do we need to get the requestId in this method if we don't use it here? Since this is a ThreadLocal we should extract the value where it is needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. Therefore I moved the retrieval of the requestId to the places to the method where it is actually used and removed it here.

@sbrossie
Copy link
Member

sbrossie commented Oct 9, 2015

Would be nice to see a test case where this exception is generated and verify we see both the pluginName and the requestId.

@holkra
Copy link
Contributor Author

holkra commented Oct 12, 2015

Hi Stephane, I pushed the changes I did regarding your comments. Tomorrow I will write an test for this feature and I will add the request id to the general logging.

@holkra
Copy link
Contributor Author

holkra commented Oct 13, 2015

Hi Stephane, at the moment I am writing a test to check the expected log messages in the case of an timeout. You asked me to check for the exception, but most of the time I haven't changed the exception in any way, I have just added the logging. I seems that reading the logging from test is not so straightforward especially if you don't want to change the logging framework. We are using slf4j-simple for that in tests. The question is in which direction should we go. Should I try to create a test that checks the logging or should I completely change the way a report the error and put everything into the exception. But that would also not so easy, because unwrapExceptionFromDispatchedTask for example just get its information about the exception from the PaymentStateContext and I am not sure if we want to change that. What do you think?

@holkra
Copy link
Contributor Author

holkra commented Oct 14, 2015

Just to give an update to yesterday's comment. I decided to create some testing support for logging messages. I wrote a first test that tests the logging for creating payment. Tomorrow I will have a look how I can reach the code paths that aren't tested yet.

@@ -79,6 +80,13 @@ protected void logAPICall(final String transactionType, final Account account, f
logLine.append(", currency = ")
.append(currency);
}

final String requestId = Request.getPerThreadRequestData() != null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's actually get rid of all requestId logging (here and below). I've opened #409 for tracking (we should really use the user token instead, which follows bus events and notifications).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if I understand it correctly. How is the userToken normally filled? Is that something that happens completely on behalf of the client? So, Groupon could for example decide to put the request id into userToken? I am just wondering how we enable them to analyze cross application requests in the future, because I convinced that they don't want to lose this ability.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the killbill-api level, the user token is created by the client and passed in the call context. At the JAX-RS level, it is automatically generated for the client.

#409 is about setting the user token to the request id, if specified.

for (final String pluginName : paymentControlPluginNames) {
final PaymentControlPluginApi plugin = paymentControlPluginRegistry.getServiceForName(pluginName);
if (plugin != null) {
try {
log.info("Calling onSuccessCall of plugin {} for requestId {}", pluginName, requestId);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debug instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@pierre
Copy link
Member

pierre commented Oct 14, 2015

Could you also squash and rebase? It looks like there are merge conflicts.

@holkra holkra force-pushed the feature_KB-390-better-exception-logging-for-dispatchWithExceptionHandling branch from cd2a2fc to 2a0064c Compare October 16, 2015 09:15
@pierre
Copy link
Member

pierre commented Oct 16, 2015

Thanks! I've fixed some trivial style issues and squashed the commits to simplify the history.

@pierre pierre closed this Oct 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants