Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

intermittent 500 errors #354

Closed
mjhaller opened this issue Oct 31, 2018 · 1 comment
Closed

intermittent 500 errors #354

mjhaller opened this issue Oct 31, 2018 · 1 comment

Comments

@mjhaller
Copy link

We are intermittently (0.03% of all requests) seeing these in production (... are redacted bits of our logs):

2018-10-31 13:11:30,341 ERROR [qtp33708465-46] c.i.w.a.AssignmentsResource [?:?] getAssignment failed for applicationName=..., experimentLabel=..., userID=..., context=PROD, createAssignment=true, ignoreSamplingPercent=false, headers=com.sun.jersey.spi.container.ContainerRequest@2ebdc003 with error:
com.intuit.wasabi.repository.RepositoryException: Error while getExperimentsForApps
        at com.intuit.wasabi.repository.cassandra.impl.CassandraExperimentRepository.getExperimentsForApps(CassandraExperimentRepository.java:303) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.assignment.cache.impl.AssignmentsMetadataCacheImpl.getExperimentsByAppName(AssignmentsMetadataCacheImpl.java:177) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.assignment.impl.AssignmentsImpl.populateAssignmentsMetadata(AssignmentsImpl.java:753) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.assignment.impl.AssignmentsImpl.doAssignments(AssignmentsImpl.java:467) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.assignment.impl.AssignmentsImpl.doSingleAssignment(AssignmentsImpl.java:273) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.api.AssignmentsResource.getAssignment(AssignmentsResource.java:172) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.api.AssignmentsResource.getAssignment(AssignmentsResource.java:155) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.autumn.web.WebFilter.doFilter(WebFilter.java:65) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.Server.handle(Server.java:518) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.NullPointerException: null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.repository.cassandra.impl.ExperimentHelper.makeExperiment(ExperimentHelper.java:60) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at com.intuit.wasabi.repository.cassandra.impl.CassandraExperimentRepository.lambda$getExperimentsForApps$1(CassandraExperimentRepository.java:296) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[na:1.8.0_181]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_181]
        at com.intuit.wasabi.repository.cassandra.impl.CassandraExperimentRepository.getExperimentsForApps(CassandraExperimentRepository.java:295) ~[wasabi-main-1.0.20171128101048-SNAPSHOT-build-all.jar:1.0.20171128101048-SNAPSHOT]
        ... 57 common frames omitted

It seems clear that the experimentPojo.getCreated() is resulting in a null. This pojo seems to be coming from experimentAccessor.asyncGetExperimentByAppName(appName.toString()) call. And while i see this data in Mysql there is no way creation_time is null as is it not nullable. (of coruse, i verified that) I don't understand enough about the architecture and the role of caching and cassandra here, so not really sure what could lead to this.

I was hoping someone could give me some clues on where to look for this data and how to clear it.

of course, we're on a very old version, and we plan to upgrade, but if it is a known bug, I'd love to understand that as go do our maintenance.
@mjhaller
Copy link
Author

I believe this was due to a failing cassandra node, and we were regularly getting inconsistent reads on the experiment definitions which was poisoning the cache. Once we removed that from our ring these went away

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mjhaller