New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [1.2.2] Search fails due to IndexClosedException #1533

Closed
voodoodrul opened this Issue Nov 4, 2015 · 2 comments

Comments

Projects
None yet
4 participants
@voodoodrul
Contributor

voodoodrul commented Nov 4, 2015

When performing a query that crosses an index range with one or more closed indices, Graylog throws this exception. The easiest way to recreate this is to close any index and then perform a "Search in all messages" query.

2015-11-04_15:32:04.22852 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2015-11-04_15:32:04.22855 org.elasticsearch.indices.IndexClosedException: [graylog_4400] closed
2015-11-04_15:32:04.22855       at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:721)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.<init>(TransportSearchTypeAction.java:118)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:71)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:65)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22860       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.FilterClient.execute(FilterClient.java:60)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.execute(GlobalTimeoutClient.java:53)
2015-11-04_15:32:04.22861       at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:333)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.searches.Searches.search(Searches.java:252)
2015-11-04_15:32:04.22862       at org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2015-11-04_15:32:04.22863       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-11-04_15:32:04.22863       at java.lang.reflect.Method.invoke(Method.java:497)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
2015-11-04_15:32:04.22871       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
2015-11-04_15:32:04.22871       at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
2015-11-04_15:32:04.22871       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
2015-11-04_15:32:04.22874       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
2015-11-04_15:32:04.22874       at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
2015-11-04_15:32:04.22874       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-11-04_15:32:04.22875       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-11-04_15:32:04.22875       at java.lang.Thread.run(Thread.java:745)

joschi added a commit that referenced this issue Nov 9, 2015

@joschi joschi self-assigned this Nov 9, 2015

@joschi joschi added this to the 1.2.3 milestone Nov 9, 2015

@joschi joschi added the elasticsearch label Nov 9, 2015

@NickMeves

This comment has been minimized.

NickMeves commented Nov 9, 2015

I believe I noticed this as well with the update to 1.2.2 reverting back to MongoDB managed index ranges, so I'll add information here for my experience rather than opening a new issue:

I had a few reopened indices (and normally opened indices) that I closed manually prior to the index retention job by hitting a limit.

The ES indices closed and the Recalculate Range jobs triggered like normal, but the closed indices were still present in MongoDB in the index_ranges collection. When any search in that range happened, they were in the indices list to search, which triggered the index closed exception.

I had to go in manually and remove the closed indices from MongoDB's index_ranges collection to fix.

(FYI indices were created prior to 1.2.1, some reopened during 1.2.1, all closed during 1.2.2).

@bernd

This comment has been minimized.

Member

bernd commented Nov 20, 2015

Fixed in #1547 This will be in Graylog 1.3.

@bernd bernd closed this Nov 20, 2015

@joschi joschi modified the milestones: 1.2.3, 1.3.0 Nov 23, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment