[Bug] [1.2.2] Search fails due to IndexClosedException

[jskrivseth]

When performing a query that crosses an index range with one or more closed indices, Graylog throws this exception. The easiest way to recreate this is to close any index and then perform a "Search in all messages" query.

2015-11-04_15:32:04.22852 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2015-11-04_15:32:04.22855 org.elasticsearch.indices.IndexClosedException: [graylog_4400] closed
2015-11-04_15:32:04.22855       at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:721)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.<init>(TransportSearchTypeAction.java:118)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:71)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:65)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22860       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.FilterClient.execute(FilterClient.java:60)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.execute(GlobalTimeoutClient.java:53)
2015-11-04_15:32:04.22861       at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:333)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.searches.Searches.search(Searches.java:252)
2015-11-04_15:32:04.22862       at org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2015-11-04_15:32:04.22863       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-11-04_15:32:04.22863       at java.lang.reflect.Method.invoke(Method.java:497)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
2015-11-04_15:32:04.22871       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
2015-11-04_15:32:04.22871       at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
2015-11-04_15:32:04.22871       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
2015-11-04_15:32:04.22874       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
2015-11-04_15:32:04.22874       at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
2015-11-04_15:32:04.22874       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-11-04_15:32:04.22875       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-11-04_15:32:04.22875       at java.lang.Thread.run(Thread.java:745)

[NickMeves]

I believe I noticed this as well with the update to 1.2.2 reverting back to MongoDB managed index ranges, so I'll add information here for my experience rather than opening a new issue:

I had a few reopened indices (and normally opened indices) that I closed manually prior to the index retention job by hitting a limit.

The ES indices closed and the Recalculate Range jobs triggered like normal, but the closed indices were still present in MongoDB in the index_ranges collection. When any search in that range happened, they were in the indices list to search, which triggered the index closed exception.

I had to go in manually and remove the closed indices from MongoDB's index_ranges collection to fix.

(FYI indices were created prior to 1.2.1, some reopened during 1.2.1, all closed during 1.2.2).

[bernd]

Fixed in #1547 This will be in Graylog 1.3.