Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [1.2.2] Search fails due to IndexClosedException #1533

voodoodrul opened this issue Nov 4, 2015 · 2 comments

[Bug] [1.2.2] Search fails due to IndexClosedException #1533

voodoodrul opened this issue Nov 4, 2015 · 2 comments


Copy link

@voodoodrul voodoodrul commented Nov 4, 2015

When performing a query that crosses an index range with one or more closed indices, Graylog throws this exception. The easiest way to recreate this is to close any index and then perform a "Search in all messages" query.

2015-11-04_15:32:04.22852 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2015-11-04_15:32:04.22855 org.elasticsearch.indices.IndexClosedException: [graylog_4400] closed
2015-11-04_15:32:04.22855       at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(
2015-11-04_15:32:04.22856       at$BaseAsyncAction.<init>(
2015-11-04_15:32:04.22856       at$AsyncAction.<init>(
2015-11-04_15:32:04.22857       at$AsyncAction.<init>(
2015-11-04_15:32:04.22857       at
2015-11-04_15:32:04.22857       at
2015-11-04_15:32:04.22858       at
2015-11-04_15:32:04.22858       at
2015-11-04_15:32:04.22859       at
2015-11-04_15:32:04.22859       at
2015-11-04_15:32:04.22860       at
2015-11-04_15:32:04.22860       at org.elasticsearch.client.node.NodeClient.execute(
2015-11-04_15:32:04.22860       at org.elasticsearch.client.FilterClient.execute(
2015-11-04_15:32:04.22861       at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.execute(
2015-11-04_15:32:04.22861       at
2015-11-04_15:32:04.22861       at
2015-11-04_15:32:04.22862       at
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke(
2015-11-04_15:32:04.22863       at sun.reflect.DelegatingMethodAccessorImpl.invoke(
2015-11-04_15:32:04.22863       at java.lang.reflect.Method.invoke(
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(
2015-11-04_15:32:04.22868       at org.glassfish.jersey.server.ServerRuntime$
2015-11-04_15:32:04.22868       at org.glassfish.jersey.internal.Errors$
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors$
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(
2015-11-04_15:32:04.22870       at org.glassfish.jersey.internal.Errors.process(
2015-11-04_15:32:04.22870       at org.glassfish.jersey.process.internal.RequestScope.runInScope(
2015-11-04_15:32:04.22870       at org.glassfish.jersey.server.ServerRuntime.process(
2015-11-04_15:32:04.22871       at org.glassfish.jersey.server.ApplicationHandler.handle(
2015-11-04_15:32:04.22871       at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(
2015-11-04_15:32:04.22871       at
2015-11-04_15:32:04.22872       at
2015-11-04_15:32:04.22872       at$DefaultChannelHandlerContext.sendUpstream(
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(
2015-11-04_15:32:04.22873       at
2015-11-04_15:32:04.22874       at com.codahale.metrics.InstrumentedExecutorService$
2015-11-04_15:32:04.22874       at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$
2015-11-04_15:32:04.22874       at java.util.concurrent.ThreadPoolExecutor.runWorker(
2015-11-04_15:32:04.22875       at java.util.concurrent.ThreadPoolExecutor$
2015-11-04_15:32:04.22875       at
joschi pushed a commit that referenced this issue Nov 9, 2015
@joschi joschi self-assigned this Nov 9, 2015
@joschi joschi added this to the 1.2.3 milestone Nov 9, 2015
@joschi joschi added the elasticsearch label Nov 9, 2015
Copy link

@NickMeves NickMeves commented Nov 9, 2015

I believe I noticed this as well with the update to 1.2.2 reverting back to MongoDB managed index ranges, so I'll add information here for my experience rather than opening a new issue:

I had a few reopened indices (and normally opened indices) that I closed manually prior to the index retention job by hitting a limit.

The ES indices closed and the Recalculate Range jobs triggered like normal, but the closed indices were still present in MongoDB in the index_ranges collection. When any search in that range happened, they were in the indices list to search, which triggered the index closed exception.

I had to go in manually and remove the closed indices from MongoDB's index_ranges collection to fix.

(FYI indices were created prior to 1.2.1, some reopened during 1.2.1, all closed during 1.2.2).

Copy link

@bernd bernd commented Nov 20, 2015

Fixed in #1547 This will be in Graylog 1.3.

@bernd bernd closed this Nov 20, 2015
@joschi joschi modified the milestones: 1.2.3, 1.3.0 Nov 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants