Skip to content

/ graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [1.2.2] Search fails due to IndexClosedException #1533

Closed
voodoodrul opened this issue Nov 4, 2015 · 2 comments
Closed

[Bug] [1.2.2] Search fails due to IndexClosedException #1533

voodoodrul opened this issue Nov 4, 2015 · 2 comments
Assignees
@joschi
Labels
elasticsearch
Milestone
1.3.0

Comments

@voodoodrul
Copy link
Contributor

@voodoodrul voodoodrul commented Nov 4, 2015

When performing a query that crosses an index range with one or more closed indices, Graylog throws this exception. The easiest way to recreate this is to close any index and then perform a "Search in all messages" query.

2015-11-04_15:32:04.22852 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2015-11-04_15:32:04.22855 org.elasticsearch.indices.IndexClosedException: [graylog_4400] closed
2015-11-04_15:32:04.22855       at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:721)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.<init>(TransportSearchTypeAction.java:118)
2015-11-04_15:32:04.22856       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:71)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.<init>(TransportSearchQueryThenFetchAction.java:65)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62)
2015-11-04_15:32:04.22857       at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22858       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
2015-11-04_15:32:04.22859       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
2015-11-04_15:32:04.22860       at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:90)
2015-11-04_15:32:04.22860       at org.elasticsearch.client.FilterClient.execute(FilterClient.java:60)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.execute(GlobalTimeoutClient.java:53)
2015-11-04_15:32:04.22861       at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:333)
2015-11-04_15:32:04.22861       at org.graylog2.indexer.searches.Searches.search(Searches.java:252)
2015-11-04_15:32:04.22862       at org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-11-04_15:32:04.22862       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2015-11-04_15:32:04.22863       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-11-04_15:32:04.22863       at java.lang.reflect.Method.invoke(Method.java:497)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2015-11-04_15:32:04.22864       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
2015-11-04_15:32:04.22866       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2015-11-04_15:32:04.22867       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:308)
2015-11-04_15:32:04.22868       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2015-11-04_15:32:04.22869       at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2015-11-04_15:32:04.22870       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:291)
2015-11-04_15:32:04.22871       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1140)
2015-11-04_15:32:04.22871       at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
2015-11-04_15:32:04.22871       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
2015-11-04_15:32:04.22872       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
2015-11-04_15:32:04.22873       at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
2015-11-04_15:32:04.22874       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
2015-11-04_15:32:04.22874       at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
2015-11-04_15:32:04.22874       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-11-04_15:32:04.22875       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-11-04_15:32:04.22875       at java.lang.Thread.run(Thread.java:745)
joschi pushed a commit that referenced this issue Nov 9, 2015
Jochen Schalanda
Set indices options to lenient for search queries
Loading status checks…
a0c70af 
Fixes #1533
@joschi joschi mentioned this issue Nov 9, 2015
Merged
@joschi joschi self-assigned this Nov 9, 2015
@joschi joschi added this to the 1.2.3 milestone Nov 9, 2015
@joschi joschi added the elasticsearch label Nov 9, 2015
@NickMeves
Copy link

@NickMeves NickMeves commented Nov 9, 2015

I believe I noticed this as well with the update to 1.2.2 reverting back to MongoDB managed index ranges, so I'll add information here for my experience rather than opening a new issue:

I had a few reopened indices (and normally opened indices) that I closed manually prior to the index retention job by hitting a limit.

The ES indices closed and the Recalculate Range jobs triggered like normal, but the closed indices were still present in MongoDB in the index_ranges collection. When any search in that range happened, they were in the indices list to search, which triggered the index closed exception.

I had to go in manually and remove the closed indices from MongoDB's index_ranges collection to fix.

(FYI indices were created prior to 1.2.1, some reopened during 1.2.1, all closed during 1.2.2).
@bernd
Copy link
Member

@bernd bernd commented Nov 20, 2015

Fixed in #1547 This will be in Graylog 1.3.
@bernd bernd closed this Nov 20, 2015
@joschi joschi modified the milestones: 1.2.3, 1.3.0 Nov 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joschi joschi

Labels
elasticsearch
Projects
None yet
Milestone
1.3.0
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
@bernd @joschi @NickMeves @voodoodrul