New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SearchPhaseExecutionException: all shards failed #2326

Closed
beezoncheg opened this Issue Jun 2, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@beezoncheg

beezoncheg commented Jun 2, 2016

After updating from 1.5 to 2.0.2 I get an error in logs. ES cluster is green.

Environment

Graylog 2.0.2 (4da1379)
elasticsearch 2.3.3
mongodb-org 3.2.6
Ubuntu 14.04

2016-06-02T12:53:09.573+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206) ~[graylog.jar:?]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152) ~[graylog.jar:?]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46) ~[graylog.jar:?]
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212) ~[graylog.jar:?]
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202) ~[graylog.jar:?]
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[graylog.jar:?]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[graylog.jar:?]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[graylog.jar:?]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[graylog.jar:?]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[graylog.jar:?]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: parse_exception: '*' or '?' not allowed as first character in WildcardQuery
    at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:747) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.MapperQueryParser.getPossiblyAnalyzedWildcardQuery(MapperQueryParser.java:659) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.MapperQueryParser.getWildcardQuerySingle(MapperQueryParser.java:644) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.MapperQueryParser.getWildcardQuery(MapperQueryParser.java:598) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParserBase.handleBareTokenQuery(QueryParserBase.java:845) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:348) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:247) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:171) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:160) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:117) ~[graylog.jar:?]
    at org.apache.lucene.queryparser.classic.MapperQueryParser.parse(MapperQueryParser.java:848) ~[graylog.jar:?]
    at org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:227) ~[graylog.jar:?]
    at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250) ~[graylog.jar:?]
    at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:76) ~[graylog.jar:?]
    at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250) ~[graylog.jar:?]
    at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:320) ~[graylog.jar:?]
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:223) ~[graylog.jar:?]
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:218) ~[graylog.jar:?]
    at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33) ~[graylog.jar:?]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:838) ~[graylog.jar:?]
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:654) ~[graylog.jar:?]
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) ~[graylog.jar:?]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) ~[graylog.jar:?]
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368) ~[graylog.jar:?]
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365) ~[graylog.jar:?]
    at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[graylog.jar:?]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[graylog.jar:?]
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300) ~[graylog.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.8.0_91]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.8.0_91]
    ... 1 more

ES has numerous errors in logs like

[2016-06-02 12:52:03,641][DEBUG][action.search            ] [log1 elastic] [graylog2_1607][0], node[4le8I2zzRf-zLdMwAPQbwQ], [R], v[5], s[STARTED], a[id=dANkn1DKQmysRG9IPJtq5g]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1e85740] lastShard [true]
RemoteTransportException[[m1log3 elastic][100.99.1.21:9300][indices:data/read/search[phase/query]]]; nested: ElasticsearchParseException[failed to parse date field [1464774741357] with format [yyyy-MM-dd HH:mm:ss.SSS]]; nested: IllegalArgumentException[Invalid format: "1464774741357" is malformed at "1357"];
Caused by: ElasticsearchParseException[failed to parse date field [1464774741357] with format [yyyy-MM-dd HH:mm:ss.SSS]]; nested: IllegalArgumentException[Invalid format: "1464774741357" is malformed at "1357"];
    at org.elasticsearch.common.joda.DateMathParser.parseDateTime(DateMathParser.java:203)
    at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:67)
    at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.parseToMilliseconds(DateFieldMapper.java:451)
    at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.innerRangeQuery(DateFieldMapper.java:435)
    at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.access$000(DateFieldMapper.java:199)
    at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType$LateParsingQuery.rewrite(DateFieldMapper.java:224)
    at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:278)
    at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:837)
    at org.elasticsearch.search.internal.ContextIndexSearcher.rewrite(ContextIndexSearcher.java:81)
    at org.elasticsearch.search.internal.DefaultSearchContext.preProcess(DefaultSearchContext.java:231)
    at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:103)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:676)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
    at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid format: "1464774741357" is malformed at "1357"
    at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
    at org.elasticsearch.common.joda.DateMathParser.parseDateTime(DateMathParser.java:200)
    ... 22 more

If it's a problem with improper timestamp in incomming messages how to see the full message to understand where it came from.

@joschi

This comment has been minimized.

Show comment
Hide comment
@joschi

joschi Jun 2, 2016

Contributor

@beezoncheg

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: parse_exception: '*' or '?' not allowed as first character in WildcardQuery

Check the allow_leading_wildcard_searches setting in your Graylog configuration file and set it accordingly.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!

Contributor

joschi commented Jun 2, 2016

@beezoncheg

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: parse_exception: '*' or '?' not allowed as first character in WildcardQuery

Check the allow_leading_wildcard_searches setting in your Graylog configuration file and set it accordingly.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!

@beezoncheg

This comment has been minimized.

Show comment
Hide comment
@beezoncheg

beezoncheg Jun 2, 2016

I'm sorry for that. Next time I'll post in proper place.

beezoncheg commented Jun 2, 2016

I'm sorry for that. Next time I'll post in proper place.

@beezoncheg

This comment has been minimized.

Show comment
Hide comment
@beezoncheg

beezoncheg Jun 2, 2016

BTW, seems that ES errors are not related to graylog.

beezoncheg commented Jun 2, 2016

BTW, seems that ES errors are not related to graylog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment