Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.lang.IllegalStateException: parser for [_doc] did not end on END_OBJECT on rollups #36726

Closed
bhavyarm opened this issue Dec 17, 2018 · 4 comments
Closed

java.lang.IllegalStateException: parser for [_doc] did not end on END_OBJECT on rollups #36726

bhavyarm opened this issue Dec 17, 2018 · 4 comments
Labels
>bug :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data

Comments

@bhavyarm
Copy link

Elasticsearch version (bin/elasticsearch --version): alpha2

Description of the problem including expected versus actual behavior:

Steps to reproduce: ES logs display this error stack on rollups in alpha2
and on Kibana logs: elastic/kibana#27314

[2018-12-14T09:40:16,899][WARN ][r.suppressed             ] [bhavyarajumandya] path: /_all/_xpack/rollup/data, params: {index=_all}
java.lang.IllegalStateException: parser for [_doc] did not end on END_OBJECT
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:336) ~[elasticsearch-x-content-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:168) ~[elasticsearch-x-content-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.rollup.action.RollupIndexCaps.parseMetadataXContent(RollupIndexCaps.java:133) ~[?:?]
	at org.elasticsearch.xpack.rollup.action.TransportGetRollupCapsAction.findRollupIndexCaps(TransportGetRollupCapsAction.java:104) ~[?:?]
	at org.elasticsearch.xpack.rollup.action.TransportGetRollupIndexCapsAction.lambda$getCapsByRollupIndex$3(TransportGetRollupIndexCapsAction.java:64) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:430) ~[?:?]
	at org.elasticsearch.xpack.rollup.action.TransportGetRollupIndexCapsAction.getCapsByRollupIndex(TransportGetRollupIndexCapsAction.java:62) ~[?:?]
	at org.elasticsearch.xpack.rollup.action.TransportGetRollupIndexCapsAction.doExecute(TransportGetRollupIndexCapsAction.java:51) ~[?:?]
	at org.elasticsearch.xpack.rollup.action.TransportGetRollupIndexCapsAction.doExecute(TransportGetRollupIndexCapsAction.java:31) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:126) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.rollup.rest.RestGetRollupIndexCapsAction.lambda$prepareRequest$0(RestGetRollupIndexCapsAction.java:45) [x-pack-rollup-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.rest.DeprecationRestHandler.handleRequest(DeprecationRestHandler.java:62) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:337) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:328) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:378) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:307) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:844) [?:?]

cc @polyfractal
@bhavyarm bhavyarm added the >bug label Dec 17, 2018
@polyfractal polyfractal added the :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data label Dec 17, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo

@jen-huang jen-huang mentioned this issue Dec 17, 2018
Closed
@polyfractal
Copy link
Contributor

Hmm, I can't reproduce this. Tried with an entirely empty node, and with the various kibana sample data installed. Just see empty responses:

image

Is there anything special that needs to be done to recreate it? What indices are in the cluster?

@walterra
Copy link

I was able to reproduce this issue in the following way:

PUT doc-issue-test
{
  "mappings": {
    "_doc": {
      "_meta": {
        "created_by": "omg"
      },
      "properties": {
        "somekeyword": {
          "type": "keyword"
        }
      }
    }
  }
}

Then GET doc-issue/_xpack/rollup/data results in:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "parser for [_doc] did not end on END_OBJECT"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "parser for [_doc] did not end on END_OBJECT"
  },
  "status": 500
}

If I recreate the same index without the _meta field the error doesn't occur.

@polyfractal
Copy link
Contributor

polyfractal commented Dec 18, 2018 •
edited

Thanks @walterra, that let me find the issue. Rollup's parser half-consumes the JSON object but doesn't finish because it is a non-rollup meta field. But the parsing framework throws an exception when the full JSON isn't consumed, which is causing the issue.

Will work on a fix.

@polyfractal polyfractal mentioned this issue Dec 18, 2018
Merged
@polyfractal polyfractal mentioned this issue Dec 18, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@walterra @polyfractal @bhavyarm @elasticmachine