java.lang.IllegalStateException: parser for [params] did not end on END_OBJECT for RestEsqlAsyncQueryAction

[pmpailis]

Elasticsearch Version

current

Installed Plugins

No response

Java Version

bundled

OS Version

N/A

Problem Description

Getting the following exception in logs, but we don't have the exact query to reproduce this. Creating this issue as a placeholder to investigate and link if it reoccurs.

java.lang.IllegalStateException: parser for [params] did not end on END_OBJECT
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.ObjectParser.throwMustEndOn(ObjectParser.java:672)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:643)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:316)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:260)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.ObjectParser.apply(ObjectParser.java:386)
	at org.elasticsearch.xpack.esql.action.RequestXContent.parseAsync(RequestXContent.java:99)
	at org.elasticsearch.xpack.esql.action.RestEsqlAsyncQueryAction.prepareRequest(RestEsqlAsyncQueryAction.java:49)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:99)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:486)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:480)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:105)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:90)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$2(SecurityRestFilter.java:80)
	at org.elasticsearch.security@9.2.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:96)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:480)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.RestController.lambda$maybeAggregateAndDispatchRequest$4(RestController.java:404)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.rest.RestContentAggregator$AggregationChunkHandler.onNext(RestContentAggregator.java:78)
	at org.elasticsearch.transport.netty4@9.2.0/org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream.handleNettyContent(Netty4HttpRequestBodyStream.java:87)
	at org.elasticsearch.transport.netty4@9.2.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:146)
	at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler@4.1.126.Final/io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:202)
	at io.netty.handler@4.1.126.Final/io.netty.handler.flow.FlowControlHandler.read(FlowControlHandler.java:139)
	at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:847)
	at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:824)
	at io.netty.common@4.1.126.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.common@4.1.126.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.common@4.1.126.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.transport@4.1.126.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.common@4.1.126.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.common@4.1.126.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1447)

Steps to Reproduce

N/A

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[inqueue]

The error hints there is a parsing issue with the params field of some request. Logging indicates there was a correlating REST API call to POST /_query/async at the time of the parsing error. This ES|QL API supports the params field. The error can be reproduced with a malformed value for params:

curl -k -H "Authorization: ApiKey $APIKEY" -H "Content-Type: application/json" $ENDPOINT/_query/async -XPOST -d'{"query": "FROM logs-*", "params": {"}'

The error is returned to the client and logged as a WARN in the ES log:

{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"parser for [params] did not end on END_OBJECT"}],"type":"illegal_state_exception","reason":"parser for [params] did not end on END_OBJECT"},"status":500}

[pmpailis]

Looking at the code, we expect params to always be an array so maybe we could just throw a parsing exception if we're met with anything different (like a START_OBJECT in this case)?

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

Lines 144 to 150 in f6547d6

	private static QueryParams parseParams(XContentParser p) throws IOException {
	List<QueryParam> namedParams = new ArrayList<>();
	List<QueryParam> unNamedParams = new ArrayList<>();
	List<XContentParseException> errors = new ArrayList<>();
	XContentParser.Token token = p.currentToken();
	if (token == XContentParser.Token.START_ARRAY) {