Getting i_o_exception BigInteger on documents creation/updation #44150

larrycinnabar · 2019-07-10T06:46:33Z

Elasticsearch 7.2, java 1.8

I'm consistently getting this message in elastic logs:

[2019-07-09T14:15:18,836][WARN ][o.e.t.OutboundHandler    ] [banana] send message failed [channel: Netty4TcpChannel{localAddress=/***:55054, remoteAddress=****/****:9300}]
java.io.IOException: can not write type [class java.math.BigInteger]
	at org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.io.stream.StreamOutput.lambda$static$11(StreamOutput.java:658) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:756) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.io.stream.StreamOutput.lambda$static$11(StreamOutput.java:658) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:756) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeMap(StreamOutput.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.script.Script.writeTo(Script.java:470) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.update.UpdateRequest.writeTo(UpdateRequest.java:891) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:248) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:80) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:169) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:155) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:202) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:132) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:127) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:91) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:246) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:643) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:137) [x-pack-security-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:553) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:541) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:749) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performRemoteAction(TransportReplicationAction.java:723) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:689) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:165) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:435) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:548) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:253) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:93) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:472) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:100) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:113) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.2.0.jar:7.2.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:337) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-7.2.0.jar:7.2.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.2.0.jar:7.2.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.2.0.jar:7.2.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]

no more debug information on what exact data it happened

On the client side, I'm sending bulk request with dozens (sometimes hundreds) index/update requests, and some of them are returning with such an error:
{"type":"i_o_exception","reason":"can not write type [class java.math.BigInteger]"}
It's the only error information that is returned.

By the index of the errored response I could find the document that caused this error. Document has no big numbers at all.

It should be something internal? I've tried to send extremely big number and the error message looks like a normal validation exception - showing the problem field, etc.
So, this case is other issue.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-07-10T08:38:47Z

Pinging @elastic/es-search

iverase · 2019-07-10T08:40:17Z

@larrycinnabar Could you provide the stack trace? That will give us some information where the error is being thrown.

larrycinnabar · 2019-07-10T08:57:05Z

@iverase I've edited the issue, provided the stack trace

iverase · 2019-07-10T09:09:34Z

What looks suspicious is the following:

at org.elasticsearch.script.Script.writeTo(Script.java:470) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.update.UpdateRequest.writeTo(UpdateRequest.java:891) ~[elasticsearch-7.2.0.jar:7.2.0]

Are you using an update command with a script? I think that is the issue, the script is returning a java.math.BigInteger object and ES does not know how to handle it.

iverase · 2019-07-10T13:13:04Z

@larrycinnabar could you provide more information about how you are performing those updates?

larrycinnabar · 2019-07-10T13:30:13Z

@iverase,
Yeap, we have upsert script that looks like this (it's not the only script, but they all are in this put-or-append manner)

if (!ctx._source.containsKey('timings') || ctx._source.timings.length==0) {
	ctx._source.timings = params['timings'];
} else {
	if (params.containsKey('timings') && params['timings'] != null) {
		for (newTiming in params['timings']) {
			boolean contains = false;
			for (oldTiming in ctx._source.timings) {
				if (oldTiming.key == newTiming.key) {
					contains = true;
					break;
				}
			}
	
			if (!contains) {
				ctx._source.timings.add(newTiming);
			}
		}
	}
}

elasticmachine · 2019-07-10T13:42:59Z

Pinging @elastic/es-core-infra

iverase · 2019-07-10T13:43:22Z

It seems related to #34656.

larrycinnabar · 2019-07-11T10:20:18Z

@iverase,

if I send the same document again - it might be upserted successfully.

Numbers in there are not so big, example: 1562839304884 - it's a millisecond timestamp (we store them in documents, and in mappings they are long).

As size of BigInt is related to available memory, can this be a memory issue? (but I doubt)

stu-elastic · 2020-05-14T20:45:13Z

This looks like we're deserializing the large value in params as a BigInt, which we can't stream. Perhaps we should deserialize it as a long instead if the value will fit in long?

rjernst · 2020-07-21T18:12:16Z

The solution here is to add BigInteger support to StreamInput/StreamOutput. We discussed that last week within core/infra, and agreed it is worthwhile to add. The change should be relatively simple, as BigInteger supports reading/writing as an array of bytes, but it is not a priority, so I am marking this issue as adoptme.

williamrandolph · 2021-01-06T19:01:50Z

I'm reviewing this for a Core/Infra team issue triage. It still looks like a good "help wanted" issue so I am going to leave the issue as is.

Kiriakos1998 · 2023-07-28T16:51:17Z

Hello @williamrandolph, if this issue is still open I would like to work on it.

iverase added the :Search/Search Search-related issues that do not fall into other categories label Jul 10, 2019

iverase added the :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache label Jul 10, 2019

iverase removed the :Search/Search Search-related issues that do not fall into other categories label Jul 10, 2019

rjernst added the Team:Core/Infra Meta label for core/infra team label May 4, 2020

stu-elastic removed the :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache label May 14, 2020

elasticmachine removed the Team:Core/Infra Meta label for core/infra team label May 14, 2020

stu-elastic added :Core/Infra/Core Core issues without another label team-discuss labels May 14, 2020

elasticmachine added the Team:Core/Infra Meta label for core/infra team label May 14, 2020

rjernst added help wanted adoptme and removed team-discuss labels Jul 21, 2020

rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020

williamrandolph removed the needs:triage Requires assignment of a team area label label Jan 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting i_o_exception BigInteger on documents creation/updation #44150

Getting i_o_exception BigInteger on documents creation/updation #44150

larrycinnabar commented Jul 10, 2019 •

edited

elasticmachine commented Jul 10, 2019

iverase commented Jul 10, 2019

larrycinnabar commented Jul 10, 2019

iverase commented Jul 10, 2019 •

edited

iverase commented Jul 10, 2019

larrycinnabar commented Jul 10, 2019

elasticmachine commented Jul 10, 2019

iverase commented Jul 10, 2019

larrycinnabar commented Jul 11, 2019 •

edited

stu-elastic commented May 14, 2020

rjernst commented Jul 21, 2020

williamrandolph commented Jan 6, 2021

Kiriakos1998 commented Jul 28, 2023

Getting i_o_exception BigInteger on documents creation/updation #44150

Getting i_o_exception BigInteger on documents creation/updation #44150

Comments

larrycinnabar commented Jul 10, 2019 • edited

elasticmachine commented Jul 10, 2019

iverase commented Jul 10, 2019

larrycinnabar commented Jul 10, 2019

iverase commented Jul 10, 2019 • edited

iverase commented Jul 10, 2019

larrycinnabar commented Jul 10, 2019

elasticmachine commented Jul 10, 2019

iverase commented Jul 10, 2019

larrycinnabar commented Jul 11, 2019 • edited

stu-elastic commented May 14, 2020

rjernst commented Jul 21, 2020

williamrandolph commented Jan 6, 2021

Kiriakos1998 commented Jul 28, 2023

larrycinnabar commented Jul 10, 2019 •

edited

iverase commented Jul 10, 2019 •

edited

larrycinnabar commented Jul 11, 2019 •

edited