Error with large document

[juliozambonin]

elasticsearch-net/src/Nest/Utf8Json/JsonWriter.cs

Line 286 in d367c03

var max = StringEncoding.UTF8.GetMaxByteCount(value.Length) + 2;

Hello. I'm using the Ingest Attachment plugin to index documents. Indexing is failing with large documents, such as a 750mb txt.
In this line mentioned, the value.length is greater than the type int can support, causing an exception: "The resulting number of bytes is greater than the maximum number that can be returned as an integer."

DebugInformation:

FailureReason: Unrecoverable/Unexpected BadRequest while attempting PUT on http://localhost:9200/busca/_doc/10213?pipeline=attachments

• [1] BadRequest: Node: http://localhost:9200/ Exception: ArgumentOutOfRangeException Took: 00:00:00.1489992

Audit exception in step 1 BadRequest:

System.ArgumentOutOfRangeException: Número excessivo de caracteres. O número de bytes resultante é maior do que pode ser retornado como int.
Nome do parâmetro: charCount
em System.Text.UTF8Encoding.GetMaxByteCount(Int32 charCount)
em Elasticsearch.Net.Utf8Json.JsonWriter.WriteString(String value)
em Serialize(Byte[][] , Object[] , JsonWriter& , DocumentoBuscaInfo , IJsonFormatterResolver )
em Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) em Nest.ProxyRequestFormatterBase2.Serialize(JsonWriter& writer, TRequestInterface value, IJsonFormatterResolver formatterResolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting)
em Elasticsearch.Net.SerializableData1.Write(Stream writableStream, IConnectionConfigurationValues settings) em Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) em Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) em Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

Inner Exception: Número excessivo de caracteres. O número de bytes resultante é maior do que pode ser retornado como int.

Nome do parâmetro: charCount
System.ArgumentOutOfRangeException: Número excessivo de caracteres. O número de bytes resultante é maior do que pode ser retornado como int.
Nome do parâmetro: charCount
em System.Text.UTF8Encoding.GetMaxByteCount(Int32 charCount)
em Elasticsearch.Net.Utf8Json.JsonWriter.WriteString(String value)
em Serialize(Byte[][] , Object[] , JsonWriter& , DocumentoBuscaInfo , IJsonFormatterResolver )
em Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) em Nest.ProxyRequestFormatterBase2.Serialize(JsonWriter& writer, TRequestInterface value, IJsonFormatterResolver formatterResolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting)
em Elasticsearch.Net.SerializableData1.Write(Stream writableStream, IConnectionConfigurationValues settings) em Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) em Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) em Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

Exception:

Elasticsearch.Net.UnexpectedElasticsearchClientException: Número excessivo de caracteres. O número de bytes resultante é maior do que pode ser retornado como int.
Nome do parâmetro: charCount ---> System.ArgumentOutOfRangeException: Número excessivo de caracteres. O número de bytes resultante é maior do que pode ser retornado como int.
Nome do parâmetro: charCount
em System.Text.UTF8Encoding.GetMaxByteCount(Int32 charCount)
em Elasticsearch.Net.Utf8Json.JsonWriter.WriteString(String value)
em Serialize(Byte[][] , Object[] , JsonWriter& , DocumentoBuscaInfo , IJsonFormatterResolver )
em Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) em Nest.ProxyRequestFormatterBase2.Serialize(JsonWriter& writer, TRequestInterface value, IJsonFormatterResolver formatterResolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver)
em Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting)
em Elasticsearch.Net.SerializableData1.Write(Stream writableStream, IConnectionConfigurationValues settings) em Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) em Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) em Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- Fim do rastreamento de pilha de exceções internas ---
em Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) em Nest.ElasticClient.Index[TDocument](IIndexRequest1 request)
em Docspider.Busca.ElasticSearchDL.IndexacaoBusca.IncluirDocumento(DocumentoBuscaInfo documentoBusca)
em Docspider.Busca.BL.IndexacaoBusca.IncluirDocumento(DocumentoBuscaInfo documentoBusca, Boolean indexacaoCompleta)
em Docspider.Busca.BL.IndexacaoBusca.ObterDocumentosParaIndexacao(Nullable1 dataBaseParaBuscaDeDocumentos, List1 idsDeDocumentosNaoIndexadosAnteriormente, ModoDeIndexacao modoDeIndexacao, Boolean& ocorreramErrosNaIndexacao)

[stevejgordon]

@juliozambonin This looks like hard one to overcome due to the extremely large file. It's failing deep in the JSON serializer because the Attachement.Content string length is simply too long for the max size of an integer. Are you able to look at splitting your content?

[stevejgordon]

This guidance may also be useful since such large documents may perform poorly in Elasticsearch also - https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html

[juliozambonin]

Thanks for the feedback Steve.
I liked the article you indicated. Unfortunately, our client needs to index large documents.
I will do a test using Bulk Api, indexing directly from a previously created json file.
Don't you see the possibility of increasing this capacity within Elasticsearch .net?

[stevejgordon]

@juliozambonin Apologies for the lack of reply. I missed this on my last sweep for updates. In the immediate future, we won't change the serializer which is where the limitation is hit right now. In the next major release, we already plan to look at using the Microsoft System.Text.Json serializer. I'm not convinced that will necessarily solve this extreme case though.

You could perhaps try using the low-level client and handle the serialisation yourself in the meantime as a possible workaround.

I'm going to close this issue on the basis that we're tracking serialisation in our wider 8.x plan.