System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter`1.IRegexpQuery

[vadim-zakharyan]

NEST/Elasticsearch.Net version:
7.15.2
Elasticsearch version:
7.17.0 in cluster (3 master, 5 data)
.NET runtime version:
5.0
Operating system version:
alpine (in Docker)
Description of the problem including expected versus actual behavior:
in short:
We have a search method with aggregations.
Today we got exception in this search method that was working fine with the same parameters few weeks (sounds not convincing
:) )
Some details:
We have 2 API nodes behind load balancer
One of the node suddenly started to raise exceptions (look below).
Second node was serving the same requests without any issues.

After docker container restart in the failure node the same method started to work as usual.

Not sure what part of the api shall i check for the issue.
Will appreciate any help!

Steps to reproduce:

1. Have no idea, actually

Expected behavior
well, to search without exceptions

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):
Exception
**FailureReason**: Unrecoverable/Unexpected BadRequest while attempting POST on http://10.0.0.5:9200/5catalog_flat_products/_search?typed_keys=true [1] BadRequest: Node: http://10.0.0.5:9200/ Exception: MissingMethodException Took: 00:00:00.0103978 **Audit exception in step 1** BadRequest: System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter1.IRegexpQuery(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'.
at Elasticsearch.Net.Nest_IAggregationContainerFormatter3.Serialize(JsonWriter& , IAggregationContainer , IJsonFormatterResolver )
at Nest.VerbatimDictionaryKeysBaseFormatter3.Serialize(JsonWriter& writer, TDictionary value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Nest_ISearchRequestFormatter1.Serialize(JsonWriter& , ISearchRequest , IJsonFormatterResolver ) at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeAsync[T](Stream stream, T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.SerializableData1.WriteAsync(Stream writableStream, IConnectionConfigurationValues settings, CancellationToken cancellationToken)
at Elasticsearch.Net.HttpConnection.SetContentAsync(HttpRequestMessage message, RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) **Inner Exception**: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter1.IRegexpQuery(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'.
System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter1.IRegexpQuery(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'. at Elasticsearch.Net.Nest_IAggregationContainerFormatter3.Serialize(JsonWriter& , IAggregationContainer , IJsonFormatterResolver ) at Nest.VerbatimDictionaryKeysBaseFormatter3.Serialize(JsonWriter& writer, TDictionary value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Nest_ISearchRequestFormatter1.Serialize(JsonWriter& , ISearchRequest , IJsonFormatterResolver )
at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeAsync[T](Stream stream, T value, IJsonFormatterResolver resolver)
at Elasticsearch.Net.SerializableData1.WriteAsync(Stream writableStream, IConnectionConfigurationValues settings, CancellationToken cancellationToken) at Elasticsearch.Net.HttpConnection.SetContentAsync(HttpRequestMessage message, RequestData requestData, CancellationToken cancellationToken) at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter1.IRegexpQuery(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'. ---> System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter1.IRegexpQuery(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'.
at Elasticsearch.Net.Nest_IAggregationContainerFormatter3.Serialize(JsonWriter& , IAggregationContainer , IJsonFormatterResolver )
at Nest.VerbatimDictionaryKeysBaseFormatter3.Serialize(JsonWriter& writer, TDictionary value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Nest_ISearchRequestFormatter1.Serialize(JsonWriter& , ISearchRequest , IJsonFormatterResolver ) at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeAsync[T](Stream stream, T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.SerializableData1.WriteAsync(Stream writableStream, IConnectionConfigurationValues settings, CancellationToken cancellationToken)
at Elasticsearch.Net.HttpConnection.SetContentAsync(HttpRequestMessage message, RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
at Nest.ElasticClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct)
at api.search.Services.SearchServiceMultiConnectionsBase.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct) in /src/api.search/Services/SearchServiceMultiConnectionsBase.cs:line 31
at api.search.Services.SearchCatalogService.SearchFlatAsync(SearchCatalogPagingDto dto, Boolean useHq, CancellationToken ct) in /src/api.search/Services/SearchCatalogService.cs:line 136
at api.logic.Managers.CatalogManager.SearchAsync(SearchCatalogPagingRequestDto request, CancellationToken ct) in /src/api.logic/Managers/CatalogManager.cs:line 69
at lambda_method1013(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

[stevejgordon]

Hi, @vadim-zakharyan.

Without reproduction steps, I don't think there's much we can do to investigate this or identify the cause.

Based on the call stack this looks to be thrown within a Regexp query used in an aggregation.

The MissingMethodException is unusual. The most likely cause would be that an incorrect DLL had been loaded which is incompatible with the compiled application. It's possible the restart resolved that somehow. Had the container recently been redeployed before the exception occurred? It's worth reviewing your build process and source of NuGet packages to ensure that the built image contains the expected assembly DLLs and versions match.

Has this re-occurred since?

[stevejgordon]

@vadim-zakharyan Checking in to see if this still requires attention?

[stevejgordon]

Closing due to no response.

[SkyChenSky]

NEST/Elasticsearch.Net version:
7.14.0
Elasticsearch version:
7.14.0 in cluster (3 master)
.NET runtime version:
6.0
Operating system version:
Windows Server 2016
Description of the problem including expected versus actual behavior:
in short:
I also encountered the same problem. We have two api node. They have normal work for a month from 2023.3.3 to 2023.4.15.
No updates were made during this period.
One of the node suddenly started to raise exceptions (look below).
Second node was serving the same requests without any issues.

System.MissingMethodException: Method not found: 'Nest.Serialize Nest.IQueryContainer.get_Boosting()'. at Elasticsearch.Net.Nest_IQueryContainerFormatter3.Serialize(JsonWriter& , IQueryContainer , IJsonFormatterResolver ) at Elasticsearch.Net.Nest_ICountRequestFormatter1.Serialize(JsonWriter& , ICountRequest , IJsonFormatterResolver ) at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) at Elasticsearch.Net.SerializableData1.Write(Stream writableStream, IConnectionConfigurationValues settings)
at Elasticsearch.Net.HttpConnection.SetContent(HttpRequestMessage message, RequestData requestData)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters