[FEATURE] Support `knn_search` endpoint

[TonyWang666]

Elastic.Clients.Elasticsearch version:
8.0.1
Elasticsearch version:
8.3.2
.NET runtime version:
6.0.11
Operating system version:
macOS Catalina Version 10.15.7
Description of the problem including expected versus actual behavior:
The latest version(8.0.1) of dotnet client doesn't have knn_search method like what official page shows.
When I use search with Knn in the body, it return 400 with following error:
{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [knn].","line":1,"col":8}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [knn].","line":1,"col":8},"status":400}

Steps to reproduce:

1. download NuGet Package Elastic.Clients.Elasticsearch with latest version 8.0.1
2. Connect to Elasticsearch Database with Uri and BasicAuthentication Info(AccountName&Password)
3. Try to get response with SearchAsync endpoint with following C# code:

var response = await _client.SearchAsync<ImageSearchDocument>(
            s => s
                .Knn(k => k
                    .QueryVector(embedding_vector)
                    .Field(_EmbeddingVectorStr)
                    .k(searchTotal)
                    .NumCandidates(numCandidate)
                 )
                .Index(FullImageSearchIndexName)
        );

Expected behavior
What I expect is the response of knn_search result like what I got from Postman with same requestBody, uri, and accountInfo:
{ "took": 5, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 20, "relation": "eq" }, "max_score": 0.052058797, "hits": [ { "_index": "XXXXXXXX", "_id": "0", "_score": 0.052058797, "_source": { ...... } } } }
Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

Invalid Elasticsearch response built from a unsuccessful (400) low level call on POST: /st1_brand_safety-full_image_search-1-index/_search?_source_excludes=embedding_vector
 - [1] BadResponse: Node: https://XXXXXXXXXXXXXX.com/ Took: 00:00:02.3115470
# OriginalException: Elastic.Transport.TransportException: Request failed to execute. Call: Status code 400 from: POST /st1_brand_safety-full_image_search-1-index/_search?_source_excludes=embedding_vector. ServerError: Type: parsing_exception Reason: "Unknown key for a START_OBJECT in [knn]."
# Audit trail of this API call:
# Request:
{"knn":{"field":"embedding_vector","k":10,"num_candidates":20,"query_vector":[0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5]}}
# Response:

{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [knn].","line":1,"col":8}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [knn].","line":1,"col":8},"status":400}

[TonyWang666]

I think the root cause is because Elasticsearch 8.3.2 doesn't support knn parameter in search endpoint. Instead, Elasticsearch 8.3.2 only support knn_search endpoint.

However, Latest Elasticsearch Dotnet Client 8.0.1 doesn't have knn_search which is different from the official doc.
Could the developer add knn_search to the Elasticsearch DotnetClient 8.0.1 by any chance so the developer who want to use this great feature would not be blocked by this issue?

[stevejgordon]

Hi, @TonyWang666.

Indeed, your assessment is correct. The knn parameter can be used in search from 8.4 of Elasticsearch. If you're able to upgrade, this should function as expected.

The new client doesn't support all endpoints, and knn_search is not available. While deprecated, we should consider including this in a future release for use by consumers targeting older server releases before 8.4.

[stevejgordon]

@TonyWang666 After careful consideration, we won't generate support for this deprecated endpoint in the new client. Instead, consumers who need this functionality should upgrade to 8.4 server, where it is available and supported on the search endpoint.