ES|QL - KNN function option changes

[carlosdelest]

KNN function allowed the use of min_candidates to set the minimum number of candidates to retrieve per shard. What it did was replace k for the underlying knn query.

This caused the following problems:

• Users were unable to modify num_candidates to explore more candidates per shard before getting the top k
• min_candidates was subject to oversampling, which was not clear in the docs

This PR adds more options to KNN function:

• Allows setting k to override the LIMIT applied to the function, and retrieve more results from each shard
• Changes min_candidates to be equivalent to setting num_candidates
• Adds visit_percentage option for disk_bbq index type

Usage including all possible options for KNN would become:

FROM test
| WHERE KNN(dense_vector, [0.1, 0.2, 0.3],
    {"k": 10, "min_candidates": 20, "rescore_oversample": 1.5, "similarity": 0.5, "boost": 2.0, "visit_percentage": 0.25})

[elasticsearchmachine]

Hi @carlosdelest, I've created a changelog YAML for you.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/kibana-esql (ES|QL-ui)

[jimczi]

I left one nit regarding how the default is set for visit_percentage, LGTM otherwise.

[jimczi]

Not sure about this wording and where this default is coming from, @benwtrent can you verify?

[benwtrent]

it doesn't default to anything. We shouldn't default it to anything. We dynamically set it according to num_candidates, which is dynamically determined via k (if not provided), which is required for knn search to work.