ES|QL - add dense_vector field type and to_dense_vector function to release builds #135604

carlosdelest · 2025-09-29T09:36:39Z

Take dense_vector field type an TO_DENSE_VECTOR function out of snapshot builds.

This means that support must be explicitly removed for some functions and operators:

Aggregation functions
Multivalued functions - makes no sense as dense_vector is internally represented as a multivalued float

It is alsosupported by the following functions:

Is null / Is not null
CASE
TO_STRING
Equals / Not equals

Feature flag is removed, and capability checks removed where applicable.

Docs have been updated for the above functions, vector similarity functions, and knn.

…of snapshot

github-actions · 2025-09-29T09:38:18Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

…tech-preview' into feature/esql-dense-vector-tech-preview

…ctor-tech-preview

elasticsearchmachine · 2025-09-30T05:49:17Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-09-30T05:49:17Z

Pinging @elastic/kibana-esql (ES|QL-ui)

…ctor-tech-preview

alex-spies

Looks good - I have some minor, non-blocking comments.

I hate to say it, but we probably need to apply the test-release and test-full-bwc labels and see what additional test failures we may now get.

alex-spies · 2025-09-30T09:03:26Z

.../plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/plugin/EsqlCorePlugin.java


    public static final FeatureFlag AGGREGATE_METRIC_DOUBLE_FEATURE_FLAG = new FeatureFlag("esql_aggregate_metric_double");
-    public static final FeatureFlag DENSE_VECTOR_FEATURE_FLAG = new FeatureFlag("esql_dense_vector");


++

I don't think we will use feature flags for under construction types going forward. We don't ask remote nodes if they have the same feature flag enabled, and don't have a good mechanism (yet) for when they don't.

alex-spies · 2025-09-30T09:06:53Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/ToString.java

    }

+    @ConvertEvaluator(extraName = "FromFloat")
+    static BytesRef fromFloat(float flt) {


Why do we need this now?

Dense vectors are internally represented as Float blocks - I don't think we had a valid ToString on floats as they are not representable AFAIU

Ah, okay. The javadoc on the type says it's arrays of doubles, which got me confused. Thanks!

alex-spies · 2025-09-30T09:07:21Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

                def(Knn.class, tri(Knn::new), "knn"),
                def(CosineSimilarity.class, CosineSimilarity::new, "v_cosine"),
                def(DotProduct.class, DotProduct::new, "v_dot_product"),
                def(L1Norm.class, L1Norm::new, "v_l1_norm"),
                def(L2Norm.class, L2Norm::new, "v_l2_norm"),


I guess these will be released in a separate PR?

yes, we want to improve performance before releasing vector similarity functions

alex-spies · 2025-09-30T09:08:00Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/ExpressionWritables.java

        entries.add(ToDatetime.ENTRY);
        entries.add(ToDateNanos.ENTRY);
        entries.add(ToDegrees.ENTRY);
-        if (EsqlCapabilities.Cap.TO_DENSE_VECTOR_FUNCTION.isEnabled()) {


Nice that you're cleaning this up, too. Not technically necessary, but nice.

...in/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/TypeResolutions.java

...c/test/java/org/elasticsearch/xpack/esql/expression/function/aggregate/AbsentErrorTests.java

...va/org/elasticsearch/xpack/esql/expression/function/scalar/multivalue/MvCountErrorTests.java

…tech-preview' into feature/esql-dense-vector-tech-preview

elasticsearchmachine · 2025-09-30T10:46:05Z

Hi @carlosdelest, I've updated the changelog YAML for you.

…ctor-tech-preview # Conflicts: # server/src/main/resources/transport/upper_bounds/9.2.csv

…tech-preview' into feature/esql-dense-vector-tech-preview

ES|QL - dense_vector field type and to_dense_vector function are out …

f943ceb

…of snapshot

carlosdelest added ES|QL-ui Impacts ES|QL UI Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team v9.2.0 Team:ES|QL :Search Relevance/ES|QL Search functionality in ES|QL labels Sep 29, 2025

carlosdelest and others added 15 commits September 29, 2025 18:51

Remove capability usage and feature flag

3df6ea3

Fix multivalued tests

48221c2

Fix aggregation and comparisons with dense_vectors

11f2cac

[CI] Auto commit changes from spotless

8a103bf

Added to_dense_vector docs

bea08d7

Merge remote-tracking branch 'carlosdelest/feature/esql-dense-vector-…

9ae198a

…tech-preview' into feature/esql-dense-vector-tech-preview

Merge remote-tracking branch 'origin/main' into feature/esql-dense-ve…

e3ae70d

…ctor-tech-preview

Add docs for vector similarity functions

972ad3e

Add docs

3383f1a

dense_vector for Case and IsNotNull

e4b791c

Fix tests

7af178d

Fix tests

ffbdafc

Add docs

3a40849

Add docs

37caab4

Fix CBE for CaseTests when CBEs are thrown on beginPositionEntry

9538a33

carlosdelest marked this pull request as ready for review September 30, 2025 05:48

carlosdelest requested review from afoucret and ioanatia September 30, 2025 05:49

elasticsearchmachine removed the Team:Search - Relevance The Search organization Search Relevance team label Sep 30, 2025

carlosdelest changed the title ~~ES|QL - add dense_vector field type and to_dense_vector function~~ ES|QL - add dense_vector field type and to_dense_vector function to release builds Sep 30, 2025

carlosdelest requested a review from alex-spies September 30, 2025 09:08

carlosdelest added 2 commits September 30, 2025 11:11

Fix capability check

179ee30

Merge remote-tracking branch 'origin/main' into feature/esql-dense-ve…

f4dbd9d

…ctor-tech-preview

alex-spies approved these changes Sep 30, 2025

View reviewed changes

elasticsearchmachine and others added 5 commits September 30, 2025 12:00

Changed method names

ec5e229

Merge branch 'main' into feature/esql-dense-vector-tech-preview

545a9d1

[CI] Auto commit changes from spotless

735b15d

Added transport version for dense_vector data type

2c31d58

Merge remote-tracking branch 'carlosdelest/feature/esql-dense-vector-…

98b1dda

…tech-preview' into feature/esql-dense-vector-tech-preview

carlosdelest added test-full-bwc Trigger full BWC version matrix tests test-release Trigger CI checks against release build labels Sep 30, 2025

carlosdelest and others added 2 commits September 30, 2025 12:46

Update docs/changelog/135604.yaml

2ed9e8b

Merge remote-tracking branch 'origin/main' into feature/esql-dense-ve…

46dabc2

…ctor-tech-preview # Conflicts: # server/src/main/resources/transport/upper_bounds/9.2.csv

carlosdelest enabled auto-merge (squash) September 30, 2025 10:53

[CI] Auto commit changes from spotless

3452979

alex-spies mentioned this pull request Sep 30, 2025

ESQL: Update mechanism for under construction data types #135697

Merged

elasticsearchmachine added 2 commits September 30, 2025 14:12

Fix mixed cluster tests

4d16a44

Merge remote-tracking branch 'carlosdelest/feature/esql-dense-vector-…

f1425b6

…tech-preview' into feature/esql-dense-vector-tech-preview

carlosdelest removed test-full-bwc Trigger full BWC version matrix tests test-release Trigger CI checks against release build labels Sep 30, 2025

carlosdelest and others added 4 commits September 30, 2025 16:14

Merge branch 'main' into feature/esql-dense-vector-tech-preview

ab90f79

Renamed capability to avoid failures on mixed clusters

7af9e82

Merge remote-tracking branch 'carlosdelest/feature/esql-dense-vector-…

1befbfa

…tech-preview' into feature/esql-dense-vector-tech-preview

Merge branch 'main' into feature/esql-dense-vector-tech-preview

6f7e9bb

afoucret approved these changes Sep 30, 2025

View reviewed changes

elasticsearchmachine and others added 2 commits September 30, 2025 18:41

Add capability check to avoid errors on mixed clusters

a901930

Merge branch 'main' into feature/esql-dense-vector-tech-preview

d0eaa52

carlosdelest disabled auto-merge September 30, 2025 17:56

carlosdelest merged commit 771aaff into elastic:main Sep 30, 2025
33 of 34 checks passed


		public static final FeatureFlag AGGREGATE_METRIC_DOUBLE_FEATURE_FLAG = new FeatureFlag("esql_aggregate_metric_double");
		public static final FeatureFlag DENSE_VECTOR_FEATURE_FLAG = new FeatureFlag("esql_dense_vector");

ES|QL - add dense_vector field type and to_dense_vector function to release builds #135604

ES|QL - add dense_vector field type and to_dense_vector function to release builds #135604

Uh oh!

Conversation

carlosdelest commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 29, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

carlosdelest Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

carlosdelest Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

carlosdelest commented Sep 29, 2025 •

edited

Loading