[ES|QL] addressing vector similarity concurrency issue with byte vectors

[pmpailis]

For ES|QL vector similarity queries, it seems that we initialize the VectorSimilarityFunctionConfig once and then pass it to the workers. However, each worker could potentially update the underlying byte representation depending on the elementType in

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

Line 2871 in 9a9bf3a

if (getElementType() == ElementType.BYTE || getElementType() == ElementType.BIT) {

Given that the same instance might be accessed by different threads, there could be a scenario where the supporting byte[] for a worker might be reset by another worker through the:

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

Line 3419 in 9a9bf3a

vectorAsBytes = new byte[vector.length];

so when we try to compute the vector similarity, we might end up with incorrect scores.

In this PR, we eagerly try to identify whether we need a byte or float array and initialize/read data accordingly, to avoid both passing this responsibility to each worker and the extra space from the float array when we have to deal with byte vectors.

Example log output (using the 6ea437ab as reference) where the 0...0 vectors indicate the issue:

1>[o.e.i.m.v.DenseVectorFieldMapper$VectorSimilarityFunctionConfig][node_s0][esql_worker][T#1] org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper$VectorSimilarityFunctionConfig@87f7d80a resetting byte vector : [0,...,0] hash: [B@6ea437ab
2> 342d2a23-5c48-4d86-ae68-efe7f15c9f7b Calculating Hamming distance between vectors: left: [-62, 60, -35, 65, -111, 100, 127, 38, -117, 109, 89, 127, 127, 6, -104, 35, -47, 19, 91, -22, 19, -53, 24, -39, 56, 78, 74, 12, 36, 58, 98, -8, 107, -1, -120, 41, 24, -38, -23, -4] hash: [B@3ef01bd9 and right: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] hash: [B@6ea437ab
2> 342d2a23-5c48-4d86-ae68-efe7f15c9f7b Computed Hamming distance: 160.0 between vectors: left: [-62, 60, -35, 65, -111, 100, 127, 38, -117, 109, 89, 127, 127, 6, -104, 35, -47, 19, 91, -22, 19, -53, 24, -39, 56, 78, 74, 12, 36, 58, 98, -8, 107, -1, -120, 41, 24, -38, -23, -4] hash: [B@3ef01bd9 and right: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] hash: [B@6ea437ab
2> 9ab2ddc9-1d74-4e94-b00a-3effab104c33 Calculating Hamming distance between vectors: left: [-1, 50, 101, -62, -119, 1, -21, -56, 124, 36, -53, 47, 18, 46, 34, -65, 124, 52, -83, -49, -99, -44, 57, -38, -12, -6, 8, 95, 19, 68, 62, -81, -55, 50, -83, 44, 30, 43, -42, 112] hash: [B@3ef01bd9 and right: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] hash: [B@6ea437ab
1> [o.e.i.m.v.DenseVectorFieldMapper$VectorSimilarityFunctionConfig][node_s0][esql_worker][T#1] org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper$VectorSimilarityFunctionConfig@87f7d80a will now set to : [46.0, 59.0, 37.0, -2.0, 32.0, -114.0, -58.0, 51.0, 55.0, 77.0, -4.0, 76.0, 85.0, 28.0, -102.0, 65.0, 120.0, 6.0, 106.0, 97.0, -119.0, -11.0, 66.0, 78.0, 26.0, -60.0, 92.0, 53.0, 83.0, -24.0, 112.0, 86.0, -35.0, 34.0, -50.0, 117.0, 110.0, -75.0, 111.0, 73.0] hash: [B@6ea437ab
2> 9ab2ddc9-1d74-4e94-b00a-3effab104c33 Computed Hamming distance: 165.0 between vectors: left: [-1, 50, 101, -62, -119, 1, -21, -56, 124, 36, -53, 47, 18, 46, 34, -65, 124, 52, -83, -49, -99, -44, 57, -38, -12, -6, 8, 95, 19, 68, 62, -81, -55, 50, -83, 44, 30, 43, -42, 112] at hash: [B@3ef01bd9 and right: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] at hash: [B@6ea437ab
2> 91d6de2f-5c80-487c-8bf0-f423b845b99f Calculating Hamming distance between vectors: left: [-55, 2, -48, 70, 112, -65, 47, 5, -5, -45, -7, 23, 95, 8, 108, 48, -87, -37, 78, -26, 59, 57, 67, 104, 11, 91, -69, 55, 17, -75, 0, -63, -26, -127, 62, -30, 72, -103, 38, 84] at hash: [B@3ef01bd9 and right: [46, 59, 37, -2, 32, -114, -58, 51, 55, 77, -4, 76, 85, 28, -102, 65, 120, 6, 106, 97, -119, -11, 66, 78, 26, -60, 92, 53, 83, -24, 112, 86, -35, 34, -50, 117, 110, -75, 111, 73] at hash: [B@6ea437ab
1> [2025-11-10T12:22:36,178][ERROR][o.e.i.m.v.DenseVectorFieldMapper$VectorSimilarityFunctionConfig][node_s0][esql_worker][T#1] org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper$VectorSimilarityFunctionConfig@7458bd87 new vector as bytes: [46, 59, 37, -2, 32, -114, -58, 51, 55, 77, -4, 76, 85, 28, -102, 65, 120, 6, 106, 97, -119, -11, 66, 78, 26, -60, 92, 53, 83, -24, 112, 86, -35, 34, -50, 117, 110, -75, 111, 73] hash: [B@6ea437ab

Closes #137625

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @pmpailis, I've created a changelog YAML for you.

[carlosdelest]

Thanks for fixing this @pmpailis ! LGTM

A nit about adding two different classes - but I'm not sure it will be worth doing

[carlosdelest]

Let's add a similar assertion to the vector() method.

I think it would be cleaner to have two separate config classes 🤔 . WDYT? Is it worthwhile?

[pmpailis]

I think that this would also introduce some overhead in updating DenseVectorSimilarityProcessor and possibly the calls for the similarity functions.

I've updated to make use of org.elasticsearch.search.vectors.VectorData in 31ff07b, but honestly I don't feel strong about either approach. IMO it should be ok to leave as-is, but happy to update or split if you prefer.

[carlosdelest]

Let's keep as it is - thanks for checking it

[benwtrent]

bit requires dims % 8 == 0, idk if that will cause flakiness here.

[pmpailis]

When setting up the docs we have:

        for (int j = 0; j < numDims; j++) {
            switch (elementType) {
                case FLOAT -> vector.add(randomFloat());
                case BYTE, BIT -> vector.add((byte) randomIntBetween(-128, 127));

So, I think that while dims might be a bit misleading (as we're adding a byte for every dim) and also given that we don't specify dims in the mapping, we should probably be ok. Will update though to make it a bit more clear.

[pmpailis]

run elasticsearch-ci/part-5

[nik9000]

Just a heads up, this got muted recently. I changed the name of one of the parameters away from a java-default toString.

[pmpailis]

Thanks @nik9000 ! Merged main & unmuted the failing VectorSimilarityFunctionsIT test.