Add vector query execution semantics: filtered ANN, threshold search, and compound retrieval (Phase 3)

[xiangfu0]

Summary

• Backend capability model: VectorBackendCapabilities declares 5 query-time capabilities per backend (topKAnn, filterAwareSearch, approximateRadius, exactRerank, runtimeSearchParams), wired into VectorBackendType.getCapabilities()
• Execution mode enum: VectorExecutionMode defines 8 explicit modes with centralized selection logic
• Filtered ANN semantics: FilterPlanNode detects AND(VECTOR_SIMILARITY, non-vector-filter) patterns; execution mode is explicit in explain output
• Threshold/radius search: New vectorDistanceThreshold query option enables distance-based filtering via ANN candidate generation + exact threshold refinement from forward index; works in both indexed and exact-scan fallback paths
• Compound retrieval: Filter + top-K, filter + threshold, and top-K + threshold patterns all wired with correct execution mode reporting
• Explain/debug: Execution mode now visible in both human-readable and structured explain output for all vector queries

Execution Mode Reference

The runtime selects one of 8 explicit modes based on query shape. The selected mode is reported in EXPLAIN output.

Mode Selection Rules (centralized in VectorQueryExecutionContext.selectExecutionMode)

 Query Shape                          → Execution Mode
 ─────────────────────────────────────────────────────────────
 No vector index on segment           → EXACT_SCAN
 Threshold + metadata filter          → ANN_THRESHOLD_THEN_FILTER
 Threshold (no filter)                → ANN_THRESHOLD_SCAN
 Metadata filter + rerank             → ANN_THEN_FILTER_THEN_RERANK
 Metadata filter (no rerank)          → ANN_THEN_FILTER
 Rerank (no filter)                   → ANN_TOP_K_WITH_RERANK
 Plain top-K                          → ANN_TOP_K

Mode Details

Mode	What happens	When selected
ANN_TOP_K	ANN index returns top-K docs	Default, no filter/rerank/threshold
ANN_TOP_K_WITH_RERANK	ANN returns candidates, exact distance rerank via forward index, return top-K	vectorExactRerank=true or IVF_PQ default
ANN_THEN_FILTER	ANN returns top-K, bitmap AND with metadata filter	WHERE vectorSimilarity(...) AND col = 'x'
ANN_THEN_FILTER_THEN_RERANK	ANN returns candidates, bitmap AND with filter, exact rerank survivors	Filter + vectorExactRerank=true
FILTER_THEN_ANN	Filter first, then ANN on subset	Reserved for future filter-aware backends
ANN_THRESHOLD_SCAN	ANN generates candidate pool, exact distance check against threshold	vectorDistanceThreshold set, no filter
ANN_THRESHOLD_THEN_FILTER	ANN candidates + threshold refinement + bitmap AND with filter	vectorDistanceThreshold + metadata filter
EXACT_SCAN	Brute-force scan of forward index	No vector index on segment

How to see the execution mode

-- EXPLAIN shows the mode
EXPLAIN PLAN FOR
SELECT productId FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
  AND category = 'electronics'

Output includes executionMode:ANN_THEN_FILTER in the vector operator's explain attributes.

How the mode is determined (code path)

1. FilterPlanNode.constructPhysicalOperator() detects AND(VECTOR_SIMILARITY, non-vector siblings) via hasNonVectorSibling()
2. VectorSimilarityFilterOperator constructor receives hasMetadataFilter flag + VectorSearchParams (which carries distanceThreshold)
3. VectorQueryExecutionContext.selectExecutionMode(hasVectorIndex, hasMetadataFilter, hasThresholdPredicate, exactRerank) returns the mode
4. Mode is stored in VectorExplainContext and reported via toExplainString() and explainAttributes()

Threshold search semantics

When vectorDistanceThreshold is set:

• Indexed segments: ANN generates a candidate pool (size = vectorMaxCandidates, default topK * 10), then exact distance is computed for each candidate. Only candidates within the threshold are returned. This is approximate — vectors within the threshold but outside the ANN candidate pool will be missed. Increase vectorMaxCandidates for better recall.
• Unindexed segments: ExactVectorScanFilterOperator performs brute-force scan and returns all vectors within the threshold (exact, no recall loss).
• Distance space: The threshold compares against internal distance values. For EUCLIDEAN, this is squared L2 (sum of squared differences, no sqrt). The SQL l2Distance() function returns sqrt, so vectorDistanceThreshold=100 matches vectors where l2Distance() <= 10.

Sample Table Config

HNSW (default, supports realtime)

{
  "tableName": "products_OFFLINE",
  "tableType": "OFFLINE",
  "fieldConfigList": [
    {
      "name": "embedding",
      "indexTypes": ["VECTOR"],
      "encodingType": "RAW",
      "properties": {
        "vectorIndexType": "HNSW",
        "vectorDimension": "128",
        "vectorDistanceFunction": "COSINE"
      }
    }
  ]
}

IVF_FLAT (offline only, tunable nprobe)

{
  "tableName": "articles_OFFLINE",
  "tableType": "OFFLINE",
  "fieldConfigList": [
    {
      "name": "embedding",
      "indexTypes": ["VECTOR"],
      "encodingType": "RAW",
      "properties": {
        "vectorIndexType": "IVF_FLAT",
        "vectorDimension": "128",
        "vectorDistanceFunction": "EUCLIDEAN",
        "nlist": "64",
        "version": "1"
      }
    }
  ]
}

IVF_PQ (offline only, compressed, rerank enabled by default)

{
  "tableName": "images_OFFLINE",
  "tableType": "OFFLINE",
  "fieldConfigList": [
    {
      "name": "embedding",
      "indexTypes": ["VECTOR"],
      "encodingType": "RAW",
      "properties": {
        "vectorIndexType": "IVF_PQ",
        "vectorDimension": "128",
        "vectorDistanceFunction": "EUCLIDEAN",
        "nlist": "64",
        "pqM": "16",
        "pqNbits": "8",
        "version": "1"
      }
    }
  ]
}

Sample Queries

1. Basic top-K ANN (unchanged from Phase 1/2)

Execution mode: ANN_TOP_K

SELECT productId, productName
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
ORDER BY l2Distance(embedding, ARRAY[0.1, 0.2, ...]) ASC
LIMIT 10

2. Top-K ANN with exact rerank (Phase 2)

Execution mode: ANN_TOP_K_WITH_RERANK

SET vectorExactRerank = true;
SET vectorMaxCandidates = 100;

SELECT productId, cosineDistance(embedding, ARRAY[0.1, 0.2, ...]) AS score
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
ORDER BY score ASC
LIMIT 10

3. Filtered ANN — metadata filter + top-K (Phase 3, new)

Execution mode: ANN_THEN_FILTER

SELECT productId, productName
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
  AND category = 'electronics'
  AND price < 100
ORDER BY l2Distance(embedding, ARRAY[0.1, 0.2, ...]) ASC
LIMIT 10

ANN returns top-10 candidates, bitmap AND with metadata filters reduces the set. Result is <= 10 rows.

4. Filtered ANN with rerank (Phase 3, new)

Execution mode: ANN_THEN_FILTER_THEN_RERANK

SET vectorExactRerank = true;
SET vectorMaxCandidates = 200;

SELECT productId, cosineDistance(embedding, ARRAY[0.1, 0.2, ...]) AS score
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
  AND category = 'electronics'
ORDER BY score ASC
LIMIT 10

5. Threshold/radius search — find all similar vectors within a distance (Phase 3, new)

Execution mode: ANN_THRESHOLD_SCAN

SET vectorDistanceThreshold = 0.3;

SELECT productId, cosineDistance(embedding, ARRAY[0.1, 0.2, ...]) AS score
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 100)
ORDER BY score ASC
LIMIT 50

ANN generates a candidate pool (default: topK * 10 = 1000), exact distance is computed for each. Only candidates with distance <= 0.3 are returned.

6. Compound: metadata filter + threshold search (Phase 3, new)

Execution mode: ANN_THRESHOLD_THEN_FILTER

SET vectorDistanceThreshold = 0.5;

SELECT articleId, cosineDistance(embedding, ARRAY[0.1, 0.2, ...]) AS score
FROM articles
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 100)
  AND publishDate > '2024-01-01'
  AND language = 'en'
ORDER BY score ASC
LIMIT 20

7. IVF with tuned nprobe (Phase 2, works with all Phase 3 modes)

SET vectorNprobe = 16;
SET vectorDistanceThreshold = 100.0;

SELECT imageId, l2Distance(embedding, ARRAY[0.1, 0.2, ...]) AS dist
FROM images
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 200)
  AND label = 'cat'
ORDER BY dist ASC
LIMIT 50

8. EXPLAIN to see execution mode

EXPLAIN PLAN FOR
SELECT productId
FROM products
WHERE vectorSimilarity(embedding, ARRAY[0.1, 0.2, ...], 10)
  AND category = 'electronics'

Output includes:

VECTOR_SIMILARITY_INDEX(
  indexLookUp:vector_index,
  executionMode:ANN_THEN_FILTER,
  backend:HNSW,
  distanceFunction:COSINE,
  effectiveCandidateCount:10,
  topK to search:10
)

Query Options Reference

Option	Type	Default	Description
vectorNprobe	int	4	IVF probe count (IVF_FLAT/IVF_PQ only)
vectorExactRerank	boolean	backend-dependent	Re-score ANN candidates with exact distance
vectorMaxCandidates	int	topK x 10	Max ANN candidates before refinement
vectorDistanceThreshold	float	(none)	Distance threshold for radius search (new in Phase 3)

Benchmark Results

Exact brute-force scan baseline for Phase 3 query patterns. Measured on synthetic Gaussian vectors with L2 (Euclidean squared) distance. All latencies are single-segment, single-threaded.

Filtered Exact Scan (50K vectors, 128-dim, topK=10)

Filter Selectivity	p50 Latency	p95 Latency	Avg Results
1% (highly selective)	0.37ms	1.17ms	10
10%	1.53ms	1.69ms	10
50%	6.48ms	7.19ms	10
90% (low selectivity)	9.42ms	9.90ms	10
No filter (baseline)	9.63ms	10.52ms	10

Threshold/Radius Scan (50K vectors, 128-dim)

Threshold	p50 Latency	p95 Latency	Avg Results
200.0 (tight)	2.51ms	2.91ms	1,740
240.0 (medium)	3.22ms	3.68ms	16,612
260.0 (loose)	3.34ms	3.87ms	28,922

Filtered Threshold Scan (50K vectors, 128-dim, threshold=240.0)

Filter Selectivity	p50 Latency	p95 Latency	Avg Results
10%	0.36ms	0.44ms	1,684
50%	1.85ms	2.30ms	8,312

Scale Test (100K vectors, 128-dim)

Workload	p50	p95
Exact scan (topK=10)	20.56ms	21.32ms
Filtered 1% (topK=10)	0.60ms	0.71ms
Threshold=240	6.79ms	7.94ms
Filtered 10% + threshold=240	0.75ms	0.95ms

Backward Compatibility

All existing VECTOR_SIMILARITY queries work unchanged. No SQL, schema, table config, or wire protocol changes. The new vectorDistanceThreshold query option is purely additive.

Test plan

• VectorBackendCapabilitiesTest — capability model for all backends (8 tests)
• VectorExecutionModeTest — mode properties and presence checks (6 tests)
• VectorBackendTypeTest — existing + new capability integration (8 tests)
• VectorQueryExecutionContextTest — mode selection logic for all query shapes (16 tests)
• VectorSearchParamsTest — threshold parsing, negative thresholds for dot-product (19 tests)
• VectorSimilarityFilterOperatorTest — filtered ANN, threshold refinement, execution mode reporting (21 tests)
• VectorCompoundQueryTest — compound patterns: filter+topK, filter+threshold, backward compat (9 tests)
• VectorTest (integration) — filtered ANN, filtered reduces results, EXPLAIN execution mode
• IvfFlatVectorTest (integration) — threshold search, compound filter+threshold
• Checkstyle, spotless, and license checks pass on all modified modules
• BenchmarkVectorPhase3 — performance benchmark for filtered and threshold workloads

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.20818% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.94%. Comparing base (4259af7) to head (bb94fe2).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...operator/filter/ExactVectorScanFilterOperator.java	31.03%	19 Missing and 1 partial ⚠️
...ava/org/apache/pinot/core/plan/FilterPlanNode.java	11.76%	13 Missing and 2 partials ⚠️
...perator/filter/VectorSimilarityFilterOperator.java	78.26%	6 Missing and 4 partials ⚠️
...e/operator/filter/VectorQueryExecutionContext.java	89.61%	4 Missing and 4 partials ⚠️
...t/segment/spi/index/creator/VectorBackendType.java	82.14%	4 Missing and 1 partial ⚠️
...e/pinot/common/utils/config/QueryOptionsUtils.java	55.55%	3 Missing and 1 partial ⚠️
...not/core/operator/filter/VectorExplainContext.java	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18114      +/-   ##
============================================
+ Coverage     63.93%   63.94%   +0.01%     
- Complexity     1594     1614      +20     
============================================
  Files          3178     3181       +3     
  Lines        193466   193717     +251     
  Branches      29880    29911      +31     
============================================
+ Hits         123683   123871     +188     
- Misses        60010    60065      +55     
- Partials       9773     9781       +8     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	63.93% <76.20%> (+0.06%)	⬆️
java-21	34.30% <0.00%> (-29.60%)	⬇️
temurin	63.94% <76.20%> (+0.01%)	⬆️
unittests	63.94% <76.20%> (+0.01%)	⬆️
unittests1	55.84% <76.20%> (+0.02%)	⬆️
unittests2	34.32% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds Phase 3 vector query execution semantics, introducing explicit execution modes, backend capability modeling, filtered ANN over-fetching, and distance-threshold (“radius”) search with explain/debug visibility.

Changes:

• Introduces VectorBackendCapabilities + wires capabilities into VectorBackendType for backend-neutral planning decisions.
• Adds VectorExecutionMode and plumbs execution-mode reporting into vector operators and explain output.
• Adds vectorDistanceThreshold query option with threshold refinement paths for ANN and exact-scan fallback, plus new unit/integration tests and a perf benchmark.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java	Adds vectorDistanceThreshold query option key constant.
pinot-common/src/main/java/org/apache/pinot/common/utils/config/QueryOptionsUtils.java	Parses/validates vectorDistanceThreshold query option.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/VectorBackendCapabilities.java	New immutable capability model for vector backends.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/VectorBackendType.java	Attaches capabilities to each backend type via getCapabilities().
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/VectorExecutionMode.java	New enum describing vector execution strategies.
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/VectorQueryExecutionContext.java	Centralized execution-mode selection + context container.
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/VectorSearchParams.java	Adds threshold parameter support to vector search params.
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/VectorExplainContext.java	Extends explain context with execution mode.
pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java	Detects vector+AND patterns to mark filtered execution semantics and pass flags into operators.
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/VectorSimilarityFilterOperator.java	Implements filtered over-fetch + threshold refinement and exposes execution mode in explain.
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ExactVectorScanFilterOperator.java	Adds threshold-based exact scan fallback and explain execution mode.
pinot-core/src/test/java/org/apache/pinot/core/operator/filter/VectorSearchParamsTest.java	Unit tests for distance-threshold parsing and defaults.
pinot-core/src/test/java/org/apache/pinot/core/operator/filter/VectorSimilarityFilterOperatorTest.java	Unit tests for filtered/threshold behavior and execution-mode reporting.
pinot-core/src/test/java/org/apache/pinot/core/operator/filter/VectorQueryExecutionContextTest.java	Tests for mode selection rules and builder behavior.
pinot-core/src/test/java/org/apache/pinot/core/operator/filter/VectorCompoundQueryTest.java	Tests compound patterns (filter+topK, filter+threshold) and mode reporting.
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/creator/VectorBackendCapabilitiesTest.java	Unit tests for capability model and backend integration.
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/creator/VectorExecutionModeTest.java	Unit tests for execution mode enum properties.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/VectorTest.java	Adds integration coverage for filtered ANN + explain output.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/IvfFlatVectorTest.java	Adds integration coverage for threshold search and filter+threshold compound query.
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkVectorPhase3.java	Adds a Phase 3 benchmark harness for filtered/threshold patterns.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

[xiangfu0]

Documentation for this PR has been created in the pinot-contrib/pinot-docs repository:

PR: pinot-contrib/pinot-docs#722

Documentation includes:

• Comprehensive guide to 8 vector query execution modes (ANN_TOP_K, ANN_TOP_K_WITH_RERANK, ANN_THEN_FILTER, ANN_THEN_FILTER_THEN_RERANK, ANN_THRESHOLD_SCAN, ANN_THRESHOLD_THEN_FILTER, EXACT_SCAN, UNSUPPORTED)
• vectorDistanceThreshold query option for distance-based filtering
• Filtered ANN patterns and how Pinot optimizes AND(VECTOR_SIMILARITY, metadata-filter) queries
• How to view execution modes in EXPLAIN output
• Example SQL queries for various vector retrieval strategies
• Performance tuning tips for different vector index backends (HNSW, IVF_FLAT, IVF_PQ)
• Compound retrieval strategies combining vector and metadata filtering

See Vector Query Execution Semantics for the full documentation.