From 01fc38b65200c2bfd17334ae7f35ffb75dbdabf5 Mon Sep 17 00:00:00 2001 From: "redisdocsapp[bot]" <177626021+redisdocsapp[bot]@users.noreply.github.com> Date: Wed, 26 Nov 2025 00:24:30 +0000 Subject: [PATCH] Update for redisvl 0.12.1 --- content/develop/ai/redisvl/api/query.md | 274 +++++++++++++++++- content/develop/ai/redisvl/api/schema.md | 44 ++- .../develop/ai/redisvl/user_guide/_index.md | 1 + .../ai/redisvl/user_guide/advanced_queries.md | 95 ++++++ .../ai/redisvl/user_guide/getting_started.md | 15 + .../ai/redisvl/user_guide/hybrid_queries.md | 16 + .../ai/redisvl/user_guide/svs_vamana.md | 87 +++++- 7 files changed, 510 insertions(+), 22 deletions(-) diff --git a/content/develop/ai/redisvl/api/query.md b/content/develop/ai/redisvl/api/query.md index 55002adf73..0160330f2f 100644 --- a/content/develop/ai/redisvl/api/query.md +++ b/content/develop/ai/redisvl/api/query.md @@ -12,7 +12,7 @@ queries for different use cases. Each query class wraps the `redis-py` Query mod ## VectorQuery -### `class VectorQuery(vector, vector_field_name, return_fields=None, filter_expression=None, dtype='float32', num_results=10, return_score=True, dialect=2, sort_by=None, in_order=False, hybrid_policy=None, batch_size=None, ef_runtime=None, normalize_vector_distance=False)` +### `class VectorQuery(vector, vector_field_name, return_fields=None, filter_expression=None, dtype='float32', num_results=10, return_score=True, dialect=2, sort_by=None, in_order=False, hybrid_policy=None, batch_size=None, ef_runtime=None, epsilon=None, search_window_size=None, use_search_history=None, search_buffer_capacity=None, normalize_vector_distance=False)` Bases: `BaseVectorQuery`, `BaseQuery` @@ -57,6 +57,22 @@ expression. * **ef_runtime** (*Optional* *[* *int* *]*) – Controls the size of the dynamic candidate list for HNSW algorithm at query time. Higher values improve recall at the expense of slower search performance. Defaults to None, which uses the index-defined value. + * **epsilon** (*Optional* *[* *float* *]*) – The range search approximation factor for HNSW and SVS-VAMANA + indexes. Sets boundaries for candidates within radius \* (1 + epsilon). Higher values + allow more extensive search and more accurate results at the expense of run time. + Defaults to None, which uses the index-defined value (typically 0.01). + * **search_window_size** (*Optional* *[* *int* *]*) – The size of the search window for SVS-VAMANA KNN searches. + Increasing this value generally yields more accurate but slower search results. + Defaults to None, which uses the index-defined value (typically 10). + * **use_search_history** (*Optional* *[* *str* *]*) – For SVS-VAMANA indexes, controls whether to use the + search buffer or entire search history. Options are "OFF", "ON", or "AUTO". + "AUTO" is always evaluated internally as "ON". Using the entire history may yield + a slightly better graph at the cost of more search time. + Defaults to None, which uses the index-defined value (typically "AUTO"). + * **search_buffer_capacity** (*Optional* *[* *int* *]*) – Tuning parameter for SVS-VAMANA indexes using + two-level compression (LVQx or LeanVec types). Determines the number of vector + candidates to collect in the first level of search before the re-ranking level. + Defaults to None, which uses the index-defined value (typically SEARCH_WINDOW_SIZE). * **normalize_vector_distance** (*bool*) – Redis supports 3 distance metrics: L2 (euclidean), IP (inner product), and COSINE. By default, L2 distance returns an unbounded value. COSINE distance returns a value between 0 and 2. IP returns a value determined by @@ -215,6 +231,19 @@ Set the EF_RUNTIME parameter for the query. * **TypeError** – If ef_runtime is not an integer * **ValueError** – If ef_runtime is not positive +#### `set_epsilon(epsilon)` + +Set the epsilon parameter for the query. + +* **Parameters:** + **epsilon** (*float*) – The range search approximation factor for HNSW and SVS-VAMANA + indexes. Sets boundaries for candidates within radius \* (1 + epsilon). + Higher values allow more extensive search and more accurate results at the + expense of run time. +* **Raises:** + * **TypeError** – If epsilon is not a float or int + * **ValueError** – If epsilon is negative + #### `set_filter(filter_expression=None)` Set the filter expression for the query. @@ -235,6 +264,40 @@ Set the hybrid policy for the query. * **Raises:** **ValueError** – If hybrid_policy is not one of the valid options +#### `set_search_buffer_capacity(search_buffer_capacity)` + +Set the SEARCH_BUFFER_CAPACITY parameter for the query. + +* **Parameters:** + **search_buffer_capacity** (*int*) – Tuning parameter for SVS-VAMANA indexes using + two-level compression. Determines the number of vector candidates to collect + in the first level of search before the re-ranking level. +* **Raises:** + * **TypeError** – If search_buffer_capacity is not an integer + * **ValueError** – If search_buffer_capacity is not positive + +#### `set_search_window_size(search_window_size)` + +Set the SEARCH_WINDOW_SIZE parameter for the query. + +* **Parameters:** + **search_window_size** (*int*) – The size of the search window for SVS-VAMANA KNN searches. + Increasing this value generally yields more accurate but slower search results. +* **Raises:** + * **TypeError** – If search_window_size is not an integer + * **ValueError** – If search_window_size is not positive + +#### `set_use_search_history(use_search_history)` + +Set the USE_SEARCH_HISTORY parameter for the query. + +* **Parameters:** + **use_search_history** (*str*) – For SVS-VAMANA indexes, controls whether to use the + search buffer or entire search history. Options are "OFF", "ON", or "AUTO". +* **Raises:** + * **TypeError** – If use_search_history is not a string + * **ValueError** – If use_search_history is not one of "OFF", "ON", or "AUTO" + #### `slop(slop)` Allow a maximum of N intervening non matched terms between @@ -331,6 +394,15 @@ Return the EF_RUNTIME parameter for the query. * **Return type:** Optional[int] +#### `property epsilon: float | None` + +Return the epsilon parameter for the query. + +* **Returns:** + The epsilon value for the query. +* **Return type:** + Optional[float] + #### `property filter: str | `[`FilterExpression`]({{< relref "filter/#filterexpression" >}})` ` The filter expression for the query. @@ -357,9 +429,77 @@ Return the parameters for the query. Return self as the query object. +#### `property search_buffer_capacity: int | None` + +Return the SEARCH_BUFFER_CAPACITY parameter for the query. + +* **Returns:** + The SEARCH_BUFFER_CAPACITY value for the query. +* **Return type:** + Optional[int] + +#### `property search_window_size: int | None` + +Return the SEARCH_WINDOW_SIZE parameter for the query. + +* **Returns:** + The SEARCH_WINDOW_SIZE value for the query. +* **Return type:** + Optional[int] + +#### `property use_search_history: str | None` + +Return the USE_SEARCH_HISTORY parameter for the query. + +* **Returns:** + The USE_SEARCH_HISTORY value for the query. +* **Return type:** + Optional[str] + +#### `NOTE` +**Runtime Parameters for Performance Tuning** + +VectorQuery supports runtime parameters for HNSW and SVS-VAMANA indexes that can be adjusted at query time without rebuilding the index: + +**HNSW Parameters:** + +- `ef_runtime`: Controls search accuracy (higher = better recall, slower search) + +**SVS-VAMANA Parameters:** + +- `search_window_size`: Size of search window for KNN searches +- `use_search_history`: Whether to use search buffer (OFF/ON/AUTO) +- `search_buffer_capacity`: Tuning parameter for 2-level compression + +Example with HNSW runtime parameters: + +```python +from redisvl.query import VectorQuery + +query = VectorQuery( + vector=[0.1, 0.2, 0.3], + vector_field_name="embedding", + num_results=10, + ef_runtime=150 # Higher for better recall +) +``` + +Example with SVS-VAMANA runtime parameters: + +```python +query = VectorQuery( + vector=[0.1, 0.2, 0.3], + vector_field_name="embedding", + num_results=10, + search_window_size=20, + use_search_history='ON', + search_buffer_capacity=30 +) +``` + ## VectorRangeQuery -### `class VectorRangeQuery(vector, vector_field_name, return_fields=None, filter_expression=None, dtype='float32', distance_threshold=0.2, epsilon=None, num_results=10, return_score=True, dialect=2, sort_by=None, in_order=False, hybrid_policy=None, batch_size=None, normalize_vector_distance=False)` +### `class VectorRangeQuery(vector, vector_field_name, return_fields=None, filter_expression=None, dtype='float32', distance_threshold=0.2, epsilon=None, search_window_size=None, use_search_history=None, search_buffer_capacity=None, num_results=10, return_score=True, dialect=2, sort_by=None, in_order=False, hybrid_policy=None, batch_size=None, normalize_vector_distance=False)` Bases: `BaseVectorQuery`, `BaseQuery` @@ -384,6 +524,18 @@ distance threshold. This controls how extensive the search is beyond the specified radius. Higher values increase recall at the expense of performance. Defaults to None, which uses the index-defined epsilon (typically 0.01). + * **search_window_size** (*Optional* *[* *int* *]*) – The size of the search window for SVS-VAMANA range searches. + Increasing this value generally yields more accurate but slower search results. + Defaults to None, which uses the index-defined value (typically 10). + * **use_search_history** (*Optional* *[* *str* *]*) – For SVS-VAMANA indexes, controls whether to use the + search buffer or entire search history. Options are "OFF", "ON", or "AUTO". + "AUTO" is always evaluated internally as "ON". Using the entire history may yield + a slightly better graph at the cost of more search time. + Defaults to None, which uses the index-defined value (typically "AUTO"). + * **search_buffer_capacity** (*Optional* *[* *int* *]*) – Tuning parameter for SVS-VAMANA indexes using + two-level compression (LVQx or LeanVec types). Determines the number of vector + candidates to collect in the first level of search before the re-ranking level. + Defaults to None, which uses the index-defined value (typically SEARCH_WINDOW_SIZE). * **num_results** (*int*) – The MAX number of results to return. Defaults to 10. * **return_score** (*bool* *,* *optional*) – Whether to return the vector @@ -597,6 +749,38 @@ Set the hybrid policy for the query. * **Raises:** **ValueError** – If hybrid_policy is not one of the valid options +#### `set_search_buffer_capacity(search_buffer_capacity)` + +Set the SEARCH_BUFFER_CAPACITY parameter for the range query. + +* **Parameters:** + **search_buffer_capacity** (*int*) – Tuning parameter for SVS-VAMANA indexes using + two-level compression. +* **Raises:** + * **TypeError** – If search_buffer_capacity is not an integer + * **ValueError** – If search_buffer_capacity is not positive + +#### `set_search_window_size(search_window_size)` + +Set the SEARCH_WINDOW_SIZE parameter for the range query. + +* **Parameters:** + **search_window_size** (*int*) – The size of the search window for SVS-VAMANA range searches. +* **Raises:** + * **TypeError** – If search_window_size is not an integer + * **ValueError** – If search_window_size is not positive + +#### `set_use_search_history(use_search_history)` + +Set the USE_SEARCH_HISTORY parameter for the range query. + +* **Parameters:** + **use_search_history** (*str*) – Controls whether to use the search buffer or entire history. + Must be one of "OFF", "ON", or "AUTO". +* **Raises:** + * **TypeError** – If use_search_history is not a string + * **ValueError** – If use_search_history is not one of the valid options + #### `slop(slop)` Allow a maximum of N intervening non matched terms between @@ -728,6 +912,63 @@ Return the parameters for the query. Return self as the query object. +#### `property search_buffer_capacity: int | None` + +Return the SEARCH_BUFFER_CAPACITY parameter for the query. + +* **Returns:** + The SEARCH_BUFFER_CAPACITY value for the query. +* **Return type:** + Optional[int] + +#### `property search_window_size: int | None` + +Return the SEARCH_WINDOW_SIZE parameter for the query. + +* **Returns:** + The SEARCH_WINDOW_SIZE value for the query. +* **Return type:** + Optional[int] + +#### `property use_search_history: str | None` + +Return the USE_SEARCH_HISTORY parameter for the query. + +* **Returns:** + The USE_SEARCH_HISTORY value for the query. +* **Return type:** + Optional[str] + +#### `NOTE` +**Runtime Parameters for Range Queries** + +VectorRangeQuery supports runtime parameters for controlling range search behavior: + +**HNSW & SVS-VAMANA Parameters:** + +- `epsilon`: Range search approximation factor (default: 0.01) + +**SVS-VAMANA Parameters:** + +- `search_window_size`: Size of search window +- `use_search_history`: Whether to use search buffer (OFF/ON/AUTO) +- `search_buffer_capacity`: Tuning parameter for 2-level compression + +Example: + +```python +from redisvl.query import VectorRangeQuery + +query = VectorRangeQuery( + vector=[0.1, 0.2, 0.3], + vector_field_name="embedding", + distance_threshold=0.3, + epsilon=0.05, # Approximation factor + search_window_size=20, # SVS-VAMANA only + use_search_history='AUTO' # SVS-VAMANA only +) +``` + ## HybridQuery ### `class HybridQuery(*args, **kwargs)` @@ -773,6 +1014,12 @@ Instantiates a AggregateHybridQuery object. * **text_weights** (*Optional* *[* *Dict* *[* *str* *,* *float* *]* *]*) – The importance weighting of individual words within the query text. Defaults to None, as no modifications will be made to the text_scorer score. + +#### `NOTE` +AggregateHybridQuery uses FT.AGGREGATE commands which do NOT support runtime +parameters. For runtime parameter support (ef_runtime, search_window_size, etc.), +use VectorQuery or VectorRangeQuery which use FT.SEARCH commands. + * **Raises:** * **ValueError** – If the text string is empty, or if the text string becomes empty after stopwords are removed. @@ -985,6 +1232,29 @@ The `stopwords` parameter in [HybridQuery](#hybridquery) (and `AggregateHybridQu For index-level stopwords configuration (server-side), see `redisvl.schema.IndexInfo.stopwords`. Using query-time stopwords with index-level `STOPWORDS 0` is counterproductive. +#### `NOTE` +**Runtime Parameters for Hybrid Queries** + +**Important:** AggregateHybridQuery uses FT.AGGREGATE commands which do NOT support runtime parameters. +Runtime parameters (`ef_runtime`, `search_window_size`, `use_search_history`, `search_buffer_capacity`) +are only supported with FT.SEARCH commands. + +For runtime parameter support, use [VectorQuery](#vectorquery) or [VectorRangeQuery](#vectorrangequery) instead of AggregateHybridQuery. + +Example with VectorQuery (supports runtime parameters): + +```python +from redisvl.query import VectorQuery + +query = VectorQuery( + vector=[0.1, 0.2, 0.3], + vector_field_name="embedding", + return_fields=["description"], + num_results=10, + ef_runtime=150 # Runtime parameters work with VectorQuery +) +``` + ## TextQuery ### `class TextQuery(text, text_field_name, text_scorer='BM25STD', filter_expression=None, return_fields=None, num_results=10, return_score=True, dialect=2, sort_by=None, in_order=False, params=None, stopwords='english', text_weights=None)` diff --git a/content/develop/ai/redisvl/api/schema.md b/content/develop/ai/redisvl/api/schema.md index ac0ea1ce91..692d285022 100644 --- a/content/develop/ai/redisvl/api/schema.md +++ b/content/develop/ai/redisvl/api/schema.md @@ -748,11 +748,16 @@ HNSW (Hierarchical Navigable Small World) - Graph-based approximate search with **Performance characteristics:** -- **Search speed**: Very fast approximate search with tunable accuracy +- **Search speed**: Very fast approximate search with tunable accuracy (via `ef_runtime` at query time) - **Memory usage**: Higher than compressed SVS-VAMANA but reasonable for most applications -- **Recall quality**: Excellent recall rates (95-99%), often better than other approximate methods +- **Recall quality**: Excellent recall rates (95-99%), tunable via `ef_runtime` parameter - **Build time**: Moderate construction time, faster than SVS-VAMANA for smaller datasets +**Runtime parameters** (adjustable at query time without rebuilding index): + +- `ef_runtime`: Controls search accuracy (higher = better recall, slower search). Default: 10 +- `epsilon`: Range search approximation factor for VectorRangeQuery. Default: 0.01 + ### `class HNSWVectorField(*, name, type='vector', path=None, attrs)` Bases: `BaseField` @@ -866,10 +871,10 @@ Configuration for the model, should be a dictionary conforming to [ConfigDict][p dims: 768 distance_metric: cosine datatype: float32 - # Balanced settings for good recall and performance - m: 16 - ef_construction: 200 - ef_runtime: 10 + # Index-time parameters (set during index creation) + m: 16 # Graph connectivity + ef_construction: 200 # Build-time accuracy + # Note: ef_runtime can be set at query time via VectorQuery ``` **High-recall configuration:** @@ -882,10 +887,10 @@ Configuration for the model, should be a dictionary conforming to [ConfigDict][p dims: 768 distance_metric: cosine datatype: float32 - # Tuned for maximum accuracy + # Index-time parameters tuned for maximum accuracy m: 32 ef_construction: 400 - ef_runtime: 50 + # Note: ef_runtime=50 can be set at query time for higher recall ``` ### `SVS-VAMANA Vector Fields` @@ -908,6 +913,13 @@ SVS-VAMANA (Scalable Vector Search with VAMANA graph algorithm) provides fast ap : - **vs FLAT**: Much faster search, significantly lower memory usage with compression, but approximate results - **vs HNSW**: Better memory efficiency with compression, similar or better recall, Intel-optimized +**Runtime parameters** (adjustable at query time without rebuilding index): + +- `epsilon`: Range search approximation factor. Default: 0.01 +- `search_window_size`: Size of search window for KNN searches. Higher = better recall, slower search +- `use_search_history`: Whether to use search buffer (OFF/ON/AUTO). Default: AUTO +- `search_buffer_capacity`: Tuning parameter for 2-level compression. Default: search_window_size + **Compression selection guide:** - **No compression**: Best performance, standard memory usage @@ -1053,10 +1065,10 @@ Dimensionality reduction for LeanVec types (must be < dims) dims: 768 distance_metric: cosine datatype: float32 - # Standard settings for balanced performance + # Index-time parameters (set during index creation) graph_max_degree: 40 construction_window_size: 250 - search_window_size: 20 + # Note: search_window_size and other runtime params can be set at query time ``` **High-performance configuration with compression:** @@ -1069,14 +1081,14 @@ Dimensionality reduction for LeanVec types (must be < dims) dims: 768 distance_metric: cosine datatype: float32 - # Tuned for better recall + # Index-time parameters tuned for better recall graph_max_degree: 64 construction_window_size: 500 - search_window_size: 40 # Maximum compression with dimensionality reduction compression: LeanVec4x8 reduce: 384 # 50% dimensionality reduction training_threshold: 1000 + # Note: search_window_size=40 can be set at query time for higher recall ``` **Important Notes:** @@ -1085,7 +1097,7 @@ Dimensionality reduction for LeanVec types (must be < dims) - **Datatype limitations**: SVS-VAMANA only supports float16 and float32 datatypes (not bfloat16 or float64). - **Compression compatibility**: The reduce parameter is only valid with LeanVec compression types (LeanVec4x8 or LeanVec8x8). - **Platform considerations**: Intel’s proprietary LVQ and LeanVec optimizations are not available in Redis Open Source. On non-Intel platforms and Redis Open Source, SVS-VAMANA with compression falls back to basic 8-bit scalar quantization. -- **Performance tip**: Start with default parameters and tune search_window_size first for your speed vs accuracy requirements. +- **Performance tip**: Runtime parameters like `search_window_size`, `epsilon`, and `use_search_history` can be adjusted at query time without rebuilding the index. Start with defaults and tune `search_window_size` first for your speed vs accuracy requirements. ### `FLAT Vector Fields` @@ -1439,15 +1451,15 @@ This section provides detailed guidance for choosing between vector search algor **Recall Quality:** : - FLAT: 100% (exact search) - - HNSW: 95-99% (tunable via ef_runtime) - - SVS-VAMANA: 90-95% (depends on compression) + - HNSW: 95-99% (tunable via `ef_runtime` at query time) + - SVS-VAMANA: 90-95% (tunable via `search_window_size` at query time, also depends on compression) ### `Migration Considerations` **From FLAT to HNSW:** : - Straightforward migration - Expect slight recall reduction but major speed improvement - - Tune ef_runtime to balance speed vs accuracy + - Tune `ef_runtime` at query time to balance speed vs accuracy (no index rebuild needed) **From HNSW to SVS-VAMANA:** : - Requires Redis >= 8.2 with RediSearch >= 2.8.10 diff --git a/content/develop/ai/redisvl/user_guide/_index.md b/content/develop/ai/redisvl/user_guide/_index.md index 8ac9b40668..b34697faee 100644 --- a/content/develop/ai/redisvl/user_guide/_index.md +++ b/content/develop/ai/redisvl/user_guide/_index.md @@ -69,6 +69,7 @@ User guides provide helpful resources for using RedisVL and its different compon * [Creating an SVS-VAMANA Index](svs_vamana/#creating-an-svs-vamana-index) * [Loading Sample Data](svs_vamana/#loading-sample-data) * [Performing Vector Searches](svs_vamana/#performing-vector-searches) + * [Runtime Parameters for Performance Tuning](svs_vamana/#runtime-parameters-for-performance-tuning) * [Understanding Compression Types](svs_vamana/#understanding-compression-types) * [Compression Types Explained](svs_vamana/#compression-types-explained) * [Hybrid Queries with SVS-VAMANA](svs_vamana/#hybrid-queries-with-svs-vamana) diff --git a/content/develop/ai/redisvl/user_guide/advanced_queries.md b/content/develop/ai/redisvl/user_guide/advanced_queries.md index 6a60fb6810..8ee2d9b7fc 100644 --- a/content/develop/ai/redisvl/user_guide/advanced_queries.md +++ b/content/develop/ai/redisvl/user_guide/advanced_queries.md @@ -157,6 +157,10 @@ keys = index.load(data) print(f"Loaded {len(keys)} products into the index") ``` + 16:27:50 redisvl.index.index INFO Index already exists, overwriting. + Loaded 6 products into the index + + ## 1. TextQuery: Full Text Search The `TextQuery` class enables full text search with advanced scoring algorithms. It's ideal for keyword-based search with relevance ranking. @@ -181,6 +185,10 @@ results = index.query(text_query) result_print(results) ``` + +
scoreproduct_idbrief_descriptioncategoryprice
4.080705480646511prod_1comfortable running shoes for athletesfootwear89.99
1.4504838715161907prod_5basketball shoes with excellent ankle supportfootwear139.99
1.431980178975859prod_2lightweight running jacket with water resistanceouterwear129.99
+ + ### Text Search with Different Scoring Algorithms RedisVL supports multiple text scoring algorithms. Let's compare `BM25STD` and `TFIDF`: @@ -201,6 +209,13 @@ results = index.query(bm25_query) result_print(results) ``` + Results with BM25 scoring: + + + +
scoreproduct_idbrief_descriptionprice
4.165936382048982prod_1comfortable running shoes for athletes89.99
1.769051138581863prod_4yoga mat with extra cushioning for comfort39.99
1.2306902673750557prod_5basketball shoes with excellent ankle support139.99
+ + ```python # TFIDF scoring @@ -217,6 +232,13 @@ results = index.query(tfidf_query) result_print(results) ``` + Results with TFIDF scoring: + + + +
scoreproduct_idbrief_descriptionprice
1.3333333333333333prod_1comfortable running shoes for athletes89.99
1.3333333333333333prod_1comfortable running shoes for athletes89.99
1.0prod_5basketball shoes with excellent ankle support139.99
+ + ### Text Search with Filters Combine text search with filters to narrow results: @@ -239,6 +261,10 @@ result_print(results) ``` +
scoreproduct_idbrief_descriptioncategoryprice
2.385806908729779prod_1comfortable running shoes for athletesfootwear89.99
2.385806908729779prod_1comfortable running shoes for athletesfootwear89.99
1.9340948871093797prod_5basketball shoes with excellent ankle supportfootwear139.99
+ + + ```python # Search for products under $100 price_filtered_query = TextQuery( @@ -253,6 +279,10 @@ results = index.query(price_filtered_query) result_print(results) ``` + +
scoreproduct_idbrief_descriptionprice
2.2775029612659465prod_1comfortable running shoes for athletes89.99
1.1387514806329733prod_1comfortable running shoes for athletes89.99
1.1190633543347508prod_4yoga mat with extra cushioning for comfort39.99
1.1190633543347508prod_4yoga mat with extra cushioning for comfort39.99
+ + ### Text Search with Multiple Fields and Weights You can search across multiple text fields with different weights to prioritize certain fields. @@ -271,6 +301,10 @@ results = index.query(weighted_query) result_print(results) ``` + +
scoreproduct_idbrief_description
3.040323653363804prod_1comfortable running shoes for athletes
3.040323653363804prod_1comfortable running shoes for athletes
1.289396591406253prod_5basketball shoes with excellent ankle support
+ + ### Text Search with Custom Stopwords Stopwords are common words that are filtered out before processing the query. You can specify which language's default stopwords should be filtered out, like `english`, `french`, or `german`. You can also define your own list of stopwords: @@ -291,6 +325,10 @@ result_print(results) ``` +
scoreproduct_idbrief_description
4.1444591833267275prod_1comfortable running shoes for athletes
4.1444591833267275prod_1comfortable running shoes for athletes
1.4875097606385526prod_5basketball shoes with excellent ankle support
+ + + ```python # Use custom stopwords custom_stopwords_query = TextQuery( @@ -306,6 +344,10 @@ result_print(results) ``` +
scoreproduct_idbrief_description
2.5107799078325prod_1comfortable running shoes for athletes
2.5107799078325prod_1comfortable running shoes for athletes
2.482820220115406prod_3professional tennis racket for competitive players
+ + + ```python # No stopwords no_stopwords_query = TextQuery( @@ -320,6 +362,10 @@ results = index.query(no_stopwords_query) result_print(results) ``` + +
scoreproduct_idbrief_description
3.69730364515632prod_1comfortable running shoes for athletes
3.69730364515632prod_1comfortable running shoes for athletes
1.5329921800414583prod_5basketball shoes with excellent ankle support
+ + ## 2. AggregateHybridQuery: Combining Text and Vector Search The `AggregateHybridQuery` combines text search and vector similarity to provide the best of both worlds: @@ -378,6 +424,9 @@ company_index.create(overwrite=True, drop=True) print(f"Index created with STOPWORDS 0: {company_index}") ``` + Index created with STOPWORDS 0: + + ```python # Load sample data with company names containing common stopwords @@ -395,6 +444,9 @@ for i, company in enumerate(companies): print(f"✓ Loaded {len(companies)} companies") ``` + ✓ Loaded 5 companies + + ```python # Search for "Bank of Glasberliner" - with STOPWORDS 0, "of" is indexed and searchable @@ -412,6 +464,10 @@ for doc in results.docs: print(f" - {doc.company_name}: {doc.description}") ``` + Found 1 results for 'Bank of Glasberliner': + - Bank of Glasberliner: Major financial institution + + **Comparison: With vs Without Stopwords** If we had used the default stopwords (not specifying `stopwords` in the schema), the word "of" would be filtered out during indexing. This means: @@ -442,6 +498,9 @@ custom_stopwords_schema = { print("Custom stopwords:", custom_stopwords_schema["index"]["stopwords"]) ``` + Custom stopwords: ['inc', 'llc', 'corp'] + + **YAML Format:** You can also define stopwords in YAML schema files: @@ -479,6 +538,9 @@ company_index.delete(drop=True) print("✓ Cleaned up company_index") ``` + ✓ Cleaned up company_index + + ### Basic Aggregate Hybrid Query Let's search for "running" with both text and semantic search: @@ -501,6 +563,10 @@ results = index.query(hybrid_query) result_print(results) ``` + +
vector_distanceproduct_idbrief_descriptioncategorypricevector_similaritytext_scorehybrid_score
5.96046447754e-08prod_1comfortable running shoes for athletesfootwear89.990.9999999701984.829774426092.14893230697
5.96046447754e-08prod_1comfortable running shoes for athletesfootwear89.990.9999999701984.829774426092.14893230697
5.96046447754e-08prod_1comfortable running shoes for athletesfootwear89.990.9999999701984.829774426092.14893230697
0.0038834810257prod_4yoga mat with extra cushioning for comfortaccessories39.990.99805825948700.698640781641
0.0038834810257prod_4yoga mat with extra cushioning for comfortaccessories39.990.99805825948700.698640781641
+ + ### Adjusting the Alpha Parameter The `alpha` parameter controls the weight between vector and text search: @@ -526,6 +592,13 @@ results = index.query(vector_heavy_query) result_print(results) ``` + Results with alpha=0.9 (vector-heavy): + + + +
vector_distanceproduct_idbrief_descriptionvector_similaritytext_scorehybrid_score
-1.19209289551e-07prod_4yoga mat with extra cushioning for comfort1.00000005961.538380705411.05383812419
-1.19209289551e-07prod_4yoga mat with extra cushioning for comfort1.00000005961.538380705411.05383812419
-1.19209289551e-07prod_4yoga mat with extra cushioning for comfort1.00000005961.538380705411.05383812419
+ + ### Aggregate Hybrid Query with Filters You can also combine hybrid search with filters: @@ -547,6 +620,10 @@ results = index.query(filtered_hybrid_query) result_print(results) ``` + +
vector_distanceproduct_idbrief_descriptioncategorypricevector_similaritytext_scorehybrid_score
-1.19209289551e-07prod_3professional tennis racket for competitive playersequipment199.991.00000005961.547237055061.16417115824
-1.19209289551e-07prod_3professional tennis racket for competitive playersequipment199.991.00000005961.547237055061.16417115824
-1.19209289551e-07prod_3professional tennis racket for competitive playersequipment199.991.00000005961.547237055061.16417115824
0.411657452583prod_2lightweight running jacket with water resistanceouterwear129.990.79417127370800.555919891596
0.411657452583prod_2lightweight running jacket with water resistanceouterwear129.990.79417127370800.555919891596
+ + ### Using Different Text Scorers AggregateHybridQuery supports the same text scoring algorithms as TextQuery: @@ -568,6 +645,24 @@ results = index.query(hybrid_tfidf) result_print(results) ``` + +
vector_distanceproduct_idbrief_descriptionvector_similaritytext_scorehybrid_score
0prod_5basketball shoes with excellent ankle support131.6
0prod_2lightweight running jacket with water resistance100.7
0prod_2lightweight running jacket with water resistance100.7
+ + +### Runtime Parameters for Vector Search Tuning + +**Important:** `AggregateHybridQuery` uses FT.AGGREGATE commands which do NOT support runtime parameters. + +Runtime parameters (such as `ef_runtime` for HNSW indexes or `search_window_size` for SVS-VAMANA indexes) are only supported with FT.SEARCH commands. + +**For runtime parameter support, use `VectorQuery` or `VectorRangeQuery` instead:** + +- `VectorQuery`: Supports all runtime parameters (HNSW and SVS-VAMANA) +- `VectorRangeQuery`: Supports all runtime parameters (HNSW and SVS-VAMANA) +- `AggregateHybridQuery`: Does NOT support runtime parameters (uses FT.AGGREGATE) + +See the **Runtime Parameters** section earlier in this notebook for examples of using runtime parameters with `VectorQuery`. + ## 3. MultiVectorQuery: Multi-Vector Search The `MultiVectorQuery` allows you to search over multiple vector fields simultaneously. This is useful when you have different types of embeddings (e.g., text and image embeddings) and want to find results that match across multiple modalities. diff --git a/content/develop/ai/redisvl/user_guide/getting_started.md b/content/develop/ai/redisvl/user_guide/getting_started.md index ac6bb7aa8f..210ca4b71d 100644 --- a/content/develop/ai/redisvl/user_guide/getting_started.md +++ b/content/develop/ai/redisvl/user_guide/getting_started.md @@ -292,6 +292,21 @@ query = VectorQuery( ) ``` +**Note:** For HNSW and SVS-VAMANA indexes, you can tune search performance using runtime parameters: + +```python +# Example with HNSW runtime parameters +query = VectorQuery( + vector=[0.1, 0.1, 0.5], + vector_field_name="user_embedding", + return_fields=["user", "age", "job"], + num_results=3, + ef_runtime=50 # Higher for better recall (HNSW only) +) +``` + +See the [SVS-VAMANA guide](09_svs_vamana.ipynb) and [Advanced Queries guide](11_advanced_queries.ipynb) for more details on runtime parameters. + ### Executing queries With our `VectorQuery` object defined above, we can execute the query over the `SearchIndex` using the `query` method. diff --git a/content/develop/ai/redisvl/user_guide/hybrid_queries.md b/content/develop/ai/redisvl/user_guide/hybrid_queries.md index 5fccf7dfd3..e1c1729fd5 100644 --- a/content/develop/ai/redisvl/user_guide/hybrid_queries.md +++ b/content/develop/ai/redisvl/user_guide/hybrid_queries.md @@ -139,6 +139,22 @@ result_print(index.query(v))
vector_distanceusercredit_scoreagejoboffice_locationlast_updated
0derricklow14doctor-122.4194,37.77491741627789
0.217882037163taimurlow15CEO-122.0839,37.38611742232589
0.653301358223joemedium35dentist-122.0839,37.38611742232589
+**Performance Tip:** For HNSW and SVS-VAMANA indexes, you can add runtime parameters to tune search performance: + +```python +# Example with runtime parameters for better recall +v = VectorQuery( + vector=[0.1, 0.1, 0.5], + vector_field_name="user_embedding", + return_fields=["user", "credit_score", "age"], + filter_expression=t, + ef_runtime=100, # HNSW: higher for better recall + search_window_size=40 # SVS-VAMANA: larger window for better recall +) +``` + +These parameters can be adjusted at query time without rebuilding the index. See the [Advanced Queries guide](11_advanced_queries.ipynb) for more details. + ```python # use multiple tags as a list diff --git a/content/develop/ai/redisvl/user_guide/svs_vamana.md b/content/develop/ai/redisvl/user_guide/svs_vamana.md index 34c4266f9e..57f96bdaee 100644 --- a/content/develop/ai/redisvl/user_guide/svs_vamana.md +++ b/content/develop/ai/redisvl/user_guide/svs_vamana.md @@ -265,6 +265,76 @@ for i, result in enumerate(results, 1): ================================================== +## Runtime Parameters for Performance Tuning + +SVS-VAMANA supports runtime parameters that can be adjusted at query time without rebuilding the index. These parameters allow you to fine-tune the trade-off between search speed and accuracy. + +**Available Runtime Parameters:** + +- **`search_window_size`**: Controls the size of the search window during KNN search (higher = better recall, slower search) +- **`epsilon`**: Approximation factor for range queries (default: 0.01) +- **`use_search_history`**: Whether to use search buffer (OFF/ON/AUTO, default: AUTO) +- **`search_buffer_capacity`**: Tuning parameter for 2-level compression (default: search_window_size) + +Let's see how these parameters affect search performance: + + +```python +# Example 1: Basic query with default parameters +basic_query = VectorQuery( + vector=query_vector.tolist(), + vector_field_name="embedding", + return_fields=["content", "category"], + num_results=5 +) + +print("🔍 Basic Query (default parameters):") +results = index.query(basic_query) +print(f"Found {len(results)} results\n") + +# Example 2: Query with tuned runtime parameters for higher recall +tuned_query = VectorQuery( + vector=query_vector.tolist(), + vector_field_name="embedding", + return_fields=["content", "category"], + num_results=5, + search_window_size=40, # Larger window for better recall + use_search_history='ON', # Use search history + search_buffer_capacity=50 # Larger buffer capacity +) + +print("🎯 Tuned Query (higher recall parameters):") +results = index.query(tuned_query) +print(f"Found {len(results)} results") +print("\nNote: Higher search_window_size improves recall but may increase latency") +``` + +### Runtime Parameters with Range Queries + +Runtime parameters are also useful for range queries, where you want to find all vectors within a certain distance threshold: + + +```python +from redisvl.query import VectorRangeQuery + +# Range query with runtime parameters +range_query = VectorRangeQuery( + vector=query_vector.tolist(), + vector_field_name="embedding", + return_fields=["content", "category"], + distance_threshold=0.3, + epsilon=0.05, # Approximation factor + search_window_size=30, # Search window size + use_search_history='AUTO' # Automatic history management +) + +results = index.query(range_query) +print(f"🎯 Range Query Results: Found {len(results)} vectors within distance threshold 0.3") +for i, result in enumerate(results[:3], 1): + distance = result.get('vector_distance', 'N/A') + print(f"{i}. {result['content'][:50]}... (distance: {distance})") +``` + ## Understanding Compression Types SVS-VAMANA supports different compression algorithms that trade off between memory usage and search quality. Let's explore the available options. @@ -525,16 +595,25 @@ print(f"\nEstimated memory savings: {manual_savings}%") - **Applications** that can tolerate slight recall trade-offs for speed and memory savings ### Parameter Tuning Guidelines -- **Start with CompressionAdvisor** recommendations -- **Increase search_window_size** if you need higher recall + +**Index-time parameters** (set during index creation): +- **Start with CompressionAdvisor** recommendations for compression and datatype - **Use LeanVec** for high-dimensional vectors (≥1024 dims) - **Use LVQ** for lower-dimensional vectors (<1024 dims) +- **graph_max_degree**: Higher values improve recall but increase memory usage +- **construction_window_size**: Higher values improve index quality but slow down build time + +**Runtime parameters** (adjustable at query time without rebuilding index): +- **search_window_size**: Start with 20, increase to 40-100 for higher recall +- **epsilon**: Use 0.01-0.05 for range queries (higher = faster but less accurate) +- **use_search_history**: Use 'AUTO' (default) or 'ON' for better recall +- **search_buffer_capacity**: Usually set equal to search_window_size ### Performance Considerations - **Index build time** increases with higher construction_window_size -- **Search latency** increases with higher search_window_size +- **Search latency** increases with higher search_window_size (tunable at query time!) - **Memory usage** decreases with more aggressive compression -- **Recall quality** may decrease with more aggressive compression +- **Recall quality** may decrease with more aggressive compression or lower search_window_size ## Cleanup