From 2d4f0a575a609753fa5c7013bf1cf5c28990ef56 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 18 Nov 2025 14:28:03 +0100 Subject: [PATCH 01/11] Update ES|QL search tutorial for 9.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add FORK/FUSE hybrid search with RRF - Add RERANK for semantic reranking - Add COMPLETION for LLM text generation - Add KNN/TEXT_EMBEDDING for vector search - Reorganize steps for better logical flow (FTS → filtering → combining → AI) - Fix QSTR, RERANK, COMPLETION syntax - Use approach-based tabs for hybrid search (OR vs FORK/FUSE) - Add callout annotations with numbered list explanations - Update title to 'ES|QL for search tutorial' --- .../esql/esql-search-tutorial.md | 339 +++++++++++++----- 1 file changed, 253 insertions(+), 86 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 989e375757bda..4a5248a82d623 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -1,16 +1,20 @@ --- applies_to: - stack: preview 9.0, ga 9.1 - serverless: ga -navigation_title: Search and filter with ES|QL + stack: + serverless: +navigation_title: "ES|QL for search" --- -# Search and filter with {{esql}} +# {{esql}} for search tutorial -This is a hands-on introduction to the basics of full-text search and semantic search, using {{esql}}. +This hands-on tutorial covers full-text search, semantic search, hybrid search, vector search, and AI-powered search capabilities using {{esql}}. In this scenario, we're implementing search for a cooking blog. The blog contains recipes with various attributes including textual content, categorical data, and numerical ratings. +:::{note} +This tutorial uses a small dataset for learning purposes. The goal is to demonstrate search concepts and {{esql}} syntax. +::: + ## Requirements You need a running {{es}} cluster, together with {{kib}} to use the Dev Tools API Console. Refer to [choose your deployment type](docs-content://deploy-manage/deploy.md#choosing-your-deployment-type) for deployment options. @@ -114,9 +118,9 @@ PUT /cooking_blog/_mapping } ``` -1. `analyzer`: Used for text analysis. If you don't specify it, the `standard` analyzer is used by default for `text` fields. It’s included here for demonstration purposes. To know more about analyzers, refer to [Anatomy of an analyzer](docs-content://manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md). -2. `ignore_above`: Prevents indexing values longer than 256 characters in the `keyword` field. This is the default value and it’s included here for demonstration purposes. It helps to save disk space and avoid potential issues with Lucene’s term byte-length limit. For more information, refer [ignore_above parameter](/reference/elasticsearch/mapping-reference/ignore-above.md). -3. `description`: A field declared with both `text` and `keyword` [data types](/reference/elasticsearch/mapping-reference/field-data-types.md). Such fields are called [Multi-fields](/reference/elasticsearch/mapping-reference/multi-fields.md). This enables both full-text search and exact matching/filtering on the same field. If you use [dynamic mapping](docs-content://manage-data/data-store/mapping/dynamic-field-mapping.md), these multi-fields will be created automatically. Other fields in the mapping like `author`, `category`, `tags` are also declared as multi-fields. +1. The `standard` analyzer is used by default for `text` fields. It's included here explicitly for demonstration purposes. To know more about analyzers, refer to [Anatomy of an analyzer](docs-content://manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md). +2. [Multi-fields](/reference/elasticsearch/mapping-reference/multi-fields.md) enable both full-text search and exact matching/filtering on the same field. The `title` field is declared with both `text` and `keyword` data types. If you use [dynamic mapping](docs-content://manage-data/data-store/mapping/dynamic-field-mapping.md), these multi-fields will be created automatically. Other fields in the mapping like `description`, `author`, `category`, and `tags` are also declared as multi-fields. +3. The `ignore_above` parameter prevents indexing values longer than 256 characters in the `keyword` field. This is the default value and it's included here for demonstration purposes. For more information, refer to the [ignore_above parameter](/reference/elasticsearch/mapping-reference/ignore-above.md). ::::{tip} Full-text search is powered by [text analysis](docs-content://solutions/search/full-text/text-analysis-during-search.md). Text analysis normalizes and standardizes text data so it can be efficiently stored in an inverted index and searched in near real-time. Analysis happens at both [index and search time](docs-content://manage-data/data-store/text-analysis/index-search-analysis.md). This tutorial won't cover analysis in detail, but it's important to understand how text is processed to create effective search queries. @@ -220,7 +224,7 @@ This is fundamentally different from full-text search - it's a binary yes/no fil Now that you understand basic searching, explore how to control the precision of your text matches. -### Require all search terms (AND logic) +### Require all search terms (`AND` logic) By default, searches with match use OR logic between terms. To require ALL terms to match, use the function syntax with the `operator` parameter to specify AND logic: @@ -261,73 +265,9 @@ FROM cooking_blog This query only matches documents where the words "rich and creamy" appear exactly in that order in the description field. -## Step 5: Semantic search and hybrid search - -### Index semantic content - -{{es}} allows you to semantically search for documents based on the meaning of the text, rather than just the presence of specific keywords. This is useful when you want to find documents that are conceptually similar to a given query, even if they don't contain the exact search terms. - -ES|QL supports semantic search when your mappings include fields of the [`semantic_text`](/reference/elasticsearch/mapping-reference/semantic-text.md) type. This example mapping update adds a new field called `semantic_description` with the type `semantic_text`: - -```console -PUT /cooking_blog/_mapping -{ - "properties": { - "semantic_description": { - "type": "semantic_text" - } - } -} -``` +## Step 5: Multi-field search and query strings -Next, index a document with content into the new field: - -```console -POST /cooking_blog/_doc -{ - "title": "Mediterranean Quinoa Bowl", - "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.", - "author": "Jamie Oliver", - "date": "2023-06-01", - "category": "Main Course", - "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"], - "rating": 4.7 -} -``` - -### Perform semantic search - -Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Here's an example natural language query against the `semantic_description` field: - -```esql -FROM cooking_blog -| WHERE semantic_description:"What are some easy to prepare but nutritious plant-based meals?" -| LIMIT 5 -``` - -:::{tip} -If you'd like to test out the semantic search workflow against a large dataset, follow the [semantic-search-tutorial](docs-content://solutions/search/semantic-search/semantic-search-semantic-text.md). -::: - -### Perform hybrid search - -You can combine full-text and semantic queries. In this example we combine full-text and semantic search with custom weights: - -```esql -FROM cooking_blog METADATA _score -| WHERE match(semantic_description, "easy to prepare vegetarian meals", { "boost": 0.75 }) - OR match(tags, "vegetarian", { "boost": 0.25 }) -| SORT _score DESC -| LIMIT 5 -``` - -This query searches the `semantic_description` field for documents that are semantically similar to "easy to prepare vegetarian meals" with a higher weight, while also matching the `tags` field for "vegetarian" with a lower weight. The results are sorted by relevance score. - -Learn how to combine these with complex criteria in [Step 8](#step-8-complex-search-solutions). - -## Step 6: Advanced search features - -Once you're comfortable with basic search precision, use the following advanced features for powerful search capabilities. +Once you're comfortable with basic search precision, learn how to search across multiple fields and use query string syntax for complex patterns. ### Use query string for complex patterns @@ -335,7 +275,7 @@ The `QSTR` function enables powerful search patterns using a compact query langu ```esql FROM cooking_blog -| WHERE QSTR(description, "fluffy AND pancak* OR (creamy -vegan)") +| WHERE QSTR("description: (fluffy AND pancak*) OR (creamy -vegan)") | KEEP title, description | LIMIT 1000 ``` @@ -346,7 +286,7 @@ Query string syntax lets you: - Enable fuzzy matching: `pancake~1` for typo tolerance - Group terms: `(thai AND curry) OR pasta` - Search exact phrases: `"fluffy pancakes"` -- Search across fields: `QSTR("title,description", "pancake OR (creamy AND rich)")` +- Search across fields: `QSTR("(title:pancake OR description:pancake) AND creamy")` ### Search across multiple fields @@ -366,15 +306,15 @@ In many cases, matches in certain fields (like the title) might be more relevant ```esql FROM cooking_blog METADATA _score -| WHERE match(title, "vegetarian curry", {"boost": 2.0}) # Title matches are twice as important - OR match(description, "vegetarian curry") - OR match(tags, "vegetarian curry") +| WHERE MATCH(title, "vegetarian curry", {"boost": 2.0}) # Title matches are twice as important + OR MATCH(description, "vegetarian curry") + OR MATCH(tags, "vegetarian curry") | KEEP title, description, tags, _score | SORT _score DESC | LIMIT 1000 ``` -## Step 7: Filtering and exact matching +## Step 6: Filtering and exact matching Filtering allows you to narrow down your search results based on exact criteria. Unlike full-text searches, filters are binary (yes/no) and do not affect the relevance score. Filters execute faster than queries because excluded results don't need to be scored. @@ -423,9 +363,9 @@ FROM cooking_blog | LIMIT 1000 ``` -## Step 8: Complex search solutions +## Step 7: Complex search solutions -Real-world search often requires combining multiple types of criteria. This section shows how to build sophisticated search experiences. +Real-world search often requires combining multiple types of criteria. This section shows how to build sophisticated search experiences by combining the techniques you've learned. ### Combine filters with full-text search @@ -450,7 +390,7 @@ FROM cooking_blog METADATA _score | WHERE NOT category.keyword == "Dessert" | EVAL tags_concat = MV_CONCAT(tags.keyword, ",") # Convert multi-value field to string | WHERE tags_concat LIKE "*vegetarian*" AND rating >= 4.5 # Wildcard pattern matching -| WHERE match(title, "curry spicy", {"boost": 2.0}) OR match(description, "curry spicy") +| WHERE MATCH(title, "curry spicy", {"boost": 2.0}) OR MATCH(description, "curry spicy") | EVAL category_boost = CASE(category.keyword == "Main Course", 1.0, 0.0) # Conditional boost | EVAL date_boost = CASE(DATE_DIFF("month", date, NOW()) <= 1, 0.5, 0.0) # Boost recent content | EVAL custom_score = _score + category_boost + date_boost # Combine scores @@ -459,14 +399,241 @@ FROM cooking_blog METADATA _score | LIMIT 1000 ``` +## Step 8: Semantic search and hybrid search + +### Index semantic content + +{{es}} allows you to semantically search for documents based on the meaning of the text, rather than just the presence of specific keywords. This is useful when you want to find documents that are conceptually similar to a given query, even if they don't contain the exact search terms. + +ES|QL supports semantic search when your mappings include fields of the [`semantic_text`](/reference/elasticsearch/mapping-reference/semantic-text.md) type. This example mapping update adds a new field called `semantic_description` with the type `semantic_text`: + +```console +PUT /cooking_blog/_mapping +{ + "properties": { + "semantic_description": { + "type": "semantic_text" + } + } +} +``` + +Next, index a document with content into the new field: + +```console +POST /cooking_blog/_doc +{ + "title": "Mediterranean Quinoa Bowl", + "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.", + "author": "Jamie Oliver", + "date": "2023-06-01", + "category": "Main Course", + "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"], + "rating": 4.7 +} +``` + +### Perform semantic search + +Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Use the [`MATCH` function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) (or the [`:` operator](/reference/query-languages/esql/functions-operators/operators.md#esql-match-operator) shorthand) on `semantic_text` fields to perform semantic queries: + +```esql +FROM cooking_blog METADATA _score +| WHERE semantic_description:"I need plant-based meals that aren't difficult to make" +| SORT _score DESC +| LIMIT 5 +``` + +When you use `MATCH` or `:` on a `semantic_text` field, {{esql}} automatically performs a semantic search rather than a lexical search. This means the query finds documents based on semantic similarity, not just keyword matching. + +:::{tip} +If you'd like to test out the semantic search workflow against a large dataset, follow the [semantic-search-tutorial](docs-content://solutions/search/semantic-search/semantic-search-semantic-text.md). +::: + +### Perform hybrid search + +Hybrid search combines different search strategies (like lexical and semantic search) to leverage the strengths of each approach. {{esql}} provides two ways to implement hybrid search: + +::::{tab-set} + +:::{tab-item} `FORK` and `FUSE` +```{applies_to} +stack: preview 9.2 +serverless: preview +``` + +Use [`FORK`](/reference/query-languages/esql/commands/fork.md) and [`FUSE`](/reference/query-languages/esql/commands/fuse.md) to execute different search strategies in parallel, then merge and score the combined results. + +```esql +FROM cooking_blog METADATA _id, _index, _score +| FORK ( + WHERE title:"vegetarian curry" <1> + | SORT _score DESC +) ( + WHERE semantic_description:"easy vegetarian curry recipes" <2> + | SORT _score DESC +) +| FUSE <3> +| KEEP title, description, rating, _score +| SORT _score DESC +| LIMIT 5 +``` + +1. Lexical search on the title field +2. Semantic search on the semantic_description field +3. Merge results using RRF (Reciprocal Rank Fusion) algorithm +::: + +:::{tab-item} OR with boost +```{applies_to} +stack: preview 9.0 +serverless: preview +``` + +Use OR logic to combine queries with the [`MATCH` function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) and boost parameters to control relative weights: + +```esql +FROM cooking_blog METADATA _score +| WHERE MATCH(semantic_description, "easy to make vegetarian meals", { "boost": 0.75 }) + OR MATCH(tags, "vegetarian", { "boost": 0.25 }) +| SORT _score DESC +| LIMIT 5 +``` + +This query uses OR logic to combine two searches: a semantic search on `semantic_description` with 75% weight and a lexical search on `tags` with 25% weight. Documents matching either query contribute to the final score based on their boost values. +::: + +:::: + + + + +## Step 9: Advanced AI-powered search + +```{applies_to} +stack: preview 9.2 +serverless: preview +``` + +{{esql}} provides commands for AI-powered search enhancements and text generation tasks. You will need to set up an inference endpoint with appropriate models to follow along with these examples. + +### Semantic reranking with `RERANK` + +The [`RERANK` command](/reference/query-languages/esql/commands/rerank.md) re-scores search results using inference models for improved relevance. This is particularly useful for improving the ranking of initial search results by applying more sophisticated semantic understanding. + +:::{tip} +RERANK requires a configured inference endpoint with a reranking model. We recommend using [Elastic Rerank](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md). +Follow the setup steps in [Download and deploy](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md#ml-nlp-rerank-deploy-steps) if you want to follow along. +::: + +```esql +FROM cooking_blog METADATA _score +| WHERE description:"vegetarian recipes" <1> +| SORT _score DESC +| LIMIT 100 <2> +| RERANK "healthy quick meals" ON description WITH { "inference_id": "my_reranker_model" } <3> +| LIMIT 5 <4> +| KEEP title, description, _score +``` + +1. Perform initial lexical search +2. Limit to top 100 results for reranking +3. Re-score using reranking model +4. Return top 5 results after reranking + +### Text generation with `COMPLETION` + +:::{tip} +COMPLETION requires a configured inference endpoint with an LLM. Refer to the [Inference API documentation](docs-content://explore-analyze/elastic-inference/inference-api.md) for setup instructions. +::: + +The [`COMPLETION` command](/reference/query-languages/esql/commands/completion.md) sends prompts to a Large Language Model (LLM) for text generation tasks like question answering, summarization, or translation. + +```esql +FROM cooking_blog +| WHERE category.keyword == "Dessert" +| LIMIT 3 +| EVAL prompt = CONCAT("Summarize this recipe: ", title, " - ", description) +| COMPLETION summary = prompt WITH { "inference_id": "my_llm_endpoint" } +| KEEP title, summary +``` + +This enables you to: +- Generate summaries of search results +- Answer questions about document content +- Translate or transform text using LLMs +- Create dynamic content based on your data + +### Vector search with KNN and TEXT_EMBEDDING + +:::{note} +This subsection is for advanced users who want explicit control over vector search. For most use cases, the `semantic_text` field type covered in [Step 8](#step-8-semantic-search-and-hybrid-search) is recommended as it handles embeddings automatically. +::: + +The [`KNN` function](/reference/query-languages/esql/functions-operators/dense-vector-functions.md#esql-knn) finds the k nearest vectors to a query vector through approximate search on indexed `dense_vector` fields. + +#### `KNN` with pre-computed vectors + +First, add a `dense_vector` field and index a document with a pre-computed vector. This example uses a toy example for readability: the vector has only 3 dimensions. + +```console +PUT /cooking_blog/_mapping +{ + "properties": { + "recipe_vector": { + "type": "dense_vector", + "dims": 3 + } + } +} + +POST /cooking_blog/_doc/6 +{ + "title": "Quick Vegan Stir-Fry", + "description": "A fast and healthy vegan stir-fry with tofu and vegetables", + "recipe_vector": [0.5, 0.8, 0.3], + "rating": 4.6 +} +``` + +Then search using a query vector: + +```esql +FROM cooking_blog METADATA _score +| WHERE KNN(recipe_vector, [0.5, 0.8, 0.3]) +| SORT _score DESC +| LIMIT 5 +``` + +#### `KNN` with `TEXT_EMBEDDING` + +```{applies_to} +stack: preview 9.3 +serverless: preview +``` + +The [`TEXT_EMBEDDING` function](/reference/query-languages/esql/functions-operators/dense-vector-functions.md#esql-text_embedding) generates dense vector embeddings from text input at query time using an inference model. + +:::{tip} +You'll need to set up an inference endpoint with an embedding model to use this feature. Refer to the [Inference API documentation](docs-content://explore-analyze/elastic-inference/inference-api.md) for setup instructions. +::: + +```esql +FROM cooking_blog METADATA _score +| WHERE KNN(recipe_vector, TEXT_EMBEDDING("vegan recipe", "my_embedding_model")) +| SORT _score DESC +| LIMIT 5 +``` + +This approach gives you full control over vector embeddings, unlike `semantic_text` which handles embeddings automatically. + ## Learn more ### Documentation This tutorial introduced the basics of search and filtering in {{esql}}. Building a real-world search experience requires understanding many more advanced concepts and techniques. Here are some resources once you're ready to dive deeper: -- [Search with {{esql}}](docs-content://solutions/search/esql-for-search.md): Learn about all the search capabilities in ES|QL, refer to Using ES|QL for search. {{esql}}. -- [{{esql}} search functions](/reference/query-languages/esql/functions-operators/search-functions.md): Explore the full list of search functions available in {{esql}}. +- [Search with {{esql}}](docs-content://solutions/search/esql-for-search.md): Learn about all the search capabilities in {{esql}}. - [Semantic search](docs-content://solutions/search/semantic-search.md): Understand your various options for semantic search in Elasticsearch. - [The `semantic_text` workflow](docs-content://solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. From ac64b8d6e05b5dd89859a34ae0e5b41ef0470cea Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 18 Nov 2025 16:18:20 +0100 Subject: [PATCH 02/11] Update learn more section --- docs/reference/query-languages/esql/esql-search-tutorial.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 4a5248a82d623..30e077319b41f 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -631,11 +631,13 @@ This approach gives you full control over vector embeddings, unlike `semantic_te ### Documentation -This tutorial introduced the basics of search and filtering in {{esql}}. Building a real-world search experience requires understanding many more advanced concepts and techniques. Here are some resources once you're ready to dive deeper: +This tutorial introduced search, filtering, and AI-powered capabilities in {{esql}}. Here are some resources once you're ready to dive deeper: -- [Search with {{esql}}](docs-content://solutions/search/esql-for-search.md): Learn about all the search capabilities in {{esql}}. +- [Search with {{esql}}](docs-content://solutions/search/esql-for-search.md): Learn about all search capabilities in {{esql}}. - [Semantic search](docs-content://solutions/search/semantic-search.md): Understand your various options for semantic search in Elasticsearch. - [The `semantic_text` workflow](docs-content://solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. +- [Inference API](docs-content://explore-analyze/elastic-inference/inference-api.md): Set up inference endpoints for the `RERANK`, `COMPLETION`, and `TEXT_EMBEDDING` commands. +- [Text analysis](docs-content://manage-data/data-store/text-analysis/index.md): Understand how text analysis works for full-text search. ### Related blog posts From b8f3c05134edbef49e7eb8f89b1c8e3ee98660d5 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 18 Nov 2025 16:35:50 +0100 Subject: [PATCH 03/11] remove link --- docs/reference/query-languages/esql/esql-search-tutorial.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 30e077319b41f..659f109623475 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -637,7 +637,6 @@ This tutorial introduced search, filtering, and AI-powered capabilities in {{esq - [Semantic search](docs-content://solutions/search/semantic-search.md): Understand your various options for semantic search in Elasticsearch. - [The `semantic_text` workflow](docs-content://solutions/search/semantic-search.md#_semantic_text_workflow): Learn how to use the `semantic_text` field type for semantic search. This is the recommended approach for most users looking to perform semantic search in {{es}}, because it abstracts away the complexity of setting up inference endpoints and models. - [Inference API](docs-content://explore-analyze/elastic-inference/inference-api.md): Set up inference endpoints for the `RERANK`, `COMPLETION`, and `TEXT_EMBEDDING` commands. -- [Text analysis](docs-content://manage-data/data-store/text-analysis/index.md): Understand how text analysis works for full-text search. ### Related blog posts From 5c8b636cd4f2209eef15dda274c85b4c05916131 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:02:57 +0100 Subject: [PATCH 04/11] Remove keyword multi-field from description and clarify text vs keyword usage --- .../query-languages/esql/esql-search-tutorial.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 659f109623475..29b9aee6665e5 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -76,12 +76,7 @@ PUT /cooking_blog/_mapping } }, "description": { - "type": "text", - "fields": { - "keyword": { - "type": "keyword" - } - } + "type": "text" <1> }, "author": { "type": "text", @@ -119,8 +114,9 @@ PUT /cooking_blog/_mapping ``` 1. The `standard` analyzer is used by default for `text` fields. It's included here explicitly for demonstration purposes. To know more about analyzers, refer to [Anatomy of an analyzer](docs-content://manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md). -2. [Multi-fields](/reference/elasticsearch/mapping-reference/multi-fields.md) enable both full-text search and exact matching/filtering on the same field. The `title` field is declared with both `text` and `keyword` data types. If you use [dynamic mapping](docs-content://manage-data/data-store/mapping/dynamic-field-mapping.md), these multi-fields will be created automatically. Other fields in the mapping like `description`, `author`, `category`, and `tags` are also declared as multi-fields. +2. [Multi-fields](/reference/elasticsearch/mapping-reference/multi-fields.md) enable both full-text search and exact matching/filtering on the same field. The `title`, `author`, `category`, and `tags` fields are declared with both `text` and `keyword` data types. The `text` version allows full-text search, while the `keyword` version enables exact filtering. If you use [dynamic mapping](docs-content://manage-data/data-store/mapping/dynamic-field-mapping.md), these multi-fields will be created automatically. 3. The `ignore_above` parameter prevents indexing values longer than 256 characters in the `keyword` field. This is the default value and it's included here for demonstration purposes. For more information, refer to the [ignore_above parameter](/reference/elasticsearch/mapping-reference/ignore-above.md). +4. `keyword` fields store exact values and are used for filtering, sorting, and aggregations. Unlike `text` fields, they are not analyzed. Use `keyword` when you need exact matching (e.g., filtering by category or finding a specific author name), but use `text` when you want to search within longer content like descriptions. ::::{tip} Full-text search is powered by [text analysis](docs-content://solutions/search/full-text/text-analysis-during-search.md). Text analysis normalizes and standardizes text data so it can be efficiently stored in an inverted index and searched in near real-time. Analysis happens at both [index and search time](docs-content://manage-data/data-store/text-analysis/index-search-analysis.md). This tutorial won't cover analysis in detail, but it's important to understand how text is processed to create effective search queries. From bda415ef8930669bceb4d2b26e2512e6091dc4a2 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:04:35 +0100 Subject: [PATCH 05/11] wrap match operator in quotes in prose for clarity --- docs/reference/query-languages/esql/esql-search-tutorial.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 29b9aee6665e5..06f3d84bde825 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -147,7 +147,7 @@ Full-text search involves executing text-based queries across one or more docume {{esql}} provides multiple functions for full-text search, including `MATCH`, `MATCH_PHRASE`, and `QSTR`. For basic text matching, you can use either: 1. Full [match function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) syntax: `match(field, "search terms")` -2. Compact syntax using the [match operator `:`](/reference/query-languages/esql/functions-operators/operators.md#esql-match-operator): `field:"search terms"` +2. Compact syntax using the [match operator "`:`"](/reference/query-languages/esql/functions-operators/operators.md#esql-match-operator): `field:"search terms"` Both are equivalent for basic matching and can be used interchangeably. The compact syntax is more concise, while the function syntax allows for more configuration options. We use the compact syntax in most examples for brevity. @@ -431,7 +431,7 @@ POST /cooking_blog/_doc ### Perform semantic search -Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Use the [`MATCH` function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) (or the [`:` operator](/reference/query-languages/esql/functions-operators/operators.md#esql-match-operator) shorthand) on `semantic_text` fields to perform semantic queries: +Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Use the [`MATCH` function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) (or the ["`:`" operator](/reference/query-languages/esql/functions-operators/operators.md#esql-match-operator) shorthand) on `semantic_text` fields to perform semantic queries: ```esql FROM cooking_blog METADATA _score @@ -440,7 +440,7 @@ FROM cooking_blog METADATA _score | LIMIT 5 ``` -When you use `MATCH` or `:` on a `semantic_text` field, {{esql}} automatically performs a semantic search rather than a lexical search. This means the query finds documents based on semantic similarity, not just keyword matching. +When you use `MATCH` or "`:`" on a `semantic_text` field, {{esql}} automatically performs a semantic search rather than a lexical search. This means the query finds documents based on semantic similarity, not just keyword matching. :::{tip} If you'd like to test out the semantic search workflow against a large dataset, follow the [semantic-search-tutorial](docs-content://solutions/search/semantic-search/semantic-search-semantic-text.md). From be4fd18a858631f37b507e5a48adb1b0e48cc32f Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:06:56 +0100 Subject: [PATCH 06/11] remove OR boost tab now that we have fork/fuse --- .../esql/esql-search-tutorial.md | 32 +------------------ 1 file changed, 1 insertion(+), 31 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 06f3d84bde825..bd6dbc730a0db 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -448,17 +448,12 @@ If you'd like to test out the semantic search workflow against a large dataset, ### Perform hybrid search -Hybrid search combines different search strategies (like lexical and semantic search) to leverage the strengths of each approach. {{esql}} provides two ways to implement hybrid search: - -::::{tab-set} - -:::{tab-item} `FORK` and `FUSE` ```{applies_to} stack: preview 9.2 serverless: preview ``` -Use [`FORK`](/reference/query-languages/esql/commands/fork.md) and [`FUSE`](/reference/query-languages/esql/commands/fuse.md) to execute different search strategies in parallel, then merge and score the combined results. +Hybrid search combines different search strategies (like lexical and semantic search) to leverage the strengths of each approach. Use [`FORK`](/reference/query-languages/esql/commands/fork.md) and [`FUSE`](/reference/query-languages/esql/commands/fuse.md) to execute different search strategies in parallel, then merge and score the combined results. ```esql FROM cooking_blog METADATA _id, _index, _score @@ -478,31 +473,6 @@ FROM cooking_blog METADATA _id, _index, _score 1. Lexical search on the title field 2. Semantic search on the semantic_description field 3. Merge results using RRF (Reciprocal Rank Fusion) algorithm -::: - -:::{tab-item} OR with boost -```{applies_to} -stack: preview 9.0 -serverless: preview -``` - -Use OR logic to combine queries with the [`MATCH` function](/reference/query-languages/esql/functions-operators/search-functions.md#esql-match) and boost parameters to control relative weights: - -```esql -FROM cooking_blog METADATA _score -| WHERE MATCH(semantic_description, "easy to make vegetarian meals", { "boost": 0.75 }) - OR MATCH(tags, "vegetarian", { "boost": 0.25 }) -| SORT _score DESC -| LIMIT 5 -``` - -This query uses OR logic to combine two searches: a semantic search on `semantic_description` with 75% weight and a lexical search on `tags` with 25% weight. Documents matching either query contribute to the final score based on their boost values. -::: - -:::: - - - ## Step 9: Advanced AI-powered search From 5bc91fc605924d94344ea1c5eace370b86e261ff Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:07:27 +0100 Subject: [PATCH 07/11] add explicit limits to FORKs --- docs/reference/query-languages/esql/esql-search-tutorial.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index bd6dbc730a0db..f30847bea15b8 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -460,9 +460,11 @@ FROM cooking_blog METADATA _id, _index, _score | FORK ( WHERE title:"vegetarian curry" <1> | SORT _score DESC + | LIMIT 5 ) ( WHERE semantic_description:"easy vegetarian curry recipes" <2> | SORT _score DESC + | LIMIT 5 ) | FUSE <3> | KEEP title, description, rating, _score From 9f319e5baf8038d61d2e3597492deab4484507f1 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:23:43 +0100 Subject: [PATCH 08/11] add note about using same embedding model for queries and docs --- docs/reference/query-languages/esql/esql-search-tutorial.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index f30847bea15b8..d3e77dd7a02df 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -595,6 +595,10 @@ FROM cooking_blog METADATA _score This approach gives you full control over vector embeddings, unlike `semantic_text` which handles embeddings automatically. +:::{note} +The vectors in your index must be generated by the same embedding model you specify in `TEXT_EMBEDDING`, otherwise the similarity calculations will be meaningless and you won't get relevant results. +::: + ## Learn more ### Documentation From 97498e756be94df0d46f11483e4e2927557277f1 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 16:44:38 +0100 Subject: [PATCH 09/11] provide rrf and linear combo examples in tabs --- .../esql/esql-search-tutorial.md | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index d3e77dd7a02df..ee60e2017f0fa 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -455,6 +455,14 @@ serverless: preview Hybrid search combines different search strategies (like lexical and semantic search) to leverage the strengths of each approach. Use [`FORK`](/reference/query-languages/esql/commands/fork.md) and [`FUSE`](/reference/query-languages/esql/commands/fuse.md) to execute different search strategies in parallel, then merge and score the combined results. +`FUSE` supports different merging algorithms, namely Reciprocal Rank Fusion (RRF) and linear combination. + +::::{tab-set} + +:::{tab-item} RRF (default) + +RRF is a ranking algorithm that combines rankings from multiple search strategies without needing to tune weights. It's effective when you want to blend different search approaches without manual score tuning. + ```esql FROM cooking_blog METADATA _id, _index, _score | FORK ( @@ -474,7 +482,36 @@ FROM cooking_blog METADATA _id, _index, _score 1. Lexical search on the title field 2. Semantic search on the semantic_description field -3. Merge results using RRF (Reciprocal Rank Fusion) algorithm +3. Merge results using RRF algorithm (default) +::: + +:::{tab-item} Linear combination + +Linear combination allows you to specify explicit weights for each search strategy. This gives you fine-grained control over how much each strategy contributes to the final score. + +```esql +FROM cooking_blog METADATA _id, _index, _score +| FORK ( + WHERE title:"vegetarian curry" <1> + | SORT _score DESC + | LIMIT 5 +) ( + WHERE semantic_description:"easy vegetarian curry recipes" <2> + | SORT _score DESC + | LIMIT 5 +) +| FUSE {"method": "linear", "weights": [0.7, 0.3]} <3> +| KEEP title, description, rating, _score +| SORT _score DESC +| LIMIT 5 +``` + +1. Lexical search on the title field (70% weight) +2. Semantic search on the semantic_description field (30% weight) +3. Merge results using linear combination with explicit weights +::: + +:::: ## Step 9: Advanced AI-powered search From b4f2a94b6f51f3076b808409bc470648b2e3633d Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Tue, 25 Nov 2025 17:10:31 +0100 Subject: [PATCH 10/11] use rerank default and rejig section accordingly --- .../query-languages/esql/esql-search-tutorial.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index ee60e2017f0fa..1d66f6c3b4899 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -527,8 +527,7 @@ serverless: preview The [`RERANK` command](/reference/query-languages/esql/commands/rerank.md) re-scores search results using inference models for improved relevance. This is particularly useful for improving the ranking of initial search results by applying more sophisticated semantic understanding. :::{tip} -RERANK requires a configured inference endpoint with a reranking model. We recommend using [Elastic Rerank](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md). -Follow the setup steps in [Download and deploy](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md#ml-nlp-rerank-deploy-steps) if you want to follow along. +`RERANK` requires an inference endpoint configured for the `rerank` task. Refer to the [setup instructions](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md#ml-nlp-rerank-deploy). ::: ```esql @@ -536,16 +535,19 @@ FROM cooking_blog METADATA _score | WHERE description:"vegetarian recipes" <1> | SORT _score DESC | LIMIT 100 <2> -| RERANK "healthy quick meals" ON description WITH { "inference_id": "my_reranker_model" } <3> +| RERANK "healthy quick meals" ON description <3> | LIMIT 5 <4> | KEEP title, description, _score ``` 1. Perform initial lexical search 2. Limit to top 100 results for reranking -3. Re-score using reranking model +3. Re-score using the default reranking model ([Elastic Rerank](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-rerank.md)) 4. Return top 5 results after reranking +You can configure `RERANK` to use a specific reranking model. See the [`RERANK` command documentation](/reference/query-languages/esql/commands/rerank.md) for configuration options. + + ### Text generation with `COMPLETION` :::{tip} From fb5d383ea52808b94bff5539f4b1a20499b6467e Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Mon, 1 Dec 2025 10:25:13 +0100 Subject: [PATCH 11/11] fix FUSE linear syntax --- docs/reference/query-languages/esql/esql-search-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/query-languages/esql/esql-search-tutorial.md b/docs/reference/query-languages/esql/esql-search-tutorial.md index 1d66f6c3b4899..6dc6cce11bc36 100644 --- a/docs/reference/query-languages/esql/esql-search-tutorial.md +++ b/docs/reference/query-languages/esql/esql-search-tutorial.md @@ -500,7 +500,7 @@ FROM cooking_blog METADATA _id, _index, _score | SORT _score DESC | LIMIT 5 ) -| FUSE {"method": "linear", "weights": [0.7, 0.3]} <3> +| FUSE LINEAR WITH { "weights": { "fork1": 0.7, "fork2": 0.3 } } <3> | KEEP title, description, rating, _score | SORT _score DESC | LIMIT 5