From 328e21878ad5ba476cfbf49a425415f005b6c5ef Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 12 Feb 2025 10:51:36 +0000 Subject: [PATCH 1/3] DOC-4835 Python vector embedding example --- content/develop/clients/redis-py/vecsearch.md | 223 ++++++++++++++++++ 1 file changed, 223 insertions(+) create mode 100644 content/develop/clients/redis-py/vecsearch.md diff --git a/content/develop/clients/redis-py/vecsearch.md b/content/develop/clients/redis-py/vecsearch.md new file mode 100644 index 0000000000..05cf576954 --- /dev/null +++ b/content/develop/clients/redis-py/vecsearch.md @@ -0,0 +1,223 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Store and query vector embeddings with Redis +linkTitle: Vector embeddings +title: Vector embeddings +weight: 5 +--- + +[Redis Query Engine]({{< relref "/develop/interact/search-and-query" >}}) +lets you index vector fields in [hash]({{< relref "/develop/data-types/hashes" >}}) +or [JSON]({{< relref "/develop/data-types/json" >}}) objects (see the +[Vectors]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors" >}}) +reference page for more information). +Among other things, vector fields can store *text embeddings*, which are AI-generated vector +representations of the semantic information in pieces of text. The +[vector distance]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +between two embeddings indicates how similar they are semantically. By comparing the +similarity of an embedding generated from some query text with embeddings stored in hash +or JSON fields, Redis can retrieve documents that closely match the query in terms +of their meaning. + +The [RedisVL]({{< relref "/develop/clients/redis-vl" >}}) library provides a +high-level Python API to help you use vector embeddings with Redis Query +Engine. The example below shows how RedisVL can retrieve data +that has a similar meaning to the query text you supply. + +## Initialize + +Start by importing the required classes: + +```python +from redisvl.utils.vectorize.text.huggingface import ( + HFTextVectorizer +) +from redisvl.index import SearchIndex +from redisvl.query import VectorQuery +``` + +The first of these imports is the +[`HFTextVectorizer`](https://docs.redisvl.com/en/stable/api/vectorizer.html#hftextvectorizer) +class, which generates +an embedding from a section sample of text. Here, we create an instance of `HFTextVectorizer` +that uses the +[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) +model for the embeddings. This model generates vectors with 384 dimensions, regardless +of the length of the input text, but note that the input is truncated to 256 +tokens (see +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6). +at the [Hugging Face](https://huggingface.co/) docs to learn more about how tokens +are related to the original text). + +```python +hf = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2") +``` + +## Create the index + +RedisVL's +[`SearchIndex`](https://docs.redisvl.com/en/stable/api/searchindex.html#searchindex-api) +class provides the +[`from_dict()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.from_dict) +method, which lets you specify your index schema with a Python dictionary, as shown +below. Another option is +[`from_yaml()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.from_yaml), +which loads the index schema from a [YAML](https://en.wikipedia.org/wiki/YAML) file. + +The schema in the example below specifies hash objects for storage and includes +three fields: the text content to index, a +[tag]({{< relref "/develop/interact/search-and-query/advanced-concepts/tags" >}}) +field to represent the "genre" of the text, and the embedding vector generated from +the original text content. The attributes of the vector field (specified in the +`attrs` object) specify [HNSW]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#hnsw-index" >}}) indexing, the +[L2]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +vector distance metric, `Float32` values to represent the vector's components, +and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model. + +```python +index = SearchIndex.from_dict({ + "index": { + "name": "vector_idx", + "prefix": "doc", + "storage_type": "hash", + }, + "fields": [ + {"name": "content", "type": "text"}, + {"name": "genre", "type": "tag"}, + { + "name": "embedding", + "type": "vector", + "attrs": { + "algorithm": "HNSW", + "dims": 384, + "distance_metric": "l2", + "datatype": "float32", + }, + }, + ], +}) +``` + +When you have created the `SearchIndex` object, you must connect to the Redis +server using the +[`connect()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.connect) +method and then use the +[`create()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.create) +method to actually create the index in the database. The `overwrite` parameter for `create()` +ensures that the index you create replaces any existing index with +the same name. The `drop` parameter deletes any objects that were +indexed by the index you are replacing. + +```python +index.connect("redis://localhost:6379") +index.create(overwrite=True, drop=True) +``` + +## Add data + +You can now supply the data objects, which will be indexed automatically +when you add them using the +[`load()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.load) +method. Use the +[`embed()`](https://docs.redisvl.com/en/stable/api/vectorizer.html#redisvl.utils.vectorize.text.huggingface.HFTextVectorizer.embed) +method of the `HFTextVectorizer` instance to create the embedding that represents the `content` field. By default, `embed()` returns a list of `float` values to represent the embedding +vector, but you can also supply the `as_buffer` parameter to encode this list as a +binary string. Use the string representation when you are indexing hash objects +(as we are here), but use the default list of `float` for JSON objects. + +```python +hf = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2") + +data = [ + { + "content": "That is a very happy person", + "genre": "persons", + "embedding": hf.embed("That is a very happy person", as_buffer=True) + }, + { + "content": "That is a happy dog", + "genre": "pets", + "embedding": hf.embed("That is a happy dog", as_buffer=True) + }, + { + "content": "Today is a sunny day", + "genre": "weather", + "embedding": hf.embed("Today is a sunny day", as_buffer=True) + } +] + +index.load(data) +``` + +## Run a query + +After you have created the index, you are ready to run a query against it. +To do this, you must create another embedding vector from your chosen query +text. Redis calculates the similarity between the query vector and each +embedding vector in the index as it runs the query. It then ranks the +results in order of this numeric similarity value. + +Create a +[`VectorQuery`](https://docs.redisvl.com/en/stable/api/query.html#vectorquery) +instance by supplying the embedding vector for your +query text, along with the hash or JSON field to match against and the number +of results you want, as shown in the example below. Then, call the +[`query()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.query) +method of `SearchIndex` with your `VectorQuery` instance to run the query. + +```python +query = VectorQuery( + vector=hf.embed("That is a happy person"), + vector_field_name="embedding", + return_fields=["content"], + num_results=3, +) + +results = index.query(query) +print(results) +``` + +The code is now ready to run, but note that it may take a while to complete when +you run it for the first time (which happens because RedisVL must download the +`all-MiniLM-L6-v2` model data before it can +generate the embeddings). When you run the code, it outputs the following results +as a list of dict objects: + +```Python +[ + { + 'id': 'doc:cf3c28e7fdd44753af3080a9a7f8d8e9', + 'vector_distance': '0.114169985056', + 'content': 'That is a very happy person' + }, + { + 'id': 'doc:614508f297b644d0be47dde5373bb059', + 'vector_distance': '0.610845386982', + 'content': 'That is a happy dog' + }, + { + 'id': 'doc:930a7dfca0d74808baee490d747c9534', + 'vector_distance': '1.48624813557', + 'content': 'Today is a sunny day' + } +] +``` + +Note that the results are ordered according to the value of the `vector_distance` +field, with the lowest distance indicating the greatest similarity to the query. +As you would expect, the text *"That is a very happy person"* is the result that is +most similar in meaning to the query text *"That is a happy person"*. + +## Learn more + +See the [RedisVL documentation](https://docs.redisvl.com/en/stable/index.html) +for more details about its features and examples of how to use them. From 4b786edc771827dab2d105a2a92bb6f9b8eafa65 Mon Sep 17 00:00:00 2001 From: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> Date: Fri, 14 Feb 2025 13:48:43 +0000 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: David Dougherty --- content/develop/clients/redis-py/vecsearch.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/develop/clients/redis-py/vecsearch.md b/content/develop/clients/redis-py/vecsearch.md index 05cf576954..a2494ef6c1 100644 --- a/content/develop/clients/redis-py/vecsearch.md +++ b/content/develop/clients/redis-py/vecsearch.md @@ -54,7 +54,7 @@ that uses the model for the embeddings. This model generates vectors with 384 dimensions, regardless of the length of the input text, but note that the input is truncated to 256 tokens (see -[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6). +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) at the [Hugging Face](https://huggingface.co/) docs to learn more about how tokens are related to the original text). @@ -99,8 +99,8 @@ index = SearchIndex.from_dict({ "attrs": { "algorithm": "HNSW", "dims": 384, - "distance_metric": "l2", - "datatype": "float32", + "distance_metric": "L2", + "datatype": "FLOAT32", }, }, ], @@ -112,7 +112,7 @@ server using the [`connect()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.connect) method and then use the [`create()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.create) -method to actually create the index in the database. The `overwrite` parameter for `create()` +method to create the index in the database. The `overwrite` parameter for `create()` ensures that the index you create replaces any existing index with the same name. The `drop` parameter deletes any objects that were indexed by the index you are replacing. From b024e875a8dc049908a95a6c1eb8dad1345ecad1 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 14 Feb 2025 15:37:29 +0000 Subject: [PATCH 3/3] DOC-4835 updated text and examples for redis-py --- content/develop/clients/redis-py/queryjson.md | 2 +- content/develop/clients/redis-py/transpipe.md | 2 +- content/develop/clients/redis-py/vecsearch.md | 271 +++++++++--------- 3 files changed, 143 insertions(+), 132 deletions(-) diff --git a/content/develop/clients/redis-py/queryjson.md b/content/develop/clients/redis-py/queryjson.md index 189dd5e3a3..4cb28eb354 100644 --- a/content/develop/clients/redis-py/queryjson.md +++ b/content/develop/clients/redis-py/queryjson.md @@ -12,7 +12,7 @@ categories: description: Learn how to use the Redis query engine with JSON linkTitle: Index and query JSON title: Example - Index and query JSON documents -weight: 2 +weight: 3 --- This example shows how to create a diff --git a/content/develop/clients/redis-py/transpipe.md b/content/develop/clients/redis-py/transpipe.md index 139c143516..6a2916c88c 100644 --- a/content/develop/clients/redis-py/transpipe.md +++ b/content/develop/clients/redis-py/transpipe.md @@ -12,7 +12,7 @@ categories: description: Learn how to use Redis pipelines and transactions linkTitle: Pipelines/transactions title: Pipelines and transactions -weight: 2 +weight: 5 --- Redis lets you send a sequence of commands to the server together in a batch. diff --git a/content/develop/clients/redis-py/vecsearch.md b/content/develop/clients/redis-py/vecsearch.md index a2494ef6c1..be4bacd341 100644 --- a/content/develop/clients/redis-py/vecsearch.md +++ b/content/develop/clients/redis-py/vecsearch.md @@ -9,10 +9,10 @@ categories: - oss - kubernetes - clients -description: Store and query vector embeddings with Redis -linkTitle: Vector embeddings -title: Vector embeddings -weight: 5 +description: Learn how to index and query vector embeddings with Redis +linkTitle: Index and query vectors +title: Index and query vectors +weight: 4 --- [Redis Query Engine]({{< relref "/develop/interact/search-and-query" >}}) @@ -28,196 +28,207 @@ similarity of an embedding generated from some query text with embeddings stored or JSON fields, Redis can retrieve documents that closely match the query in terms of their meaning. -The [RedisVL]({{< relref "/develop/clients/redis-vl" >}}) library provides a -high-level Python API to help you use vector embeddings with Redis Query -Engine. The example below shows how RedisVL can retrieve data -that has a similar meaning to the query text you supply. +In the example below, we use the +[`sentence-transformers`](https://pypi.org/project/sentence-transformers/) +library to generate vector embeddings to store and index with +Redis Query Engine. ## Initialize -Start by importing the required classes: +Install [`redis-py`]({{< relref "/develop/clients/redis-py" >}}) if you +have not already done so. Also, install `sentence-transformers` with the +following command: + +```bash +pip install sentence-transformers +``` + +In a new Python source file, start by importing the required classes: ```python -from redisvl.utils.vectorize.text.huggingface import ( - HFTextVectorizer -) -from redisvl.index import SearchIndex -from redisvl.query import VectorQuery +from sentence_transformers import SentenceTransformer +from redis.commands.search.query import Query +from redis.commands.search.field import TextField, TagField, VectorField +from redis.commands.search.indexDefinition import IndexDefinition, IndexType + +import numpy as np +import redis ``` The first of these imports is the -[`HFTextVectorizer`](https://docs.redisvl.com/en/stable/api/vectorizer.html#hftextvectorizer) -class, which generates -an embedding from a section sample of text. Here, we create an instance of `HFTextVectorizer` -that uses the +`SentenceTransformer` class, which generates an embedding from a section of text. +Here, we create an instance of `SentenceTransformer` that uses the [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model for the embeddings. This model generates vectors with 384 dimensions, regardless of the length of the input text, but note that the input is truncated to 256 tokens (see [Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) -at the [Hugging Face](https://huggingface.co/) docs to learn more about how tokens +at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens are related to the original text). ```python -hf = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2") +model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") ``` ## Create the index -RedisVL's -[`SearchIndex`](https://docs.redisvl.com/en/stable/api/searchindex.html#searchindex-api) -class provides the -[`from_dict()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.from_dict) -method, which lets you specify your index schema with a Python dictionary, as shown -below. Another option is -[`from_yaml()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.from_yaml), -which loads the index schema from a [YAML](https://en.wikipedia.org/wiki/YAML) file. +Connect to Redis and delete any index previously created with the +name `vector_idx`. (The `dropindex()` call throws an exception if +the index doesn't already exist, which is why you need the +`try: except:` block.) + +```python +r = redis.Redis(decode_responses=True) +try: + r.ft("vector_idx").dropindex(True) +except redis.exceptions.ResponseError: + pass +``` + +Next, we create the index. The schema in the example below specifies hash objects for storage and includes three fields: the text content to index, a [tag]({{< relref "/develop/interact/search-and-query/advanced-concepts/tags" >}}) field to represent the "genre" of the text, and the embedding vector generated from -the original text content. The attributes of the vector field (specified in the -`attrs` object) specify [HNSW]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#hnsw-index" >}}) indexing, the +the original text content. The `embedding` field specifies +[HNSW]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#hnsw-index" >}}) +indexing, the [L2]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) vector distance metric, `Float32` values to represent the vector's components, and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model. ```python -index = SearchIndex.from_dict({ - "index": { - "name": "vector_idx", - "prefix": "doc", - "storage_type": "hash", - }, - "fields": [ - {"name": "content", "type": "text"}, - {"name": "genre", "type": "tag"}, - { - "name": "embedding", - "type": "vector", - "attrs": { - "algorithm": "HNSW", - "dims": 384, - "distance_metric": "L2", - "datatype": "FLOAT32", - }, - }, - ], -}) -``` - -When you have created the `SearchIndex` object, you must connect to the Redis -server using the -[`connect()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.connect) -method and then use the -[`create()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.create) -method to create the index in the database. The `overwrite` parameter for `create()` -ensures that the index you create replaces any existing index with -the same name. The `drop` parameter deletes any objects that were -indexed by the index you are replacing. +schema = ( + TextField("content"), + TagField("genre"), + VectorField("embedding", "HNSW", { + "TYPE": "FLOAT32", + "DIM": 384, + "DISTANCE_METRIC":"L2" + }) +) -```python -index.connect("redis://localhost:6379") -index.create(overwrite=True, drop=True) +r.ft("vector_idx").create_index( + schema, + definition=IndexDefinition( + prefix=["doc:"], index_type=IndexType.HASH + ) +) ``` ## Add data You can now supply the data objects, which will be indexed automatically -when you add them using the -[`load()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.load) -method. Use the -[`embed()`](https://docs.redisvl.com/en/stable/api/vectorizer.html#redisvl.utils.vectorize.text.huggingface.HFTextVectorizer.embed) -method of the `HFTextVectorizer` instance to create the embedding that represents the `content` field. By default, `embed()` returns a list of `float` values to represent the embedding -vector, but you can also supply the `as_buffer` parameter to encode this list as a -binary string. Use the string representation when you are indexing hash objects +when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as +you use the `doc:` prefix specified in the index definition. + +Use the `model.encode()` method of `SentenceTransformer` +as shown below to create the embedding that represents the `content` field. +The `astype()` option that follows the `model.encode()` call specifies that +we want a vector of `float32` values. The `tobytes()` option encodes the +vector components together as a single binary string rather than the +default Python list of `float` values. +Use the binary string representation when you are indexing hash objects (as we are here), but use the default list of `float` for JSON objects. ```python -hf = HFTextVectorizer(model="sentence-transformers/all-MiniLM-L6-v2") - -data = [ - { - "content": "That is a very happy person", - "genre": "persons", - "embedding": hf.embed("That is a very happy person", as_buffer=True) - }, - { - "content": "That is a happy dog", - "genre": "pets", - "embedding": hf.embed("That is a happy dog", as_buffer=True) - }, - { - "content": "Today is a sunny day", - "genre": "weather", - "embedding": hf.embed("Today is a sunny day", as_buffer=True) - } -] +content = "That is a very happy person" + +r.hset("doc:0", mapping={ + "content": content, + "genre": "persons", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "That is a happy dog" + +r.hset("doc:1", mapping={ + "content": content, + "genre": "pets", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "Today is a sunny day" -index.load(data) +r.hset("doc:2", mapping={ + "content": content, + "genre": "weather", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) ``` ## Run a query -After you have created the index, you are ready to run a query against it. +After you have created the index and added the data, you are ready to run a query. To do this, you must create another embedding vector from your chosen query text. Redis calculates the similarity between the query vector and each embedding vector in the index as it runs the query. It then ranks the results in order of this numeric similarity value. -Create a -[`VectorQuery`](https://docs.redisvl.com/en/stable/api/query.html#vectorquery) -instance by supplying the embedding vector for your -query text, along with the hash or JSON field to match against and the number -of results you want, as shown in the example below. Then, call the -[`query()`](https://docs.redisvl.com/en/stable/api/searchindex.html#redisvl.index.SearchIndex.query) -method of `SearchIndex` with your `VectorQuery` instance to run the query. +The code below creates the query embedding using `model.encode()`, as with +the indexing, and passes it as a parameter when the query executes +(see +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about using query parameters with embeddings). ```python -query = VectorQuery( - vector=hf.embed("That is a happy person"), - vector_field_name="embedding", - return_fields=["content"], - num_results=3, +q = Query( + "*=>[KNN 3 @embedding $vec AS vector_distance]" +).return_field("score").dialect(2) + +query_text = "That is a happy person" + +res = r.ft("vector_idx").search( + q, query_params={ + "vec": model.encode(query_text).astype(np.float32).tobytes() + } ) -results = index.query(query) -print(results) +print(res) ``` The code is now ready to run, but note that it may take a while to complete when you run it for the first time (which happens because RedisVL must download the `all-MiniLM-L6-v2` model data before it can -generate the embeddings). When you run the code, it outputs the following results -as a list of dict objects: +generate the embeddings). When you run the code, it outputs the following result +object (slightly formatted here for clarity): ```Python -[ - { - 'id': 'doc:cf3c28e7fdd44753af3080a9a7f8d8e9', - 'vector_distance': '0.114169985056', - 'content': 'That is a very happy person' - }, - { - 'id': 'doc:614508f297b644d0be47dde5373bb059', - 'vector_distance': '0.610845386982', - 'content': 'That is a happy dog' - }, - { - 'id': 'doc:930a7dfca0d74808baee490d747c9534', - 'vector_distance': '1.48624813557', - 'content': 'Today is a sunny day' - } -] +Result{ + 3 total, + docs: [ + Document { + 'id': 'doc:0', + 'payload': None, + 'vector_distance': '0.114169985056', + 'content': 'That is a very happy person' + }, + Document { + 'id': 'doc:1', + 'payload': None, + 'vector_distance': '0.610845386982', + 'content': 'That is a happy dog' + }, + Document { + 'id': 'doc:2', + 'payload': None, + 'vector_distance': '1.48624813557', + 'content': 'Today is a sunny day' + } + ] +} ``` Note that the results are ordered according to the value of the `vector_distance` field, with the lowest distance indicating the greatest similarity to the query. -As you would expect, the text *"That is a very happy person"* is the result that is -most similar in meaning to the query text *"That is a happy person"*. +As you would expect, the result for `doc:0` with the content text *"That is a very happy person"* +is the result that is most similar in meaning to the query text +*"That is a happy person"*. ## Learn more -See the [RedisVL documentation](https://docs.redisvl.com/en/stable/index.html) -for more details about its features and examples of how to use them. +See +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about the indexing options, distance metrics, and query format +for vectors.