diff --git a/content/develop/clients/redis-py/queryjson.md b/content/develop/clients/redis-py/queryjson.md index 189dd5e3a3..4cb28eb354 100644 --- a/content/develop/clients/redis-py/queryjson.md +++ b/content/develop/clients/redis-py/queryjson.md @@ -12,7 +12,7 @@ categories: description: Learn how to use the Redis query engine with JSON linkTitle: Index and query JSON title: Example - Index and query JSON documents -weight: 2 +weight: 3 --- This example shows how to create a diff --git a/content/develop/clients/redis-py/transpipe.md b/content/develop/clients/redis-py/transpipe.md index 139c143516..6a2916c88c 100644 --- a/content/develop/clients/redis-py/transpipe.md +++ b/content/develop/clients/redis-py/transpipe.md @@ -12,7 +12,7 @@ categories: description: Learn how to use Redis pipelines and transactions linkTitle: Pipelines/transactions title: Pipelines and transactions -weight: 2 +weight: 5 --- Redis lets you send a sequence of commands to the server together in a batch. diff --git a/content/develop/clients/redis-py/vecsearch.md b/content/develop/clients/redis-py/vecsearch.md new file mode 100644 index 0000000000..be4bacd341 --- /dev/null +++ b/content/develop/clients/redis-py/vecsearch.md @@ -0,0 +1,234 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Learn how to index and query vector embeddings with Redis +linkTitle: Index and query vectors +title: Index and query vectors +weight: 4 +--- + +[Redis Query Engine]({{< relref "/develop/interact/search-and-query" >}}) +lets you index vector fields in [hash]({{< relref "/develop/data-types/hashes" >}}) +or [JSON]({{< relref "/develop/data-types/json" >}}) objects (see the +[Vectors]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors" >}}) +reference page for more information). +Among other things, vector fields can store *text embeddings*, which are AI-generated vector +representations of the semantic information in pieces of text. The +[vector distance]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +between two embeddings indicates how similar they are semantically. By comparing the +similarity of an embedding generated from some query text with embeddings stored in hash +or JSON fields, Redis can retrieve documents that closely match the query in terms +of their meaning. + +In the example below, we use the +[`sentence-transformers`](https://pypi.org/project/sentence-transformers/) +library to generate vector embeddings to store and index with +Redis Query Engine. + +## Initialize + +Install [`redis-py`]({{< relref "/develop/clients/redis-py" >}}) if you +have not already done so. Also, install `sentence-transformers` with the +following command: + +```bash +pip install sentence-transformers +``` + +In a new Python source file, start by importing the required classes: + +```python +from sentence_transformers import SentenceTransformer +from redis.commands.search.query import Query +from redis.commands.search.field import TextField, TagField, VectorField +from redis.commands.search.indexDefinition import IndexDefinition, IndexType + +import numpy as np +import redis +``` + +The first of these imports is the +`SentenceTransformer` class, which generates an embedding from a section of text. +Here, we create an instance of `SentenceTransformer` that uses the +[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) +model for the embeddings. This model generates vectors with 384 dimensions, regardless +of the length of the input text, but note that the input is truncated to 256 +tokens (see +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) +at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens +are related to the original text). + +```python +model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") +``` + +## Create the index + +Connect to Redis and delete any index previously created with the +name `vector_idx`. (The `dropindex()` call throws an exception if +the index doesn't already exist, which is why you need the +`try: except:` block.) + +```python +r = redis.Redis(decode_responses=True) + +try: + r.ft("vector_idx").dropindex(True) +except redis.exceptions.ResponseError: + pass +``` + +Next, we create the index. +The schema in the example below specifies hash objects for storage and includes +three fields: the text content to index, a +[tag]({{< relref "/develop/interact/search-and-query/advanced-concepts/tags" >}}) +field to represent the "genre" of the text, and the embedding vector generated from +the original text content. The `embedding` field specifies +[HNSW]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#hnsw-index" >}}) +indexing, the +[L2]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +vector distance metric, `Float32` values to represent the vector's components, +and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model. + +```python +schema = ( + TextField("content"), + TagField("genre"), + VectorField("embedding", "HNSW", { + "TYPE": "FLOAT32", + "DIM": 384, + "DISTANCE_METRIC":"L2" + }) +) + +r.ft("vector_idx").create_index( + schema, + definition=IndexDefinition( + prefix=["doc:"], index_type=IndexType.HASH + ) +) +``` + +## Add data + +You can now supply the data objects, which will be indexed automatically +when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as +you use the `doc:` prefix specified in the index definition. + +Use the `model.encode()` method of `SentenceTransformer` +as shown below to create the embedding that represents the `content` field. +The `astype()` option that follows the `model.encode()` call specifies that +we want a vector of `float32` values. The `tobytes()` option encodes the +vector components together as a single binary string rather than the +default Python list of `float` values. +Use the binary string representation when you are indexing hash objects +(as we are here), but use the default list of `float` for JSON objects. + +```python +content = "That is a very happy person" + +r.hset("doc:0", mapping={ + "content": content, + "genre": "persons", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "That is a happy dog" + +r.hset("doc:1", mapping={ + "content": content, + "genre": "pets", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "Today is a sunny day" + +r.hset("doc:2", mapping={ + "content": content, + "genre": "weather", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) +``` + +## Run a query + +After you have created the index and added the data, you are ready to run a query. +To do this, you must create another embedding vector from your chosen query +text. Redis calculates the similarity between the query vector and each +embedding vector in the index as it runs the query. It then ranks the +results in order of this numeric similarity value. + +The code below creates the query embedding using `model.encode()`, as with +the indexing, and passes it as a parameter when the query executes +(see +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about using query parameters with embeddings). + +```python +q = Query( + "*=>[KNN 3 @embedding $vec AS vector_distance]" +).return_field("score").dialect(2) + +query_text = "That is a happy person" + +res = r.ft("vector_idx").search( + q, query_params={ + "vec": model.encode(query_text).astype(np.float32).tobytes() + } +) + +print(res) +``` + +The code is now ready to run, but note that it may take a while to complete when +you run it for the first time (which happens because RedisVL must download the +`all-MiniLM-L6-v2` model data before it can +generate the embeddings). When you run the code, it outputs the following result +object (slightly formatted here for clarity): + +```Python +Result{ + 3 total, + docs: [ + Document { + 'id': 'doc:0', + 'payload': None, + 'vector_distance': '0.114169985056', + 'content': 'That is a very happy person' + }, + Document { + 'id': 'doc:1', + 'payload': None, + 'vector_distance': '0.610845386982', + 'content': 'That is a happy dog' + }, + Document { + 'id': 'doc:2', + 'payload': None, + 'vector_distance': '1.48624813557', + 'content': 'Today is a sunny day' + } + ] +} +``` + +Note that the results are ordered according to the value of the `vector_distance` +field, with the lowest distance indicating the greatest similarity to the query. +As you would expect, the result for `doc:0` with the content text *"That is a very happy person"* +is the result that is most similar in meaning to the query text +*"That is a happy person"*. + +## Learn more + +See +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about the indexing options, distance metrics, and query format +for vectors.