vector-search

Amazon MemoryDB for Redis for Vector Search

This is a collection of projects for Python development with CDK.

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

(.venv) $ pip install -r requirements.txt

To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command.

Before synthesizing the CloudFormation, you should set approperly the cdk context configuration file, cdk.context.json.

For example:

{
  "memorydb_user_name": "memdb-admin",
  "memorydb_cluster_name": "vectordb"
}

Now this point you can now synthesize the CloudFormation template for this code.

(.venv) $ export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
(.venv) $ export CDK_DEFAULT_REGION=us-east-1 # your-aws-account-region
(.venv) $ cdk synth --all

Now we will be able to deploy all the CDK stacks at once like this:

(.venv) $ cdk deploy --require-approval never --all

Verify

After a few minutes, the cluster is running and you can connect using the Redis command line interface or any Redis client.

All MemoryDB clusters run in a virtual private cloud (VPC). You need to EC2 instance or Cloud9 in your VPC to access MemoryDB clusters. Also either EC2 instance or Cloud9 must be given a proper security group such as memorydb-client-sg created in the stack above.

ℹ️ The username and password are stored in the AWS Secrets Manager as a name such as MemoryDBSecret-xxxxxxxxxxxx.

Connect to Amazon MemoryDB using Redis command line interface

$ wget https://download.redis.io/releases/redis-6.2.6.tar.gz
$ tar -xzvf redis-6.2.6.tar.gz
$ cd redis-6.2.6
$ make MALLOC=jemalloc BUILD_TLS=yes
$ sudo make install
$ redis-cli -c --tls -h memorydb-cluster-endpoint --user 'user-name' --askpass
Please input password: ****************
clustercfg.your-memorydb-name.memorydb.region.amazonaws.com:6379>

Connect to Amazon MemoryDB using Python Redis client

$ pip install redis
$ python
Python 3.9.7 (default, Oct 13 2021, 06:44:56)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> db_host = 'clustercfg.your-memorydb-name.memorydb.region.amazonaws.com'
>>> db_port = 6379
>>> db_username = 'user-name'
>>> db_password = 'user-password'
>>> redis_client = redis.Redis(host=db_host, port=db_port,
                               username=db_username, password=db_password,
                               decode_responses=True, ssl=True, ssl_cert_reqs="none")
>>> if redis_client.ping():
...     print("Connected to Redis")
...
Connected to Redis
>>> redis_client.execute_command('ft._list')
[]
>>> redis_client.execute_command('ft.config get *')
[['timeout', '10000']]
>>>

Useful commands

• cdk ls list all stacks in the app
• cdk synth emits the synthesized CloudFormation template
• cdk deploy deploy this stack to your default AWS account/region
• cdk diff compare deployed stack with current state
• cdk docs open CDK documentation

Learn more

• Introducing Amazon MemoryDB for Redis – A Redis-Compatible, Durable, In-Memory Database Service
• Build with Redis data structures for microservices using Amazon MemoryDB for Redis and Amazon ECS
• Amazon MemoryDB for Redis - Authenticating users with Access Control Lists (ACLs)
• Vector search - Amazon MemoryDB for Redis
• Vector search commands - Amazon MemoryDB for Redis
• Amazon MemoryDB for Redis engine versions
• Amazon MemoryDB for Redis Samples
• Redis ACL
• Redis Documentation
• Redis as a vector database quick start guide

Enjoy!