diff --git a/docs/chat_message_history.ipynb b/docs/chat_message_history.ipynb index 7473785..0ec6baf 100644 --- a/docs/chat_message_history.ipynb +++ b/docs/chat_message_history.ipynb @@ -6,123 +6,167 @@ "id": "6-0_o3DxsFGi" }, "source": [ - "Google Database\n", + "# Google Memorystore for Redis\n", "\n", - "Use [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store chat message history for LangChain." + "> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n", + "\n", + "This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store chat message history with the `MemorystoreChatMessageHistory` class.\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/chat_message_history.ipynb)" ] }, { "cell_type": "markdown", - "metadata": { - "id": "dWakBoPnsFGj" - }, + "metadata": {}, "source": [ - "## Pre-reqs" + "## Before You Begin\n", + "\n", + "To run this notebook, you will need to do the following:\n", + "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n", + "* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n", + "\n", + "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts." ] }, { - "cell_type": "markdown", - "metadata": { - "id": "EudfLv_UsFGk" - }, + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ - "### Setting Up a Memorystore for Redis Instance\n", - "\n", - "Before proceeding, an active Memorystore for Redis instance is needed to store chat message history:\n", - "\n", - "* Create a Memorystore for Redis Instance (>= 5.0): If an instance doesn't exist, follow the instructions at https://cloud.google.com/memorystore/docs/redis/create-instance-console to create a new one. Ensure the version is greater than or equal to 5.0.\n", - "* Obtain Endpoint: Note the endpoint associated with the instance." + "# @markdown Please specify an endpoint associated with the instance or demo purpose.\n", + "ENDPOINT = \"redis://127.0.0.1:6379\" # @param {type:\"string\"}" ] }, { "cell_type": "markdown", - "metadata": { - "id": "J5nxjYxHsFGk" - }, + "metadata": {}, "source": [ - "### Installing the LangChain Memorystore for Redis Module\n", + "### 🦜🔗 Library Installation\n", "\n", - "Interaction with the Memorystore for Redis instance from LangChain requires installing the necessary module:" + "The integration lives in its own `langchain-google-memorystore-redis` package, so we need to install it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "tags": [], - "id": "iLwVMVkYsFGk" + "id": "iLwVMVkYsFGk", + "tags": [] }, "outputs": [], "source": [ - "# Install Memorystore for Redis for LangChain module\n", - "%pip install langchain_google_memorystore_redis" + "%pip install -upgrade --quiet langchain-google-memorystore-redis" ] }, { "cell_type": "markdown", - "metadata": { - "id": "2L7kMu__sFGl" - }, + "metadata": {}, "source": [ - "## Basic Usage" + "**Colab only:** Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# # Automatically restart kernel after installs so that your environment can access the new packages\n", + "# import IPython\n", + "\n", + "# app = IPython.Application.instance()\n", + "# app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", - "metadata": { - "id": "A2fT1iEhsFGl" - }, + "metadata": {}, "source": [ - "### Initialize a MemorystoreChatMessageHistory\n", + "### ☁ Set Your Google Cloud Project\n", + "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n", "\n", - "Each chat message history object must have a unique session ID. If the session ID already has messages stored in Redis, they will can be retrieved." + "If you don't know your project ID, try the following:\n", + "\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "YEDKWR6asFGl" - }, + "metadata": {}, "outputs": [], "source": [ - "import redis\n", - "from langchain_google_memorystore_redis import MemorystoreChatMessageHistory\n", + "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n", "\n", - "# Connect to a Memorystore for Redis instance\n", - "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n", + "PROJECT_ID = \"my-project-id\" # @param {type:\"string\"}\n", "\n", - "message_history = MemorystoreChatMessageHistory(redis_client, session_id=\"session1\")" + "# Set the project id\n", + "!gcloud config set project {PROJECT_ID}" ] }, { "cell_type": "markdown", - "metadata": { - "id": "EmoJcTgosFGl" - }, + "metadata": {}, "source": [ - "### Add Messages" + "### 🔐 Authentication\n", + "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n", + "\n", + "* If you are using Colab to run this notebook, use the cell below and continue.\n", + "* If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)." ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from google.colab import auth\n", + "\n", + "auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", "metadata": { - "id": "gB1PGe6wsFGm" + "id": "2L7kMu__sFGl" }, - "outputs": [], "source": [ - "message_history.add_ai_message(\"Hey! I am AI!\")\n", - "message_history.add_user_message(\"Hey! I am human!\")" + "## Basic Usage" ] }, { "cell_type": "markdown", "metadata": { - "id": "02xxvmzTsFGm" + "id": "A2fT1iEhsFGl" + }, + "source": [ + "### MemorystoreChatMessageHistory\n", + "\n", + "To initialize the `MemorystoreMessageHistory` class you need to provide only 2 things:\n", + "\n", + "1. `redis_client` - An instance of a Memorystore Redis.\n", + "1. `session_id` - Each chat message history object must have a unique session ID. If the session ID already has messages stored in Redis, they will can be retrieved." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "YEDKWR6asFGl" }, + "outputs": [], "source": [ - "### Retrieve All Messages Stored in the Session" + "import redis\n", + "from langchain_google_memorystore_redis import MemorystoreChatMessageHistory\n", + "\n", + "# Connect to a Memorystore for Redis instance\n", + "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n", + "\n", + "message_history = MemorystoreChatMessageHistory(redis_client, session_id=\"session1\")" ] }, { @@ -142,7 +186,11 @@ "id": "sFJdt3ubsFGo" }, "source": [ - "### Clear Messages" + "#### Cleaning up\n", + "\n", + "When the history of a specific session is obsolete and can be deleted, it can be done the following way.\n", + "\n", + "**Note:** Once deleted, the data is no longer stored in Memorystore for Redis and is gone forever." ] }, { @@ -158,6 +206,9 @@ } ], "metadata": { + "colab": { + "provenance": [] + }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", @@ -174,9 +225,6 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" - }, - "colab": { - "provenance": [] } }, "nbformat": 4, diff --git a/docs/document_loader.ipynb b/docs/document_loader.ipynb index 0fe4e31..3ad4cd5 100644 --- a/docs/document_loader.ipynb +++ b/docs/document_loader.ipynb @@ -6,233 +6,290 @@ "id": "6-0_o3DxsFGi" }, "source": [ - "Google Database\n", + "# Google Memorystore for Redis\n", "\n", - "Use [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store Documents for LangChain." + "> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n", + "\n", + "This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to [save, load and delete langchain documents](https://python.langchain.com/docs/modules/data_connection/document_loaders/) with `MemorystoreDocumentLoader` and `MemorystoreDocumentSaver`.\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/document_loader.ipynb)" ] }, { "cell_type": "markdown", - "metadata": { - "id": "dWakBoPnsFGj" - }, + "metadata": {}, "source": [ - "## Pre-reqs" + "## Before You Begin\n", + "\n", + "To run this notebook, you will need to do the following:\n", + "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n", + "* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n", + "\n", + "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts." ] }, { - "cell_type": "markdown", - "metadata": { - "id": "EudfLv_UsFGk" - }, + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ - "### Setting Up a Memorystore for Redis Instance\n", - "\n", - "Before proceeding, an active Memorystore for Redis instance is needed to store chat message history:\n", - "\n", - "* Create a Memorystore for Redis Instance (>= 5.0): If an instance doesn't exist, follow the instructions at https://cloud.google.com/memorystore/docs/redis/create-instance-console to create a new one. Ensure the version is greater than or equal to 5.0.\n", - "* Obtain Endpoint: Note the endpoint associated with the instance." + "# @markdown Please specify an endpoint associated with the instance and a key prefix for demo purpose.\n", + "ENDPOINT = \"redis://127.0.0.1:6379\" # @param {type:\"string\"}\n", + "KEY_PREFIX = \"doc:\" # @param {type:\"string\"}" ] }, { "cell_type": "markdown", - "metadata": { - "id": "J5nxjYxHsFGk" - }, + "metadata": {}, "source": [ - "### Installing the LangChain Memorystore for Redis Module\n", + "### 🦜🔗 Library Installation\n", "\n", - "Interaction with the Memorystore for Redis instance from LangChain requires installing the necessary module:" + "The integration lives in its own `langchain-google-memorystore-redis` package, so we need to install it." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "tags": [], - "id": "iLwVMVkYsFGk" - }, + "metadata": {}, "outputs": [], "source": [ - "# Install Memorystore for Redis for LangChain module\n", - "%pip install langchain_google_memorystore_redis" + "%pip install -upgrade --quiet langchain-google-memorystore-redis" ] }, { "cell_type": "markdown", - "metadata": { - "id": "2L7kMu__sFGl" - }, + "metadata": {}, "source": [ - "## Basic Usage" + "**Colab only**: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# # Automatically restart kernel after installs so that your environment can access the new packages\n", + "# import IPython\n", + "\n", + "# app = IPython.Application.instance()\n", + "# app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", - "metadata": { - "id": "A2fT1iEhsFGl" - }, + "metadata": {}, "source": [ - "### Initialize a MemorystoreDocumentLoader\n", + "### ☁ Set Your Google Cloud Project\n", + "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n", "\n", - "Initialize a loader that loads all documents stored in the Memorystore for Redis instance with a specific prefix." + "If you don't know your project ID, try the following:\n", + "\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "YEDKWR6asFGl" - }, + "metadata": {}, "outputs": [], "source": [ - "import redis\n", - "from langchain_google_memorystore_redis import MemorystoreDocumentLoader\n", + "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n", "\n", - "# Connect to a Memorystore for Redis instance\n", - "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n", - "prefix = \"doc:\"\n", + "PROJECT_ID = \"my-project-id\" # @param {type:\"string\"}\n", "\n", - "loader = MemorystoreDocumentLoader(\n", - " client=redis_client,\n", - " key_prefix=prefix,\n", - " content_fields=set([\"page_content\"]),\n", - ")" + "# Set the project id\n", + "!gcloud config set project {PROJECT_ID}" ] }, { "cell_type": "markdown", - "metadata": { - "id": "EmoJcTgosFGl" - }, + "metadata": {}, "source": [ - "### Load Documents\n", + "### 🔐 Authentication\n", + "\n", + "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n", "\n", - "Load all documents stored in the Memorystore for Redis instance at once." + "- If you are using Colab to run this notebook, use the cell below and continue.\n", + "- If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)." ] }, { "cell_type": "code", "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from google.colab import auth\n", + "\n", + "auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", "metadata": { - "id": "gB1PGe6wsFGm" + "id": "2L7kMu__sFGl" }, - "outputs": [], "source": [ - "documents = loader.load()" + "## Basic Usage" ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ - "### Lazy Load Documents\n", + "### Save documents\n", "\n", - "Load the document one-by-one with lazy_load generator." - ], - "metadata": { - "id": "Vbs8gIa24YvJ" - } + "Save langchain documents with `MemorystoreDocumentSaver.add_documents()`. To initialize `MemorystoreDocumentSaver` class you need to provide 2 things:\n", + "1. `client` - A `redis.Redis` client object.\n", + "1. `key_prefix` - A prefix for the keys to store Documents in Redis.\n", + "\n", + "The Documents will be stored into randomly generated keys with the specified prefix of `key_prefix`. Alternatively, you can designate the suffixes of the keys by specifying `ids` in the `add_documents` method." + ] }, { "cell_type": "code", - "source": [ - "for document in loader.lazy_load():\n", - " # Do something\n", - " print(document)" - ], - "metadata": { - "id": "nPhpvLtA4kBM" - }, "execution_count": null, - "outputs": [] + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_google_memorystore_redis import MemorystoreDocumentSaver\n", + "\n", + "test_docs = [\n", + " Document(\n", + " page_content=\"Apple Granny Smith 150 0.99 1\",\n", + " metadata={\"fruit_id\": 1},\n", + " ),\n", + " Document(\n", + " page_content=\"Banana Cavendish 200 0.59 0\",\n", + " metadata={\"fruit_id\": 2},\n", + " ),\n", + " Document(\n", + " page_content=\"Orange Navel 80 1.29 1\",\n", + " metadata={\"fruit_id\": 3},\n", + " ),\n", + "]\n", + "doc_ids = [f\"{i}\" for i in range(len(test_docs))]\n", + "\n", + "redis_client = redis.from_url(ENDPOINT)\n", + "saver = MemorystoreDocumentSaver(\n", + " client=redis_client,\n", + " key_prefix=KEY_PREFIX,\n", + " content_field=\"page_content\",\n", + ")\n", + "saver.add_documents(test_docs, ids=docs_ids)" + ] }, { "cell_type": "markdown", "metadata": { - "id": "02xxvmzTsFGm" + "id": "A2fT1iEhsFGl" }, "source": [ - "## Customize Document Page Content & Metadata\n", + "### Load documents\n", "\n", - "When initializing a loader with more than 1 content field, the `page_content` of the loaded Documents will contain a JSON-encoded string with top level fields equal to the specified fields in `content_fields`.\n", + "Initialize a loader that loads all documents stored in the Memorystore for Redis instance with a specific prefix.\n", "\n", - "If the `metadata_fields` are specified, the `metadata` field of the loaded Documents will only have the top level fields equal to the specified `metadata_fields`. If any of the values of the metadata fields is stored as a JSON-encoded string, it will be decoded prior to being loaded to the metadata fields." + "Load langchain documents with `MemorystoreDocumentLoader.load()` or `MemorystoreDocumentLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `MemorystoreDocumentLoader` class you need to provide:\n", + "1. `client` - A `redis.Redis` client object.\n", + "1. `key_prefix` - A prefix for the keys to store Documents in Redis." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "BvS3UFsysFGm" + "id": "YEDKWR6asFGl" }, "outputs": [], "source": [ + "import redis\n", + "from langchain_google_memorystore_redis import MemorystoreDocumentLoader\n", + "\n", + "redis_client = redis.from_url(ENDPOINT)\n", "loader = MemorystoreDocumentLoader(\n", " client=redis_client,\n", - " key_prefix=prefix,\n", - " content_fields=set([\"content_field_1\", \"content_field_2\"]),\n", - " metadata_fields=set([\"title\", \"author\"]),\n", - ")" + " key_prefix=KEY_PREFIX,\n", + " content_fields=set([\"page_content\"]),\n", + ")\n", + "for doc in loader.lazy_load():\n", + " print(\"Loaded documents:\", doc)" ] }, { "cell_type": "markdown", - "metadata": { - "id": "sFJdt3ubsFGo" - }, + "metadata": {}, "source": [ - "## Save Documents\n", + "### Delete documents\n", "\n", - "You can save a list of Documents into Memorystore for Redis instance like below. The Documents will be stored into randomly generated keys with the specified prefix of `key_prefix`. Alternatively, you can designate the suffixes of the keys by specifying `ids` in the `add_documents` method." + "Delete all of keys with the specified prefix in the Memorystore for Redis instance with `MemorystoreDocumentSaver.delete()`. You can also specify the suffixes of the keys if you know." ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "H5I7K3MTsFGo" - }, + "metadata": {}, "outputs": [], "source": [ - "from langchain_google_memorystore_redis import MemorystoreDocumentSaver\n", + "docs = loader.load()\n", + "print(\"Documents before delete:\", docs)\n", "\n", - "# Connect to a Memorystore for Redis instance\n", - "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n", - "prefix = \"doc:\"\n", + "saver.delete(ids=[0])\n", + "print(\"Documents after delete:\", loader.load())\n", "\n", - "saver = MemorystoreDocumentSaver(\n", - " client=redis_client,\n", - " key_prefix=prefix,\n", - " content_field=\"page_content\",\n", - ")\n", - "saver.add_documents(documents)" + "saver.delete()\n", + "print(\"Documents after delete all:\", loader.load())" ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ - "### Clean up Saved Documents\n", - "\n", - "Delete all of keys with the specified prefix in the Memorystore for Redis instance. You can also specify the suffixes of the keys if you know.\n", - "\n" - ], + "## Advanced Usage" + ] + }, + { + "cell_type": "markdown", "metadata": { - "id": "FLVI7Kp7mhL-" - } + "id": "02xxvmzTsFGm" + }, + "source": [ + "### Customize Document Page Content & Metadata\n", + "\n", + "When initializing a loader with more than 1 content field, the `page_content` of the loaded Documents will contain a JSON-encoded string with top level fields equal to the specified fields in `content_fields`.\n", + "\n", + "If the `metadata_fields` are specified, the `metadata` field of the loaded Documents will only have the top level fields equal to the specified `metadata_fields`. If any of the values of the metadata fields is stored as a JSON-encoded string, it will be decoded prior to being loaded to the metadata fields." + ] }, { "cell_type": "code", - "source": [ - "saver.delete()" - ], + "execution_count": null, "metadata": { - "id": "1ArfDYUGmrP3" + "id": "BvS3UFsysFGm" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "loader = MemorystoreDocumentLoader(\n", + " client=redis_client,\n", + " key_prefix=prefix,\n", + " content_fields=set([\"content_field_1\", \"content_field_2\"]),\n", + " metadata_fields=set([\"title\", \"author\"]),\n", + ")" + ] } ], "metadata": { + "colab": { + "provenance": [ + { + "file_id": "1kuFhDfyzOdzS1apxQ--1efXB1pJ79yVY", + "timestamp": 1708033015250 + } + ] + }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", @@ -249,14 +306,6 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" - }, - "colab": { - "provenance": [ - { - "file_id": "1kuFhDfyzOdzS1apxQ--1efXB1pJ79yVY", - "timestamp": 1708033015250 - } - ] } }, "nbformat": 4,