chroma_db/Chroma_DB_Tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JcQ5ob7wntiS"
      },
      "outputs": [],
      "source": [
        "!pip install chromadb -q\n",
        "!pip install sentence-transformers -q"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Youtube Video: https://youtu.be/eCCHDxMaFIk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xB6Sum8xnw3e"
      },
      "outputs": [],
      "source": [
        "import chromadb\n",
        "\n",
        "client = chromadb.Client()\n",
        "\n",
        "collection = client.create_collection(\"yt_demo\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-3ILWMStvKJA"
      },
      "source": [
        "By default, Chroma uses the Sentence Transformers all-MiniLM-L6-v2 model to create embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DOLLVxcdoMxA",
        "outputId": "04f455d8-cd38-4289-f364-23f838d57453"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/root/.cache/chroma/onnx_models/all-MiniLM-L6-v2/onnx.tar.gz: 100%|██████████| 79.3M/79.3M [00:01<00:00, 71.4MiB/s]\n"
          ]
        }
      ],
      "source": [
        "collection.add(\n",
        "    documents=[\"This is a document about cat\", \"This is a document about car\"],\n",
        "    metadatas=[{\"category\": \"animal\"}, {\"category\": \"vehicle\"}],\n",
        "    ids=[\"id1\", \"id2\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tTYUHlNFoScw",
        "outputId": "15b1b81e-5054-464c-fc0e-d41b41e7df94"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['id2']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['This is a document about car']],\n",
              " 'metadatas': [[{'category': 'vehicle'}]],\n",
              " 'distances': [[0.8069301247596741]]}"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results = collection.query(\n",
        "    query_texts=[\"vehicle\"],\n",
        "    n_results=1\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ovRRWCnbtHjI",
        "outputId": "b5a95216-7b11-4743-adf6-04a6d20a0960"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "File Name: Different Types of Pet Animals.txt\n",
            "Content: Pet animals come in all shapes and sizes, each suited to different lifestyles and home environments. Dogs and cats are the most common, known for their companionship and unique personalities. Small mammals like hamsters, guinea pigs, and rabbits are often chosen for their low maintenance needs. Birds offer beauty and song, and reptiles like turtles and lizards can make intriguing pets. Even fish, with their calming presence, can be wonderful pets.\n",
            "\n",
            "File Name: Nutrition Needs of Pet Animals.txt\n",
            "Content: Proper nutrition is vital for the health and wellbeing of pets. Dogs and cats require a balanced diet that includes proteins, carbohydrates, and fats. Some may even have specific dietary needs based on their breed or age. Birds typically thrive on a diet of seeds, fruits, and vegetables, while reptiles have diverse diets ranging from live insects to fresh produce. Fish diets depend greatly on the species, with some needing live food and others subsisting on flakes or pellets.\n",
            "\n",
            "File Name: Health Care for Pets.txt\n",
            "Content: Routine health care is crucial for pets to live long, happy lives. Regular vet check-ups help catch potential issues early and keep vaccinations up to date. Dental care is also essential to prevent diseases in pets, especially in dogs and cats. Regular grooming, parasite control, and weight management are other important aspects of pet health care.\n",
            "\n",
            "File Name: Training and Behaviour of Pets.txt\n",
            "Content: Training is essential for a harmonious life with pets, particularly for dogs. It helps pets understand their boundaries and makes cohabitation easier for both pets and owners. Training should be based on positive reinforcement. Understanding pet behavior is also important, as changes in behavior can often be a sign of underlying health issues.\n",
            "\n",
            "File Name: The Emotional Bond Between Humans and Pets.txt\n",
            "Content: Pets offer more than just companionship; they provide emotional support, reduce stress, and can even help their owners lead healthier lives. The bond between pets and their owners is strong, and many people consider their pets as part of the family. This bond can be especially important in times of personal or societal stress, providing comfort and consistency.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "def read_files_from_folder(folder_path):\n",
        "    file_data = []\n",
        "\n",
        "    for file_name in os.listdir(folder_path):\n",
        "        if file_name.endswith(\".txt\"):\n",
        "            with open(os.path.join(folder_path, file_name), 'r') as file:\n",
        "                content = file.read()\n",
        "                file_data.append({\"file_name\": file_name, \"content\": content})\n",
        "\n",
        "    return file_data\n",
        "\n",
        "folder_path = \"pets\"  # your folder path\n",
        "file_data = read_files_from_folder(folder_path)\n",
        "\n",
        "for data in file_data:\n",
        "    print(f\"File Name: {data['file_name']}\")\n",
        "    print(f\"Content: {data['content']}\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a8DyrATKvb-E"
      },
      "outputs": [],
      "source": [
        "documents = []\n",
        "metadatas = []\n",
        "ids = []\n",
        "for index,data in enumerate(file_data):\n",
        "  documents.append(data['content'])\n",
        "  metadatas.append({'source':data['file_name']})\n",
        "  ids.append(str(index+1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7fW9Odtav8gE",
        "outputId": "3ebfa29e-73e6-4909-b940-81b431a77b84"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'source': 'Different Types of Pet Animals.txt'},\n",
              " {'source': 'Nutrition Needs of Pet Animals.txt'},\n",
              " {'source': 'Health Care for Pets.txt'},\n",
              " {'source': 'Training and Behaviour of Pets.txt'},\n",
              " {'source': 'The Emotional Bond Between Humans and Pets.txt'}]"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "metadatas"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UtGTFkGfotBh"
      },
      "outputs": [],
      "source": [
        "from chromadb.config import Settings\n",
        "client = chromadb.Client(Settings(\n",
        "    chroma_db_impl=\"duckdb+parquet\",\n",
        "    persist_directory=\"pet_db\" # Optional, defaults to .chromadb/ in the current directory\n",
        "))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fdnhyH9JvUlo"
      },
      "outputs": [],
      "source": [
        "pet_collection = client.create_collection(\"pet_collection\")\n",
        "pet_collection.add(\n",
        "    documents=documents,\n",
        "    metadatas=metadatas,\n",
        "    ids=ids\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "33aGfR2WwQ0J",
        "outputId": "4d2f3dbb-7e03-4499-fc09-f9481e77cfcb"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['1']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['Pet animals come in all shapes and sizes, each suited to different lifestyles and home environments. Dogs and cats are the most common, known for their companionship and unique personalities. Small mammals like hamsters, guinea pigs, and rabbits are often chosen for their low maintenance needs. Birds offer beauty and song, and reptiles like turtles and lizards can make intriguing pets. Even fish, with their calming presence, can be wonderful pets.']],\n",
              " 'metadatas': [[{'source': 'Different Types of Pet Animals.txt'}]],\n",
              " 'distances': [[0.7325009703636169]]}"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results = pet_collection.query(\n",
        "    query_texts=[\"What are the different kinds of pets people commonly own?\"],\n",
        "    n_results=1\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Iz4LH4SwbkV",
        "outputId": "7bf0837f-8eda-4005-8379-f37dea34e3db"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['5']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['Pets offer more than just companionship; they provide emotional support, reduce stress, and can even help their owners lead healthier lives. The bond between pets and their owners is strong, and many people consider their pets as part of the family. This bond can be especially important in times of personal or societal stress, providing comfort and consistency.']],\n",
              " 'metadatas': [[{'source': 'The Emotional Bond Between Humans and Pets.txt'}]],\n",
              " 'distances': [[0.6395788192749023]]}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results = pet_collection.query(\n",
        "    query_texts=[\"What are the emotional benefits of owning a pet?\"],\n",
        "    n_results=1,\n",
        "    where_document={\"$contains\":\"reptiles\"}\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c35LdjGcILAY",
        "outputId": "49b58b6c-2e64-45e6-d34f-8baeb99332d2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['4']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['Training is essential for a harmonious life with pets, particularly for dogs. It helps pets understand their boundaries and makes cohabitation easier for both pets and owners. Training should be based on positive reinforcement. Understanding pet behavior is also important, as changes in behavior can often be a sign of underlying health issues.']],\n",
              " 'metadatas': [[{'source': 'Training and Behaviour of Pets.txt'}]],\n",
              " 'distances': [[0.8881876468658447]]}"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results = pet_collection.query(\n",
        "    query_texts=[\"What are the emotional benefits of owning a pet?\"],\n",
        "    n_results=1,\n",
        "    where={\"source\": \"Training and Behaviour of Pets.txt\"},\n",
        "    # where_document={\"$contains\":\"reptiles\"}\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RKKjK5H0zTu_"
      },
      "outputs": [],
      "source": [
        "from sentence_transformers import SentenceTransformer\n",
        "model = SentenceTransformer('paraphrase-MiniLM-L3-v2')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gYIQ9hglzk0w"
      },
      "outputs": [],
      "source": [
        "documents = []\n",
        "embeddings = []\n",
        "metadatas = []\n",
        "ids = []\n",
        "for index,data in enumerate(file_data):\n",
        "  documents.append(data['content'])\n",
        "  embeding = model.encode(data['content']).tolist()\n",
        "  embeddings.append(embeding)\n",
        "  metadatas.append({'source':data['file_name']})\n",
        "  ids.append(str(index+1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OMpniZujz7Lr"
      },
      "outputs": [],
      "source": [
        "pet_collection_emb = client.create_collection(\"pet_collection_emb\")\n",
        "pet_collection_emb.add(\n",
        "    documents=documents,\n",
        "    embeddings=embeddings,\n",
        "    metadatas=metadatas,\n",
        "    ids=ids\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9JyUU8wM1TvJ",
        "outputId": "7be0a9a0-1ecd-4d6b-fe6d-8ee8b29f5601"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['1']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['Pet animals come in all shapes and sizes, each suited to different lifestyles and home environments. Dogs and cats are the most common, known for their companionship and unique personalities. Small mammals like hamsters, guinea pigs, and rabbits are often chosen for their low maintenance needs. Birds offer beauty and song, and reptiles like turtles and lizards can make intriguing pets. Even fish, with their calming presence, can be wonderful pets.']],\n",
              " 'metadatas': [[{'source': 'Different Types of Pet Animals.txt'}]],\n",
              " 'distances': [[12.040446281433105]]}"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = \"What are the different kinds of pets people commonly own?\"\n",
        "input_em = model.encode(query).tolist()\n",
        "\n",
        "results = pet_collection_emb.query(\n",
        "    query_embeddings=[input_em],\n",
        "    n_results=1\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "678hbuGO47iD",
        "outputId": "4618619c-cee5-4b96-b1c8-c25246edc287"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ids': [['2']],\n",
              " 'embeddings': None,\n",
              " 'documents': [['Proper nutrition is vital for the health and wellbeing of pets. Dogs and cats require a balanced diet that includes proteins, carbohydrates, and fats. Some may even have specific dietary needs based on their breed or age. Birds typically thrive on a diet of seeds, fruits, and vegetables, while reptiles have diverse diets ranging from live insects to fresh produce. Fish diets depend greatly on the species, with some needing live food and others subsisting on flakes or pellets.']],\n",
              " 'metadatas': [[{'source': 'Nutrition Needs of Pet Animals.txt'}]],\n",
              " 'distances': [[17.143936157226562]]}"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = \"foods that are recommended for  dogs?\"\n",
        "input_em = model.encode(query).tolist()\n",
        "\n",
        "results = pet_collection_emb.query(\n",
        "    query_embeddings=[input_em],\n",
        "    n_results=1\n",
        ")\n",
        "results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c-ih8cSs4XLC"
      },
      "outputs": [],
      "source": [
        "# pet_collection_emb.peek()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7NOZBqwYxO6I"
      },
      "outputs": [],
      "source": [
        "# !rm -rf  '/content/pet_db'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EpjgL94A0ArD",
        "outputId": "4bb28d36-57bd-4547-ba24-40c480dcf38f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  adding: pet_db/ (stored 0%)\n",
            "  adding: pet_db/index/ (stored 0%)\n",
            "  adding: pet_db/index/index_metadata_b0dc65b7-7a78-4b96-82ee-6ecadf899eec.pkl (deflated 14%)\n",
            "  adding: pet_db/index/index_b0dc65b7-7a78-4b96-82ee-6ecadf899eec.bin (deflated 15%)\n",
            "  adding: pet_db/index/index_metadata_fb7aeba0-098b-43ac-8392-5c1c4728593b.pkl (deflated 14%)\n",
            "  adding: pet_db/index/uuid_to_id_b0dc65b7-7a78-4b96-82ee-6ecadf899eec.pkl (deflated 24%)\n",
            "  adding: pet_db/index/id_to_uuid_fb7aeba0-098b-43ac-8392-5c1c4728593b.pkl (deflated 16%)\n",
            "  adding: pet_db/index/uuid_to_id_fb7aeba0-098b-43ac-8392-5c1c4728593b.pkl (deflated 22%)\n",
            "  adding: pet_db/index/index_fb7aeba0-098b-43ac-8392-5c1c4728593b.bin (deflated 14%)\n",
            "  adding: pet_db/index/id_to_uuid_b0dc65b7-7a78-4b96-82ee-6ecadf899eec.pkl (deflated 16%)\n"
          ]
        }
      ],
      "source": [
        "!zip -r 'pet_db.zip' 'pet_db'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vPvf-UcCBkja"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}