tutorials/database.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data and Data storage\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup data needed for the notes\n",
    "query_1 = \"What are the benefits of renewable energy?\"  # gt is [0, 3]\n",
    "query_2 = \"How do solar panels impact the environment?\"  # gt is [1, 2]\n",
    "\n",
    "org_documents = [\n",
    "    {\n",
    "        \"title\": \"The Impact of Renewable Energy on the Economy\",\n",
    "        \"content\": \"Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.\",\n",
    "    },\n",
    "    {\n",
    "        \"title\": \"Understanding Solar Panels\",\n",
    "        \"content\": \"Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.\",\n",
    "    },\n",
    "    {\n",
    "        \"title\": \"Pros and Cons of Solar Energy\",\n",
    "        \"content\": \"While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.\",\n",
    "    },\n",
    "    {\n",
    "        \"title\": \"Renewable Energy and Its Effects\",\n",
    "        \"content\": \"Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.\",\n",
    "    },\n",
    "]\n",
    "\n",
    "turns = [\n",
    "    {\n",
    "        \"user\": \"What are the benefits of renewable energy?\",\n",
    "        \"system\": \"I can see you are interested in renewable energy. Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.\",\n",
    "        \"user_time\": \"2021-09-01T12:00:00Z\",\n",
    "        \"system_time\": \"2021-09-01T12:00:01Z\",\n",
    "    },\n",
    "    {\n",
    "        \"user\": \"How do solar panels impact the environment?\",\n",
    "        \"system\": \"Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.\",\n",
    "        \"user_time\": \"2021-09-01T12:00:02Z\",\n",
    "        \"system_time\": \"2021-09-01T12:00:03Z\",\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Document and DialogTurn objects\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=3a69755a-a6a4-47bb-96e6-bc1700c1be81, text='Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute...', meta_data={'title': 'The Impact of Renewable Energy on the Economy'}, vector=[], parent_doc_id=None, order=None, score=None), Document(id=01768bfe-cae8-48d2-ad17-54c612cddb0e, text='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock ele...', meta_data={'title': 'Understanding Solar Panels'}, vector=[], parent_doc_id=None, order=None, score=None), Document(id=bc28fe56-af7b-4fec-ad23-e6dff47820a9, text='While solar energy offers substantial environmental benefits, such as reducing carbon footprints and...', meta_data={'title': 'Pros and Cons of Solar Energy'}, vector=[], parent_doc_id=None, order=None, score=None), Document(id=69bb9d36-c6fb-47cb-8c14-686bf2536d3c, text='Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate ...', meta_data={'title': 'Renewable Energy and Its Effects'}, vector=[], parent_doc_id=None, order=None, score=None)]\n"
     ]
    }
   ],
   "source": [
    "# create Document objects\n",
    "from adalflow.core.types import Document\n",
    "\n",
    "# we will save the content to text and title in the meta_data\n",
    "documents = [\n",
    "    Document(text=doc[\"content\"], meta_data={\"title\": doc[\"title\"]})\n",
    "    for doc in org_documents\n",
    "]\n",
    "print(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[DialogTurn(id='f70534d4-51a7-439d-853e-3db03f858820', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='What are the benefits of renewable energy?', metadata=None), assistant_response=AssistantResponse(response_str='I can see you are interested in renewable energy. Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', metadata=None), user_query_timestamp='2021-09-01T12:00:00Z', assistant_response_timestamp='2021-09-01T12:00:01Z', metadata=None, vector=None), DialogTurn(id='5c596d4b-8419-4528-b5e7-4b1bd6245d18', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='How do solar panels impact the environment?', metadata=None), assistant_response=AssistantResponse(response_str='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', metadata=None), user_query_timestamp='2021-09-01T12:00:02Z', assistant_response_timestamp='2021-09-01T12:00:03Z', metadata=None, vector=None)]\n"
     ]
    }
   ],
   "source": [
    "# create DialogTurn objects\n",
    "\n",
    "from adalflow.core.types import DialogTurn, UserQuery, AssistantResponse\n",
    "\n",
    "dialog_turns = [\n",
    "    DialogTurn(\n",
    "        user_query=UserQuery(query_str=turn[\"user\"]),\n",
    "        assistant_response=AssistantResponse(response_str=turn[\"system\"]),\n",
    "        user_query_timestamp=turn[\"user_time\"],\n",
    "        assistant_response_timestamp=turn[\"system_time\"],\n",
    "    )\n",
    "    for turn in turns\n",
    "]\n",
    "print(dialog_turns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demonstrating the data pipleine creation and application\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): DocumentSplitter(split_by=word, split_length=50, split_overlap=10)\n",
      "  (1): ToEmbeddings(\n",
      "    batch_size=2\n",
      "    (embedder): Embedder(\n",
      "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "      (model_client): OpenAIClient()\n",
      "    )\n",
      "    (batch_embedder): BatchEmbedder(\n",
      "      (embedder): Embedder(\n",
      "        model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "        (model_client): OpenAIClient()\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# prepare the data pipeline\n",
    "\n",
    "from adalflow.core.embedder import Embedder\n",
    "from adalflow.core.types import ModelClientType\n",
    "from adalflow.components.data_process import DocumentSplitter, ToEmbeddings\n",
    "from adalflow.core.component import Sequential\n",
    "\n",
    "\n",
    "model_kwargs = {\n",
    "    \"model\": \"text-embedding-3-small\",\n",
    "    \"dimensions\": 256,\n",
    "    \"encoding_format\": \"float\",\n",
    "}\n",
    "\n",
    "splitter_config = {\"split_by\": \"word\", \"split_length\": 50, \"split_overlap\": 10}\n",
    "\n",
    "splitter = DocumentSplitter(**splitter_config)\n",
    "embedder = Embedder(model_client=ModelClientType.OPENAI(), model_kwargs=model_kwargs)\n",
    "embedder_transformer = ToEmbeddings(embedder, batch_size=2)\n",
    "data_transformer = Sequential(splitter, embedder_transformer)\n",
    "print(data_transformer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare mapping functions to map the data to Document object for the pipeline\n",
    "\n",
    "from typing import Dict\n",
    "\n",
    "\n",
    "# mapping function for org_documents\n",
    "def map_to_document(doc: Dict) -> Document:\n",
    "    return Document(text=doc[\"content\"], meta_data={\"title\": doc[\"title\"]})\n",
    "\n",
    "\n",
    "def map_dialogturn_to_document(turn: DialogTurn) -> Document:\n",
    "    # it can be important to keep the original data's id\n",
    "    return Document(\n",
    "        id=turn.id,\n",
    "        text=turn.user_query.query_str + \" \" + turn.assistant_response.response_str,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=f70534d4-51a7-439d-853e-3db03f858820, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector=[], parent_doc_id=None, order=None, score=None), Document(id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector=[], parent_doc_id=None, order=None, score=None)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 2/2 [00:00<00:00, 1199.74it/s]\n",
      "Batch embedding documents: 100%|██████████| 2/2 [00:00<00:00,  4.86it/s]\n",
      "Adding embeddings to documents from batch: 2it [00:00, 18893.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=99a2bb8a-dcf9-422f-a4b9-f80a02c92f23, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None), Document(id=c7aac9dd-9073-45a7-b152-2b5a713ef455, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None), Document(id=dc6658e5-ae61-43b9-823f-d781850021c6, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None), Document(id=e14b1a27-dcb1-4504-ada4-1d49c65bcc4d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# apply data transformation to the dialog_turns\n",
    "\n",
    "dialog_turns_as_documents = [map_dialogturn_to_document(turn) for turn in dialog_turns]\n",
    "print(dialog_turns_as_documents)\n",
    "\n",
    "# apply data transformation to the documents\n",
    "output = data_transformer(dialog_turns_as_documents)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 4/4 [00:00<00:00, 2855.21it/s]\n",
      "Batch embedding documents: 100%|██████████| 3/3 [00:00<00:00,  4.67it/s]\n",
      "Adding embeddings to documents from batch: 3it [00:00, 103138.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=0ec4a917-ed07-45ed-af54-26141fdb61cb, text='Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute...', meta_data={'title': 'The Impact of Renewable Energy on the Economy'}, vector='len: 256', parent_doc_id=7ffbf15e-19de-43f1-853e-3d1689c40382, order=0, score=None), Document(id=eba7b2c7-ed17-47f0-95b7-f8704e384e0e, text='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock ele...', meta_data={'title': 'Understanding Solar Panels'}, vector='len: 256', parent_doc_id=b051b05e-fcc2-41b7-93b4-9c7389eb5c7b, order=0, score=None), Document(id=e9e9c0e6-eaae-4b13-8e7e-a145eaef757f, text='positive effect on the environment by reducing the reliance on fossil fuels.', meta_data={'title': 'Understanding Solar Panels'}, vector='len: 256', parent_doc_id=b051b05e-fcc2-41b7-93b4-9c7389eb5c7b, order=1, score=None), Document(id=6d5fec5a-d818-4293-a2fd-e2d641688eaf, text='While solar energy offers substantial environmental benefits, such as reducing carbon footprints and...', meta_data={'title': 'Pros and Cons of Solar Energy'}, vector='len: 256', parent_doc_id=89601618-2f28-4f03-aefd-6f9bd7da0611, order=0, score=None), Document(id=8d128c99-473b-4a95-8dd9-fe4e8a6167d3, text='Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate ...', meta_data={'title': 'Renewable Energy and Its Effects'}, vector='len: 256', parent_doc_id=f76f3f25-d301-4927-8c5b-2b23fd67117e, order=0, score=None)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# apply data transformation to the documents\n",
    "org_documents_as_documents = [map_to_document(doc) for doc in org_documents]\n",
    "\n",
    "# apply data transformation to the documents\n",
    "output = data_transformer(org_documents_as_documents)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use LocalDB to help with the in-memory CRUD, and more importantly to keep track of data pipeline and its transformed data and with data storage and restore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LocalDB(name='dialog_turns', items=[], transformed_items={}, transformer_setups={}, mapper_setups={})\n",
      "LocalDB(name='dialog_turns', items=[DialogTurn(id='f70534d4-51a7-439d-853e-3db03f858820', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='What are the benefits of renewable energy?', metadata=None), assistant_response=AssistantResponse(response_str='I can see you are interested in renewable energy. Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', metadata=None), user_query_timestamp='2021-09-01T12:00:00Z', assistant_response_timestamp='2021-09-01T12:00:01Z', metadata=None, vector=None), DialogTurn(id='5c596d4b-8419-4528-b5e7-4b1bd6245d18', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='How do solar panels impact the environment?', metadata=None), assistant_response=AssistantResponse(response_str='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', metadata=None), user_query_timestamp='2021-09-01T12:00:02Z', assistant_response_timestamp='2021-09-01T12:00:03Z', metadata=None, vector=None)], transformed_items={}, transformer_setups={}, mapper_setups={})\n"
     ]
    }
   ],
   "source": [
    "# create a db for the dialog_turns\n",
    "from adalflow.core.db import LocalDB\n",
    "\n",
    "dialog_turn_db = LocalDB(\"dialog_turns\")\n",
    "print(dialog_turn_db)\n",
    "\n",
    "dialog_turn_db.load(dialog_turns)\n",
    "print(dialog_turn_db)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 2/2 [00:00<00:00, 3184.74it/s]\n",
      "Batch embedding documents: 100%|██████████| 2/2 [00:00<00:00,  2.93it/s]\n",
      "Adding embeddings to documents from batch: 2it [00:00, 42581.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=4decc75e-ad04-4942-b88e-c3165c15f6ec, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None), Document(id=207c5220-f7bf-446a-a7ac-fa516baded64, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None), Document(id=f9ccaf5b-d251-4c89-920d-86bb82c01290, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None), Document(id=6d6739d1-d7d4-480e-9c41-bbfb085d0c2d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None)]\n",
      "Sequential(\n",
      "  (0): DocumentSplitter(split_by=word, split_length=50, split_overlap=10)\n",
      "  (1): ToEmbeddings(\n",
      "    batch_size=2\n",
      "    (embedder): Embedder(\n",
      "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "      (model_client): OpenAIClient()\n",
      "    )\n",
      "    (batch_embedder): BatchEmbedder(\n",
      "      (embedder): Embedder(\n",
      "        model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "        (model_client): OpenAIClient()\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")\n",
      "<function map_dialogturn_to_document at 0x13a3e60c0>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# apply data transformation to the dialog_turn_db\n",
    "\n",
    "key = \"split_and_embed\"\n",
    "dialog_turn_db.transform(data_transformer, map_fn=map_dialogturn_to_document, key=key)\n",
    "print(dialog_turn_db.transformed_items[key])\n",
    "print(dialog_turn_db.transformer_setups[key])\n",
    "print(dialog_turn_db.mapper_setups[key])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LocalDB(name='dialog_turns', items=[DialogTurn(id='f70534d4-51a7-439d-853e-3db03f858820', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='What are the benefits of renewable energy?', metadata=None), assistant_response=AssistantResponse(response_str='I can see you are interested in renewable energy. Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', metadata=None), user_query_timestamp='2021-09-01T12:00:00Z', assistant_response_timestamp='2021-09-01T12:00:01Z', metadata=None, vector=None), DialogTurn(id='5c596d4b-8419-4528-b5e7-4b1bd6245d18', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='How do solar panels impact the environment?', metadata=None), assistant_response=AssistantResponse(response_str='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', metadata=None), user_query_timestamp='2021-09-01T12:00:02Z', assistant_response_timestamp='2021-09-01T12:00:03Z', metadata=None, vector=None)], transformed_items={'split_and_embed': [Document(id=4decc75e-ad04-4942-b88e-c3165c15f6ec, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None), Document(id=207c5220-f7bf-446a-a7ac-fa516baded64, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None), Document(id=f9ccaf5b-d251-4c89-920d-86bb82c01290, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None), Document(id=6d6739d1-d7d4-480e-9c41-bbfb085d0c2d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None)]}, transformer_setups={'split_and_embed': Sequential(\n",
      "  (0): DocumentSplitter(split_by=word, split_length=50, split_overlap=10)\n",
      "  (1): ToEmbeddings(\n",
      "    batch_size=2\n",
      "    (embedder): Embedder(\n",
      "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "      (model_client): OpenAIClient()\n",
      "    )\n",
      "    (batch_embedder): BatchEmbedder(\n",
      "      (embedder): Embedder(\n",
      "        model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "        (model_client): OpenAIClient()\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")}, mapper_setups={'split_and_embed': <function map_dialogturn_to_document at 0x13a3e60c0>})\n"
     ]
    }
   ],
   "source": [
    "# save the state of the dialog_turn_db\n",
    "dialog_turn_db.save_state(\"dialog_turn_db_state.pkl\")\n",
    "\n",
    "print(dialog_turn_db)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LocalDB(name='dialog_turns', items=[DialogTurn(id='f70534d4-51a7-439d-853e-3db03f858820', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='What are the benefits of renewable energy?', metadata=None), assistant_response=AssistantResponse(response_str='I can see you are interested in renewable energy. Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', metadata=None), user_query_timestamp='2021-09-01T12:00:00Z', assistant_response_timestamp='2021-09-01T12:00:01Z', metadata=None, vector=None), DialogTurn(id='5c596d4b-8419-4528-b5e7-4b1bd6245d18', user_id=None, session_id=None, order=None, user_query=UserQuery(query_str='How do solar panels impact the environment?', metadata=None), assistant_response=AssistantResponse(response_str='Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', metadata=None), user_query_timestamp='2021-09-01T12:00:02Z', assistant_response_timestamp='2021-09-01T12:00:03Z', metadata=None, vector=None)], transformed_items={'split_and_embed': [Document(id=4decc75e-ad04-4942-b88e-c3165c15f6ec, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None), Document(id=207c5220-f7bf-446a-a7ac-fa516baded64, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None), Document(id=f9ccaf5b-d251-4c89-920d-86bb82c01290, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None), Document(id=6d6739d1-d7d4-480e-9c41-bbfb085d0c2d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None)]}, transformer_setups={'split_and_embed': Sequential(\n",
      "  (0): DocumentSplitter(split_by=word, split_length=50, split_overlap=10)\n",
      "  (1): ToEmbeddings(\n",
      "    batch_size=2\n",
      "    (embedder): Embedder(\n",
      "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "      (model_client): OpenAIClient()\n",
      "    )\n",
      "    (batch_embedder): BatchEmbedder(\n",
      "      (embedder): Embedder(\n",
      "        model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "        (model_client): OpenAIClient()\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")}, mapper_setups={'split_and_embed': <function map_dialogturn_to_document at 0x13a3e60c0>})\n"
     ]
    }
   ],
   "source": [
    "# restore the state of the restored_dialog_turn_db\n",
    "restored_dialog_turn_db = LocalDB.load_state(\"dialog_turn_db_state.pkl\")\n",
    "print(restored_dialog_turn_db)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "# check if the restored_dialog_turn_db is the same as the dialog_turn_db\n",
    "\n",
    "print(str(dialog_turn_db.__dict__) == str(restored_dialog_turn_db.__dict__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CRUD operations\n",
    "\n",
    "We will use a real user conversation by adding a generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generator(\n",
      "  model_kwargs={'model': 'gpt-3.5-turbo'}, \n",
      "  (prompt): Prompt(\n",
      "    template: \n",
      "    {% if task_desc_str or output_format_str or tools_str or examples_str or chat_history_str or context_str or steps_str %}\n",
      "    <SYS>\n",
      "    {% endif %}\n",
      "    {# task desc #}\n",
      "    {% if task_desc_str %}\n",
      "    {{task_desc_str}}\n",
      "    {% endif %}\n",
      "    {# output format #}\n",
      "    {% if output_format_str %}\n",
      "    <OUTPUT_FORMAT>\n",
      "    {{output_format_str}}\n",
      "    </OUTPUT_FORMAT>\n",
      "    {% endif %}\n",
      "    {# tools #}\n",
      "    {% if tools_str %}\n",
      "    <TOOLS>\n",
      "    {{tools_str}}\n",
      "    </TOOLS>\n",
      "    {% endif %}\n",
      "    {# example #}\n",
      "    {% if examples_str %}\n",
      "    <EXAMPLES>\n",
      "    {{examples_str}}\n",
      "    </EXAMPLES>\n",
      "    {% endif %}\n",
      "    {# chat history #}\n",
      "    {% if chat_history_str %}\n",
      "    <CHAT_HISTORY>\n",
      "    {{chat_history_str}}\n",
      "    </CHAT_HISTORY>\n",
      "    {% endif %}\n",
      "    {#contex#}\n",
      "    {% if context_str %}\n",
      "    <CONTEXT>\n",
      "    {{context_str}}\n",
      "    </CONTEXT>\n",
      "    {% endif %}\n",
      "    {# steps #}\n",
      "    {% if steps_str %}\n",
      "    <STEPS>\n",
      "    {{steps_str}}\n",
      "    </STEPS>\n",
      "    {% endif %}\n",
      "    {% if task_desc_str or output_format_str or tools_str or examples_str or chat_history_str or context_str or steps_str %}\n",
      "    </SYS>\n",
      "    {% endif %}\n",
      "    {% if input_str %}\n",
      "    <User>\n",
      "    {{input_str}}\n",
      "    </User>\n",
      "    {% endif %}\n",
      "    You:\n",
      "    , prompt_variables: ['output_format_str', 'examples_str', 'steps_str', 'tools_str', 'context_str', 'input_str', 'task_desc_str', 'chat_history_str']\n",
      "  )\n",
      "  (model_client): OpenAIClient()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# prepare the generator for the dialog turns\n",
    "\n",
    "from adalflow.core import Generator\n",
    "\n",
    "llm_kwargs = {\"model\": \"gpt-3.5-turbo\"}\n",
    "\n",
    "# we will use the default prompt, and using input_str and chat_history_str for the final prompt\n",
    "generator = Generator(model_client=ModelClientType.OPENAI(), model_kwargs=llm_kwargs)\n",
    "print(generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "\n",
      "<SYS>\n",
      "<CHAT_HISTORY>\n",
      "id: \"f70534d4-51a7-439d-853e-3db03f858820\"\n",
      "user_id: null\n",
      "session_id: null\n",
      "order: null\n",
      "user_query: \n",
      "  metadata: null\n",
      "  query_str: What are the benefits of renewable energy?\n",
      "assistant_response: \n",
      "  metadata: null\n",
      "  response_str: I can see you are interested in renewable energy. Renewable energy technologies\n",
      "    not only help in reducing greenhouse gas emissions but also contribute significantly\n",
      "    to the economy by creating jobs in the manufacturing and installation sectors. The\n",
      "    growth in renewable energy usage boosts local economies through increased investment\n",
      "    in technology and infrastructure\n",
      "user_query_timestamp: \"2021-09-01T12:00:00Z\"\n",
      "assistant_response_timestamp: \"2021-09-01T12:00:01Z\"\n",
      "metadata: null\n",
      "vector: null\n",
      "_________\n",
      "id: \"5c596d4b-8419-4528-b5e7-4b1bd6245d18\"\n",
      "user_id: null\n",
      "session_id: null\n",
      "order: null\n",
      "user_query: \n",
      "  metadata: null\n",
      "  query_str: How do solar panels impact the environment?\n",
      "assistant_response: \n",
      "  metadata: null\n",
      "  response_str: Solar panels convert sunlight into electricity by allowing photons,\n",
      "    or light particles, to knock electrons free from atoms, generating a flow of electricity.\n",
      "    Solar panels are a type of renewable energy technology that has been found to have\n",
      "    a significant positive effect on the environment by reducing the reliance on fossil\n",
      "    fuels\n",
      "user_query_timestamp: \"2021-09-01T12:00:02Z\"\n",
      "assistant_response_timestamp: \"2021-09-01T12:00:03Z\"\n",
      "metadata: null\n",
      "vector: null\n",
      "</CHAT_HISTORY>\n",
      "</SYS>\n",
      "<User>\n",
      "What are the benefits of renewable energy? Did I ask this before?\n",
      "</User>\n",
      "You:\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# lets see how the prompt will be if we pass the input_str and chat_history_str\n",
    "input_str = \"What are the benefits of renewable energy? Did I ask this before?\"\n",
    "\n",
    "\n",
    "def format_chat_history_str(turns: list) -> str:\n",
    "    chat_history_str = []\n",
    "    for turn in turns:\n",
    "        chat_history_str.append(turn.to_yaml())  # format as yaml\n",
    "    # join with newline\n",
    "    chat_history_str = \"\\n_________\\n\".join(chat_history_str)\n",
    "    return chat_history_str\n",
    "\n",
    "\n",
    "chat_history_str = format_chat_history_str(dialog_turns)\n",
    "print(generator.print_prompt(input_str=input_str, chat_history_str=chat_history_str))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "\n",
      "<SYS>\n",
      "<CHAT_HISTORY>\n",
      "user_query: \n",
      "  metadata: null\n",
      "  query_str: What are the benefits of renewable energy?\n",
      "assistant_response: \n",
      "  metadata: null\n",
      "  response_str: I can see you are interested in renewable energy. Renewable energy technologies\n",
      "    not only help in reducing greenhouse gas emissions but also contribute significantly\n",
      "    to the economy by creating jobs in the manufacturing and installation sectors. The\n",
      "    growth in renewable energy usage boosts local economies through increased investment\n",
      "    in technology and infrastructure\n",
      "</CHAT_HISTORY>\n",
      "</SYS>\n",
      "<User>\n",
      "What are the benefits of renewable energy? Did I ask this before?\n",
      "</User>\n",
      "You:\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# as we have quite a bit of empty fields, lets exclude them\n",
    "from typing import List\n",
    "\n",
    "input_str = \"What are the benefits of renewable energy? Did I ask this before?\"\n",
    "\n",
    "\n",
    "def format_chat_history_str(turns: List[DialogTurn]) -> str:\n",
    "    chat_history_str = []\n",
    "    for turn in turns:\n",
    "        chat_history_str.append(\n",
    "            turn.to_yaml(\n",
    "                exclude=[\n",
    "                    \"id\",\n",
    "                    \"user_id\",\n",
    "                    \"session_id\",\n",
    "                    \"user_query_timestamp\",\n",
    "                    \"assistant_response_timestamp\",\n",
    "                    \"order\",\n",
    "                    \"metadata\",\n",
    "                    \"vector\",\n",
    "                ],\n",
    "            )\n",
    "        )\n",
    "    chat_history_str = \"\\n_________\\n\".join(chat_history_str)\n",
    "    return chat_history_str\n",
    "\n",
    "\n",
    "chat_history_str = format_chat_history_str(dialog_turn_db.items[0:1])\n",
    "print(generator.print_prompt(input_str=input_str, chat_history_str=chat_history_str))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 4\n"
     ]
    }
   ],
   "source": [
    "print(dialog_turn_db.length, len(dialog_turn_db.transformed_items[key]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GeneratorOutput(data='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.', error=None, usage=None, raw_response='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.')\n"
     ]
    }
   ],
   "source": [
    "prompt_kwargs = {\"input_str\": input_str, \"chat_history_str\": chat_history_str}\n",
    "\n",
    "response = generator(prompt_kwargs=prompt_kwargs)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 1/1 [00:00<00:00, 1253.53it/s]\n",
      "Batch embedding documents: 100%|██████████| 1/1 [00:00<00:00,  5.00it/s]\n",
      "Adding embeddings to documents from batch: 1it [00:00, 38130.04it/s]\n"
     ]
    }
   ],
   "source": [
    "# create a turn from the last dialog\n",
    "\n",
    "new_turn = DialogTurn(\n",
    "    user_query=UserQuery(query_str=input_str),\n",
    "    assistant_response=AssistantResponse(response_str=response.data),\n",
    ")\n",
    "dialog_turn_db.add(new_turn, apply_transformer=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 6\n"
     ]
    }
   ],
   "source": [
    "print(dialog_turn_db.length, len(dialog_turn_db.transformed_items[key]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=4decc75e-ad04-4942-b88e-c3165c15f6ec, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None), Document(id=207c5220-f7bf-446a-a7ac-fa516baded64, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None), Document(id=f9ccaf5b-d251-4c89-920d-86bb82c01290, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None), Document(id=6d6739d1-d7d4-480e-9c41-bbfb085d0c2d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None), Document(id=97c0e6ab-615e-4ad8-a3bf-f6ba45a1548f, text='What are the benefits of renewable energy? Did I ask this before? Yes, you did ask about the benefit...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=0, score=None), Document(id=9cbaa3e9-bd66-4318-85bb-2c8aa6eb6694, text='in the manufacturing and installation sectors. The growth in renewable energy usage also boosts loca...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=1, score=None)]\n",
      "6f15bec8-1a24-4c1e-9835-d766b5e1950c\n",
      "Document(id=97c0e6ab-615e-4ad8-a3bf-f6ba45a1548f, text='What are the benefits of renewable energy? Did I ask this before? Yes, you did ask about the benefit...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=0, score=None)\n",
      "What are the benefits of renewable energy? Did I ask this before? Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable \n",
      "Document(id=9cbaa3e9-bd66-4318-85bb-2c8aa6eb6694, text='in the manufacturing and installation sectors. The growth in renewable energy usage also boosts loca...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=1, score=None)\n",
      "in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.\n"
     ]
    }
   ],
   "source": [
    "print(dialog_turn_db.transformed_items[key])\n",
    "\n",
    "# we will find them by comparing the id of new_turn to the parent_doc_id in the transformed_items\n",
    "new_turn_id = new_turn.id\n",
    "print(new_turn_id)\n",
    "for item in dialog_turn_db.transformed_items[key]:\n",
    "    if item.parent_doc_id == new_turn_id:\n",
    "        print(item)\n",
    "        print(item.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the conversation history is getting too long for the promp, we will use a retriever to get the most relevant part of the conversation to chat with user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from adalflow.components.retriever.faiss_retriever import FAISSRetriever\n",
    "\n",
    "retriever = FAISSRetriever(top_k=3, embedder=embedder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(id=4decc75e-ad04-4942-b88e-c3165c15f6ec, text='What are the benefits of renewable energy? I can see you are interested in renewable energy. Renewab...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=0, score=None),\n",
       " Document(id=207c5220-f7bf-446a-a7ac-fa516baded64, text='and installation sectors. The growth in renewable energy usage boosts local economies through increa...', meta_data=None, vector='len: 256', parent_doc_id=f70534d4-51a7-439d-853e-3db03f858820, order=1, score=None),\n",
       " Document(id=f9ccaf5b-d251-4c89-920d-86bb82c01290, text='How do solar panels impact the environment? Solar panels convert sunlight into electricity by allowi...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=0, score=None),\n",
       " Document(id=6d6739d1-d7d4-480e-9c41-bbfb085d0c2d, text='has been found to have a significant positive effect on the environment by reducing the reliance on ...', meta_data=None, vector='len: 256', parent_doc_id=5c596d4b-8419-4528-b5e7-4b1bd6245d18, order=1, score=None),\n",
       " Document(id=97c0e6ab-615e-4ad8-a3bf-f6ba45a1548f, text='What are the benefits of renewable energy? Did I ask this before? Yes, you did ask about the benefit...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=0, score=None),\n",
       " Document(id=9cbaa3e9-bd66-4318-85bb-2c8aa6eb6694, text='in the manufacturing and installation sectors. The growth in renewable energy usage also boosts loca...', meta_data=None, vector='len: 256', parent_doc_id=6f15bec8-1a24-4c1e-9835-d766b5e1950c, order=1, score=None)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dialog_turn_db.transformed_items[key]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.038091756, 0.08130312, 0.10887385, 0.09140143, 0.02799345, 0.016509559, -0.060777724, 0.07463354, 0.0887242, 0.083369754, 0.03921901, 0.04715675, -0.09553469, 0.05082032, 0.06218679, -0.016533043, -0.0772638, 0.027077556, 0.09825888, 0.07082906, 0.024470784, 0.075244136, 0.0071099135, 0.053262703, 0.009652103, 0.0072566913, -0.064394325, -0.02148826, 0.030036595, -0.013421356, 0.018329604, -0.021781815, -0.121085756, -0.07472748, 0.0879727, 0.012693338, -0.045747682, -0.13132498, 0.015159205, 0.018482253, 0.038467508, -0.07731077, -0.07660624, -0.06664883, -0.062233757, -0.017448938, -0.076935016, -0.023578376, 0.07200328, 0.09407865, 0.023472695, -0.056080837, -0.0056039738, -0.13677336, -0.027547246, -0.06434736, -0.007996451, 0.04652267, -0.109625354, -0.014912617, 0.12681596, -0.0016101517, 0.07120481, 0.14006118, -0.09468925, 0.07740471, -0.040087935, 0.01868187, -0.09346806, -0.005096123, 0.044620432, -0.034850907, 0.054906614, -0.034991812, 0.0056979116, -0.057771716, 0.02735937, -0.08848936, -0.0262556, 0.05758384, -0.0069161667, -0.008448526, -0.053403612, -0.053262703, -0.044949215, -0.014936103, 0.058617156, -0.03830312, -0.10840416, 0.07763955, -0.05373239, 0.007967096, 0.051196072, 0.0141846, 0.023637086, -0.022685967, 0.014372475, 0.06209285, 0.013902787, 0.10530422, 0.005557005, -0.0056509427, -0.018693613, 0.08524851, 0.090039335, -0.01460732, 0.07585473, 0.05392027, 0.04673403, -0.0024511882, -0.072426006, 0.067118526, 0.030670675, 0.06932606, 0.038091756, -0.049833976, -0.10633753, -0.038326602, 0.01106117, -0.1100011, -0.10812235, -0.027664667, 0.07787439, 0.05847625, 0.048565816, -0.014689515, -0.07176844, 0.019586021, -0.030975973, 0.058992907, -0.024329877, 0.08318188, 0.09360897, -0.11216167, -0.07857893, -0.05584599, 0.017167123, -0.016216004, -0.03752813, 0.07716986, 0.059509564, 0.0030764614, 0.029942658, 0.011067041, -0.019715184, -0.09309231, 0.017777719, -0.050726384, 0.023566633, 0.06284435, 0.035978157, 0.023425726, -0.11141017, 0.047767345, 0.019139817, -0.05932169, -0.049787007, -0.020631079, -0.051900607, -0.03731677, -0.09478319, 0.14851558, -0.066930644, -0.046076465, 0.028557075, 0.011806801, -0.04201366, -0.049364287, -0.12456146, -0.023367016, -0.047039326, -0.095393784, 0.11535556, 0.07355326, 0.0058153337, 0.056644462, -0.060355004, 0.049176414, 0.12991591, -0.06430039, -0.078344084, 0.12737958, -0.10023158, -0.05457783, 0.047555987, -0.06218679, -0.06791699, 0.01053277, -0.06932606, 0.09468925, -0.0228621, 0.036213003, 0.014125889, 0.001522085, 0.09158931, -0.02963736, -0.006399509, 0.05542327, -0.051994544, -0.06420645, 0.03943037, -0.07637139, 0.044479523, -0.003138108, -0.046170402, -0.025175316, -0.1211797, 0.03318351, 0.023167398, 0.000117513926, -0.027288916, 0.108216286, 0.0019785638, 0.16692738, 0.058570188, 0.08759695, -0.03132824, 0.051524855, -0.027382854, -0.03339487, -0.13217041, -0.08181978, 0.038044788, -4.7427555e-05, 0.031046426, 0.024963956, 0.031469148, 0.009299837, 0.12503114, 0.025151832, -0.095675595, -0.006663709, -0.0351562, -0.096192256, -0.04884763, -0.04022884, 0.045160573, -0.010908521, -0.0057125892, -0.026279086, 0.06932606, 0.08120918, 0.036377393, -0.04786128, 0.08938177, 0.056644462, -0.0039101588, -0.07825015, 0.10859204, -0.022592029, -0.031375207, 0.0064229937, 0.08346369, 0.022310216, 0.057724748, -0.031727474], [-0.07055026, 0.026867468, 0.21271624, 0.018239768, 0.08731927, 0.06647382, 0.036155272, 0.037313353, -0.015564602, 0.04734233, 0.056977563, 0.061331946, -0.07166202, 0.035692044, 0.1093228, 0.07068924, -0.010619598, -0.024319692, -0.0052316287, 0.09213689, 0.06939218, -0.022536246, 0.01539089, -0.023474293, -0.01207878, -0.021343425, -0.031129204, 0.06286061, 0.08227004, 0.015275083, 0.024481822, -0.059015784, -0.030272225, -0.03379279, 0.09042293, 0.00027486318, -0.05748712, -0.006630011, 0.05049231, -0.0058367257, 0.05957166, -0.09417511, -0.10450519, -0.070364974, -0.0736076, -0.02746967, -0.084539875, -0.09158101, 0.075228915, 0.04271001, 0.09783464, -0.07193996, 0.082594305, -0.25403655, -0.00582804, 0.00036117635, 0.09829787, 0.0754142, -0.048685703, -0.039536867, 0.2264279, 0.04067179, 0.08759721, 0.051372454, 0.028396135, 0.06929954, 0.0070237583, 0.027678125, -0.056190066, 0.0057064416, -0.0041951467, -0.06286061, 0.030967074, -0.0049971174, 0.0650378, 0.02100758, -0.032982133, -0.025222994, 0.008355551, 0.039166283, -0.031245012, -0.09653759, 0.012970502, -0.025848357, -0.004858148, 0.020243248, -0.010770149, -0.01467288, -0.093758196, 0.12794474, -0.002374065, 0.04919526, 0.11515953, 0.00019687369, -0.079398, 0.03694277, 0.024991378, -0.08583693, 0.08162151, 0.1107125, 0.012414623, -0.015622506, 0.052993763, -0.018343996, -0.014545491, -0.018668259, -0.034904547, 0.029438406, -0.027214892, 0.0061320364, -0.018008152, 0.073468626, 0.050538633, 0.042547878, 0.09292438, -0.035136163, -0.045813665, -0.01724382, 0.0036334777, -0.08477149, -0.0046525886, -0.030550165, -0.0043225354, 0.083752386, 0.0540592, 0.048083503, -0.12359035, 0.091858946, -0.05767241, 0.0037753426, -0.0033844903, 0.06749293, 0.14888284, -0.08384503, -0.038934667, -0.050584957, -0.04528095, -0.05808932, 0.010567484, -0.01136077, 0.024528146, 0.030642811, 0.018529288, 0.06401869, -0.07356127, -0.020254828, 0.03525197, -0.05063128, 0.08810677, 0.026936954, -0.038934667, -0.037938718, -0.16240922, 0.03555307, -0.030457517, -0.020034794, -0.08648545, -0.018031314, -0.016479487, 0.050816573, -0.047064394, 0.11553011, -0.015263502, -0.053595968, -0.07314437, 0.022547828, -0.10885957, -0.030017447, -0.12729621, -0.015634088, 0.016873233, -0.08847735, 0.06999439, -0.00050159363, -0.04268685, 0.115808055, -0.07745243, 0.035877336, 0.14174905, -0.015564602, 0.021204455, 0.05165039, -0.024180721, -0.05952534, 0.003940369, 0.0118240025, -0.056977563, 0.041806705, -0.00503186, 0.10144785, -0.00719168, -0.0034452896, 0.06096136, 0.066705436, 0.06684441, 0.028002387, -0.033306397, 0.07119879, -0.035761528, -0.052252594, -0.016201546, -0.0566533, 0.04604528, 0.02042854, -0.080139175, -0.03773026, -0.12192272, -0.04027804, 0.086902365, -0.0151824355, 0.019756854, 0.016792167, -0.003624792, 0.10450519, -0.004171985, 0.05938637, -0.046763293, 0.04574418, -0.033260074, -0.047851887, 0.009890008, -0.038332466, -0.011673451, -0.006821094, 0.09542584, 0.083196506, -0.048268795, -0.028558265, 0.09079351, -0.044632424, -0.037359677, 0.010602227, 0.04831512, -0.048963644, -0.08111196, 0.025593579, 0.031499792, 0.001268822, -0.06337017, 0.0018500336, 0.039629515, 0.10385666, 0.0050839735, -0.08356709, 0.018772485, 0.039305255, 0.014128583, 0.02777077, 0.124516815, 0.005179515, -0.07842521, 0.067214996, 0.08731927, 0.06457457, 0.047365494, -0.031152366], [0.012513393, 0.035173353, -0.054978263, 0.17227937, 0.10579601, -0.0017605249, -0.025069239, -0.06690791, 0.08388961, -0.042390577, 0.04597796, 0.03799656, -0.07314868, -0.008061002, 0.058671784, -0.07955928, -0.05344991, -0.017905083, 0.04064995, 0.042390577, 0.10893762, -0.009695491, 0.043048616, 0.049077123, -0.09781461, 0.017236428, -0.08563024, 0.033390272, 0.15725057, 0.019751843, -0.007015566, -0.0348974, -0.06941271, 0.008825179, 0.006262003, 0.071832605, -0.062068123, -0.08796522, 0.013511067, -0.0015270265, 0.04856767, -0.011250379, -0.016992316, -0.05646416, -0.08236126, -0.025111694, -0.027022135, -0.07921965, 0.07200242, 0.0468695, 0.06368139, -0.023371069, 0.046911955, -0.017905083, -0.0708137, 0.011430808, -0.058926508, 0.085884966, -0.16013746, 0.018510057, 0.12226826, -0.014094814, -0.02706459, 0.07866774, -0.094842814, 0.11437177, -0.02852926, 0.036192253, -0.0109001305, 0.06453047, 0.08635196, 0.0016305088, 0.120655, -0.04440715, -0.019061962, 0.039376322, 0.084908515, -0.07888001, 0.08180936, 0.05973314, 0.06839381, 0.053365003, -0.098493874, 0.031543512, -0.1017204, 0.038845647, 0.049798843, -0.019751843, -0.017692812, 0.030482156, 0.028677851, 0.012661982, -0.04678459, 0.040416453, 0.00903745, 0.049713936, -0.07217224, -0.0017446047, -0.0048795864, 0.05901142, 0.06724755, -0.022012532, -0.013075911, 0.022691801, 0.028571716, -0.08482361, -0.021556148, 0.09195592, -0.06847872, 0.028508034, -0.024114018, -0.023625795, -0.033602543, 0.054298997, 0.014784696, -0.015421509, -0.0016278555, 0.0493743, 0.035534214, -0.06962498, -0.05302537, 0.008023854, 0.07106843, 0.076035574, 0.11810774, -0.11139997, 0.033899724, -0.056209438, -0.020866267, 0.0145618105, 0.07043161, 0.08257353, 0.02706459, -0.06559183, -0.109956525, -0.122777715, 0.027022135, -0.026597593, 0.0647852, 0.03232892, 0.034961082, 0.09611644, -0.09467299, 0.07302132, -0.053195186, -0.06903062, 0.05905387, -0.028295763, 0.0086447485, 0.05731325, 0.0687759, 0.00827858, -0.023052663, 0.107069634, 0.1654867, -0.026703728, 0.10715455, 0.06376629, -0.013245728, 0.0021996612, -0.0426453, 0.037020113, -0.026279185, -0.11292832, -0.026512684, -0.020728292, 0.02608814, 0.032647323, -0.06262003, -0.008257353, -0.10121095, -0.019040734, -0.017607903, 0.04848276, 0.034557767, 0.022139896, -0.07858283, 0.022734255, 0.10401293, 0.03487617, -0.004709769, 0.102739304, -0.05128474, -0.038123924, 0.018573737, 0.0062991506, -0.033814818, 0.014689173, -0.097984426, -0.027404223, -0.031628422, 0.0066600116, -0.10010714, 0.047803495, 0.02099363, -0.017204588, 0.06461538, 0.12107954, -0.16794904, -0.01068786, 0.015517031, -0.0967108, -0.024283836, 0.033645, 0.017469928, -0.06571919, -0.023880519, -0.04703932, 0.10503183, -0.07539876, -0.0077691292, 0.13356109, 0.0046222075, 0.023349842, 0.027467905, 0.053916905, -0.02564237, 0.07162033, -0.029484482, -0.041286767, 0.0023243707, 0.004919387, 0.09713534, 0.038166378, -0.0014686519, 0.07259678, -0.05107247, 0.09144647, 0.10740927, -0.05285555, -0.13933487, -0.04525624, 0.011006266, 0.024135245, -0.088304855, -0.07166278, -0.027213179, 0.02218235, -0.07068634, -0.05323764, 0.021821488, 0.0508602, 0.057398155, 0.04767613, -0.049331848, 0.10715455, 0.035979982, -0.01834024, 0.1355989, -0.034791265, 0.0009824181, 0.04597796, -0.014922671, 0.055699985, 0.11904174, -0.080748], [0.0017427988, -0.01304154, 0.14979061, 0.10677154, 0.07750084, -0.02570888, -0.032292016, 0.07655841, 0.07639211, 0.06253287, 0.027815485, -0.044266395, -0.17373937, -0.006285163, 0.085927255, 0.027607596, 0.025251526, -0.04570776, 0.05033674, 0.04008091, 0.064195976, -0.07334307, 0.0118496455, 0.017143873, -0.007407761, 0.031765368, -0.01033206, -0.06319811, 0.09097202, -0.0742855, 0.08049443, -0.065637335, 0.038140614, 0.01577181, 0.020885868, 0.0655819, -0.052831404, -0.0108795, -0.016367756, 0.004653238, -0.0074909166, -0.023158781, 0.036311194, -0.06286549, -0.0543282, -0.03564595, -0.00048940425, -0.057100046, 0.079995506, 0.10200397, 0.011025022, -0.11198262, -0.0008402161, -0.21598232, 0.009888564, -0.038805857, 0.02609694, 0.06502753, -0.052055288, 0.06524928, 0.08032813, 0.03298498, 0.09435367, 0.10194853, -0.054577667, 0.06458403, 0.03639435, 0.089696966, -0.057709854, 0.042132076, 0.03628348, 0.025792036, 0.012050604, 0.0036623029, 0.011094318, -0.04653931, 0.10194853, -0.028064951, -0.001521051, 0.025584148, -0.006905364, 0.018848559, -0.0942428, -0.07944114, -0.053136308, -0.027593737, 0.05815335, 0.005360059, -0.1481275, 0.014067124, 0.027954077, -0.03495299, -0.036643818, 0.047537178, 0.04152227, 0.04947747, -0.030712064, 0.01431659, 0.10383339, 0.10283552, 0.041716296, -0.02773233, 0.06269918, 0.010380567, 0.037087314, 0.024655579, 0.07899764, 0.0030351724, 0.02339439, -0.0023335486, 0.089696966, -0.00959059, -0.009798479, 0.11974379, -0.09152639, -0.023311233, -0.085982695, 0.046095815, 0.03836236, -0.015522343, -0.024212083, -0.008752107, 0.025417836, 0.0118912235, 0.060038205, -0.14613177, -0.09008503, -0.037780274, -0.04512567, -0.1481275, -0.010366708, 0.11763719, 0.05396786, 0.0014725437, -0.057543542, -0.111483686, -0.043739747, -0.09984193, 0.029048957, 0.01431659, 0.055159755, 0.025140652, -0.047343146, 0.12307001, -0.115308836, -0.12916806, 0.02321422, 0.020331498, 0.10965427, 0.06513841, 0.0056476383, -0.01105967, -0.047204554, 0.0901959, 0.016977564, -0.04875679, 0.010165749, 0.030407162, -0.031820804, -0.044820767, -0.076114915, 0.045901787, -0.09756901, -0.06502753, 0.09152639, 0.07639211, -0.07506162, -0.005069015, -0.17307413, -0.017296325, -0.05163951, -0.015009551, 0.085262015, 0.015522343, -0.029187549, 0.10882271, -0.061535005, 0.03570139, 0.04676106, 0.0028515377, 0.04925572, 0.11785893, 0.03783571, 0.05865228, -0.060758885, 0.0007315077, -0.0348144, 0.05815335, -0.0018432783, 0.047010526, -0.0706821, 0.089752406, 0.025598006, 0.012348578, -0.026166236, -0.017822975, -0.0043518, 0.10499757, -0.031709928, 0.02952017, 0.04421096, -0.068797246, -0.0029520171, -0.06890812, -0.11137281, -0.04421096, -0.011170544, -0.0177814, 0.07883133, -0.06463947, 0.00133655, 0.028550023, -0.073065884, 0.089641534, 0.1430273, 0.08271191, 0.051001985, -0.035368767, 0.0119397305, -0.056656554, -0.024891187, -0.054217327, 0.15988013, -0.014372027, -0.03839008, 0.05199985, -0.077667154, 0.010352848, 0.05848597, -0.12074166, -0.05726636, 0.016894408, -0.03802974, 0.01304847, -0.024101209, -0.047731206, 0.023893321, -0.021467956, 0.03528561, 0.022618271, -0.04168858, -0.014150279, 0.004202813, -0.043379407, -0.09025134, 0.07661385, -0.02120463, 0.013339514, -0.010283552, 0.047149118, -0.03606173, 0.07362026, -0.025417836, -0.069517925, -0.008765966, -0.05654568], [0.06547112, 0.04295386, 0.12252716, 0.0344232, 8.2404214e-05, 0.018043859, -0.057194754, 0.025753822, 0.04020278, 0.012009974, 0.070649624, -0.058304437, -0.03476997, 0.047901183, 0.029591465, -0.052848507, -0.14259388, -0.0116631985, 0.05210872, 0.059321642, -0.0182866, 0.07203672, -0.036943097, 0.10181314, -0.012264275, 0.007166683, -0.039324284, -0.07370125, -0.022644406, 0.012102447, 0.008860101, -0.034723736, -0.12502395, -0.061171107, 0.13713795, -0.013917237, -0.057657123, -0.13519602, 0.072267905, -0.022713762, -0.01373229, -0.080035664, -0.09113247, -0.06278939, -0.08539912, -0.015350573, -0.10551207, -0.02716404, 0.04128934, 0.14601539, 0.029892003, -0.042445257, 0.049380757, -0.064453915, 0.022655966, -0.06440768, -0.014807292, 0.0075481357, -0.0920572, -0.0662109, 0.16182832, 0.0061899335, -0.00095579855, 0.13778526, -0.11605403, 0.05922917, -0.016541166, -0.017673966, -0.054559264, -0.00679101, 0.0076984046, 0.0014159979, 0.019743055, -0.019766174, -0.06089369, -0.05049044, 0.072684035, -0.05779583, -0.0004056545, 0.059090458, -0.018194128, -0.027210277, -0.022482578, -0.09607979, -0.034284487, -0.015466165, 0.025476402, -0.091317415, -0.10884111, 0.022066448, -0.059506588, 0.0069701774, -0.016691437, 0.05099904, 0.043855477, -0.03920869, -0.008276363, 0.060107663, 0.02952211, 0.060708743, -0.037197396, -0.072637804, -0.046213545, 0.124284156, 0.0046641235, 0.0030602892, 0.077492654, 0.07014102, 0.008680934, -0.043069452, -0.036758147, 0.020714026, 0.057657123, 0.036550082, 0.053865716, 0.008004722, -0.10699164, -0.088496976, 0.008576902, -0.080775455, -0.08507546, -0.049380757, 0.04313881, 0.062142078, 0.087294824, -0.039347403, -0.038977508, 0.066118434, -0.016529609, 0.051785067, -0.017350309, 0.07559695, 0.095432475, -0.1105981, -0.11642392, -0.025291456, 0.04877968, -0.065656066, -0.013975032, 0.034076422, 0.12086264, -0.010079593, -0.0011010106, 0.054836683, 0.019928003, -0.0446415, 0.043185044, -0.064870045, -0.011859705, 0.026262425, 0.027140923, 0.022725321, -0.111245416, 0.042445257, 0.0019448297, -0.07656792, -0.04704581, -0.035925888, -0.058165725, -0.03296674, -0.109673366, 0.14795732, -0.094230324, -0.08669375, 0.018205687, -0.049796887, -0.018228805, -0.037729118, -0.08581525, -0.08082169, -0.036411375, -0.046352256, 0.1300175, 0.029660821, 0.018933915, -0.012079328, -0.06921629, -0.018459989, 0.15304336, -0.11522177, -0.083642125, 0.16395521, -0.038029656, -0.051137753, 0.038099013, -0.08040556, -0.07111199, -0.011871263, -0.0851217, 0.08720235, 0.03021566, 0.02549952, -0.00054219714, -0.026817266, 0.058165725, -0.03645761, 0.043208163, -0.065332405, -0.049935598, -0.09108623, 0.029337164, -0.042075366, 0.013928795, 0.055946365, -0.022089567, 0.035417285, -0.06297434, 0.031672116, 0.0155933155, -0.041774828, 0.021107038, 0.045566235, 0.027926946, 0.19641335, 0.07083457, 0.040225897, -0.058997985, 0.097004525, -0.034908682, 0.0027395224, -0.12419168, -0.039994717, 0.045496877, 0.045150105, -0.02259817, 0.0004392483, 0.013408633, 0.044618383, 0.089745365, -0.010640212, -0.0021052132, 0.039971597, -0.030747382, -0.04147429, -0.006848806, -0.03747482, 0.023950592, 0.0056466525, -0.041266225, -0.03049308, 0.038954392, 0.07249909, 0.033914592, 0.04230655, 0.048502263, 0.06990984, -0.009727038, -0.06921629, 0.10662175, 0.0003032475, -0.0022338089, -0.021546286, 0.11327983, -0.010177846, 0.06352918, -0.09718947], [-0.04854006, 0.07657507, 0.20034118, 0.027323712, 0.11861532, 0.0563153, 0.051213562, 0.03377446, -0.043708127, 0.047141988, 0.022663474, 0.039415803, -0.08795586, 0.07765428, 0.12401139, 0.07726184, -0.007983724, -0.025950167, 0.0064936737, 0.07868444, 0.07858633, 0.02855009, 0.046700493, -0.01759853, -0.023681367, 0.017819278, -0.03693852, 0.039415803, 0.04760801, -0.0030536824, 0.0037128606, -0.06073026, -0.037600763, -0.077605225, 0.0644094, 0.050723013, -0.017512685, -0.039317694, 0.05450026, 0.0003746203, 0.021621052, -0.09345004, -0.11753611, -0.056217186, -0.03926864, -0.023570994, -0.07171861, -0.062496245, 0.0906539, 0.059994433, 0.10458555, -0.077703334, 0.082461685, -0.23781931, -0.024969066, -0.04044596, 0.118222885, 0.07152239, -0.07814483, -0.025999224, 0.18170023, 0.02073806, 0.08172586, 0.033014107, -0.008002119, 0.053911597, -0.037502654, 0.022737056, -0.02316629, -0.012435477, 0.014581639, -0.039293166, 0.01558727, 0.011993981, 0.06033782, -0.028819893, 0.0045222705, -0.012742072, 0.037870567, 0.06720554, -0.061760418, -0.09590279, -0.022908749, -0.031910367, -0.031419817, -0.0027639505, -0.0069167744, -0.010123754, -0.112140045, 0.14157313, 0.0051906467, 0.016629692, 0.11331737, -0.026514301, -0.061318923, 0.025189813, -0.009246893, -0.039464857, 0.056364354, 0.089967124, 0.021633316, 0.01856737, 0.079027824, 0.030463241, -0.010779867, -0.034731038, -0.0063649043, 0.006702158, -0.010007248, -0.014520321, -0.0068493234, 0.054107815, 0.024147391, 0.063820735, 0.06342829, -0.0180155, -0.056119077, -0.020848434, 0.014692014, -0.06318302, -0.013342997, -0.06288869, -0.003243771, 0.082461685, 0.046945766, 0.092174605, -0.15020683, 0.08967279, -0.07255255, -0.024539832, -0.0043720393, 0.070982784, 0.14618431, -0.11616257, -0.073190264, -0.05450026, -0.04540053, -0.071865775, 0.022491781, -0.006180947, 0.016727801, 0.045596752, 0.012024641, 0.02965383, -0.09246893, -0.04728915, 0.062937744, -0.058768056, 0.09031051, 0.028501036, -0.023190817, -0.008278054, -0.15099171, 0.046357106, -0.019045657, -0.056854904, -0.07152239, -0.026685994, 0.012558116, 0.0079898555, -0.042383637, 0.13813926, -0.015158038, -0.072111055, -0.06710743, 0.04177045, -0.075643025, -0.0363008, -0.11527957, -0.035246115, -0.007879482, -0.06431129, 0.057639785, 0.05538325, -0.006426223, 0.123717055, -0.08108814, 0.009222366, 0.08437484, 0.0218418, 0.017316464, 0.05670774, -4.553409e-05, -0.06367357, -0.013710911, -0.0001408419, -0.07888066, 0.057639785, -0.022761583, 0.099238545, -0.0048503266, 0.028795365, 0.05508892, 0.021032391, 0.08672948, 0.028623672, -0.016899494, 0.061417032, -0.020431465, -0.0758883, 0.031640563, -0.067156486, 0.07127711, 0.022994597, -0.058375616, -0.05680585, -0.14745975, -0.004488545, 0.1021328, -0.01983054, -0.024834163, 0.04208931, 0.010166678, 0.112041935, 0.016580636, 0.074416645, -0.05577569, 0.062937744, 0.014483529, -0.054254983, -0.0036760694, -0.063624516, 0.022835167, 0.011110988, 0.09237082, 0.092076495, -0.03804226, -0.03188584, 0.1136117, -0.068431914, -0.027201073, -0.025484145, -0.004727689, -0.056266245, -0.060779314, 0.009191707, 0.059062384, 0.004381237, -0.04437037, -0.017034397, 0.02592564, 0.120969966, 0.023840796, -0.07500531, 0.014299572, 0.042211946, 0.0068186643, 0.035466865, 0.11694745, -0.021461623, -0.06023971, 0.01685044, 0.090408616, 0.03823848, 0.04932494, -0.01610235]]\n",
      "2024-06-22 15:55:43 - faiss_retriever - INFO - [faiss_retriever.py:181:build_index_from_documents] - Index built with 6 chunks\n"
     ]
    }
   ],
   "source": [
    "# we will use the retriever to find top_k chunked documents, from its partent_doc_id, we will find the initial dialog_turn, and feed that to the generator\n",
    "from adalflow.utils.logger import get_logger\n",
    "\n",
    "get_logger()\n",
    "\n",
    "embeddings = [item.vector for item in dialog_turn_db.transformed_items[key]]\n",
    "print(embeddings)\n",
    "retriever.build_index_from_documents(documents=embeddings)\n",
    "\n",
    "# top_k_documents = retriever(input=input_str)\n",
    "# print(top_k_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-06-22 15:55:43 - openai_client - INFO - [openai_client.py:185:call] - api_kwargs: {'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float', 'input': ['What are the benefits of renewable energy? Did I ask this before?']}\n",
      "2024-06-22 15:55:44 - _client - INFO - [_client.py:1026:_send_single_request] - HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
      "[RetrieverOutput(doc_indices=[4, 0, 5], doc_scores=[0.9110000133514404, 0.8050000071525574, 0.6890000104904175], query='What are the benefits of renewable energy? Did I ask this before?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "top_k_documents = retriever(input=input_str)\n",
    "print(top_k_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'6f15bec8-1a24-4c1e-9835-d766b5e1950c', 'f70534d4-51a7-439d-853e-3db03f858820'}\n"
     ]
    }
   ],
   "source": [
    "# get the parent_doc_id from the top_k_documents\n",
    "\n",
    "parent_doc_ids = set(\n",
    "    [\n",
    "        dialog_turn_db.transformed_items[key][doc_index].parent_doc_id\n",
    "        for doc_index in top_k_documents[0].doc_indices\n",
    "    ]\n",
    ")\n",
    "print(parent_doc_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "fetched_dialog_turns = dialog_turn_db.fetch_items(\n",
    "    condition=lambda x: x.id in parent_doc_ids\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2024-06-22 16:01:56 - generator - INFO - [generator.py:224:call] - prompt_kwargs: {'input_str': 'What are the benefits of renewable energy? Did I ask this before?', 'chat_history_str': 'user_query: \\n  metadata: null\\n  query_str: What are the benefits of renewable energy?\\nassistant_response: \\n  metadata: null\\n  response_str: I can see you are interested in renewable energy. Renewable energy technologies\\n    not only help in reducing greenhouse gas emissions but also contribute significantly\\n    to the economy by creating jobs in the manufacturing and installation sectors. The\\n    growth in renewable energy usage boosts local economies through increased investment\\n    in technology and infrastructure\\n_________\\nuser_query: \\n  metadata: null\\n  query_str: What are the benefits of renewable energy? Did I ask this before?\\nassistant_response: \\n  metadata: null\\n  response_str: Yes, you did ask about the benefits of renewable energy before. Renewable\\n    energy technologies help in reducing greenhouse gas emissions and contribute to\\n    the economy by creating jobs in the manufacturing and installation sectors. The\\n    growth in renewable energy usage also boosts local economies through increased investment\\n    in technology and infrastructure'}\n",
      "2024-06-22 16:01:56 - generator - INFO - [generator.py:225:call] - model_kwargs: {}\n",
      "2024-06-22 16:01:56 - openai_client - INFO - [openai_client.py:185:call] - api_kwargs: {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': '<SYS>\\n<CHAT_HISTORY>\\nuser_query: \\n  metadata: null\\n  query_str: What are the benefits of renewable energy?\\nassistant_response: \\n  metadata: null\\n  response_str: I can see you are interested in renewable energy. Renewable energy technologies\\n    not only help in reducing greenhouse gas emissions but also contribute significantly\\n    to the economy by creating jobs in the manufacturing and installation sectors. The\\n    growth in renewable energy usage boosts local economies through increased investment\\n    in technology and infrastructure\\n_________\\nuser_query: \\n  metadata: null\\n  query_str: What are the benefits of renewable energy? Did I ask this before?\\nassistant_response: \\n  metadata: null\\n  response_str: Yes, you did ask about the benefits of renewable energy before. Renewable\\n    energy technologies help in reducing greenhouse gas emissions and contribute to\\n    the economy by creating jobs in the manufacturing and installation sectors. The\\n    growth in renewable energy usage also boosts local economies through increased investment\\n    in technology and infrastructure\\n</CHAT_HISTORY>\\n</SYS>\\n<User>\\nWhat are the benefits of renewable energy? Did I ask this before?\\n</User>\\nYou:'}]}\n",
      "2024-06-22 16:01:57 - _client - INFO - [_client.py:1026:_send_single_request] - HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "2024-06-22 16:01:57 - generator - INFO - [generator.py:233:call] - output: GeneratorOutput(data='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.', error=None, usage=None, raw_response='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.')\n",
      "GeneratorOutput(data='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.', error=None, usage=None, raw_response='Yes, you did ask about the benefits of renewable energy before. Renewable energy technologies help in reducing greenhouse gas emissions and contribute to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage also boosts local economies through increased investment in technology and infrastructure.')\n"
     ]
    }
   ],
   "source": [
    "chat_history_str = format_chat_history_str(fetched_dialog_turns)\n",
    "\n",
    "output = generator(\n",
    "    prompt_kwargs={\"input_str\": input_str, \"chat_history_str\": chat_history_str}\n",
    ")\n",
    "print(output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "my-project-kernel",
   "language": "python",
   "name": "my-project-kernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}