Why DSPy is the Future of LLMs Programming (And How to Get Started!).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Rawan Alkurd** \n",
    "*Senior Data Scientist*  \n",
    "[My GitHub Profile](https://github.com/rawanalkurd)  \n",
    "[My LinkedIn Profile](https://linkedin.com/in/rawan-alkurd/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why DSPy is the Future of LLMs Programming (And How to Get Started!)\n",
    "\n",
    "Imagine a world where interacting with large language models (LLMs) is as intuitive and efficient as programming with high-level abstractions. Picture this: You're working on an AI project late at night, wrestling with countless prompts and tweaks to get your language model to understand your commands. You wish there was a way to simplify this process, to make it less about guessing and more about clear, structured programming. Welcome to DSPy, the Declarative Self-Improving Language Model Programs framework developed by Stanford University. DSPy is revolutionizing how we build and optimize AI applications, making traditional prompt engineering a thing of the past.\n",
    "\n",
    "## The Tale of Traditional Prompt Engineering\n",
    "A few months ago, prompt engineering was the buzzword in AI circles. The idea was to craft specific prompts to get the best responses from LLMs. However, this approach quickly revealed its limitations: it's inconsistent, labor-intensive, and relies heavily on trial and error. As the complexity of tasks grows, maintaining these prompts becomes a daunting task. Imagine trying to cook a perfect meal without a recipe, just by trial and error, sometimes it works, often it doesn't! This is where DSPy comes into play, offering a structured, scalable, and efficient solution.\n",
    "\n",
    "## What is DSPy?\n",
    "DSPy stands for Declarative Self-Improving Language Model Programs. Unlike traditional prompt engineering, DSPy provides a systematic approach to programming LLMs, treating them as modular components within a self-improving pipeline. The framework is built around three key components: signatures, modules, and teleprompters (optimizers). The framework transforms the chaotic world of prompt engineering into a structured programming model. Think of DSPy as a master chef who not only cooks the perfect dish but also continuously improves the recipe based on feedback. This approach abstracts away the guesswork, allowing you to focus on defining what you want rather than how to achieve it.\n",
    "\n",
    "## How DSPy Compares to Other Frameworks?\n",
    "In the ever-evolving landscape of large language model (LLM) programming, various frameworks offer distinct approaches to enhance and streamline interactions with LLMs. DSPy stands out with its unique methodology and robust feature set. It offers a unique blend of high-level abstractions, automated optimization, and ease of use, making it an excellent choice for developers looking to simplify and enhance their LLM programming workflows. While other frameworks like Guidance, LMQL, Marvin, Instructor, and LangChain have their strengths, DSPy's specialized focus on LLMs and self-improving pipelines sets it apart as a powerful tool in the AI developer's toolkit.\n",
    " Here's how DSPy compares to other prominent frameworks:\n",
    "\n",
    "### 1. Controlled Generation\n",
    "\n",
    "Frameworks like Guidance and LMQL:\n",
    "Strengths: Use template-based and constraint-based techniques to control LLM outputs, ensuring they meet specific requirements.\n",
    "Limitations: Require manual setup and are more suitable for completion tasks rather than conversational models like GPT-4.\n",
    "\n",
    "DSPy:\n",
    "Strengths: Automates prompt optimization using teleprompters, reducing manual effort and providing consistent results.\n",
    "Limitations: May introduce computational overhead due to continuous optimization.\n",
    "\n",
    "### 2. Schema-Driven Generation\n",
    "Frameworks like Marvin and Instructor:\n",
    "Strengths: Utilize schemas to ensure outputs conform to predefined structures, simplifying integration with existing programming logic.\n",
    "Limitations: Can be less flexible in controlling the type of content generated.\n",
    "\n",
    "DSPy:\n",
    "Strengths: Allows for high-level task definitions using natural language signatures, supporting a wide range of tasks with customizable modules.\n",
    "Limitations: Learning curve for new users to understand and effectively use the high-level abstractions.\n",
    "\n",
    "### 3. Compiling LM Programs\n",
    "Frameworks like LangChain:\n",
    "Strengths: Provide pre-packaged utilities for rapid prototyping with straightforward prompt generation and parsing tools.\n",
    "Limitations: Limited customization options and reliance on manual prompt engineering.\n",
    "\n",
    "DSPy:\n",
    "Strengths: Offers modular design and advanced optimization strategies to build and refine complex pipelines.\n",
    "Limitations: May not be as intuitive for developers accustomed to more traditional frameworks.\n",
    "\n",
    "### 4. Open-Ended Agents\n",
    "Frameworks like AutoGPT and MetaGPT:\n",
    "Strengths: Use pre-built pipelines to perform tasks with minimal human intervention, offering convenience.\n",
    "Limitations: Fixed prompts and capabilities can limit extensibility and adaptability.\n",
    "\n",
    "DSPy:\n",
    "Strengths: Enables the creation of highly customized and extensible pipelines with continuous self-improvement.\n",
    "Limitations: Dependency on the quality and quantity of training examples for optimization effectiveness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Components of DSPy\n",
    "Let's walk through the essential DSPy components needed to build a programmable LLM pipeline step by step. First, we'll set up our language model. For this tutorial, I'll be using Gemini Pro, but feel free to use any language model that suits your needs. If you need guidance on setting up an LLM locally, you can refer to my previous blog here:\n",
    "https://medium.com/@rawanalkurd/running-llms-locally-a-guide-to-setting-up-ollama-with-docker-6ef8488e75d4\n",
    "\n",
    "Here is how to setup our LLM to be used with DSPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ralkurd\\Anaconda3\\envs\\SGPT\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import dspy\n",
    "temperature=0 # set your temperature here\n",
    "model = dspy.Google(\"models/gemini-1.0-pro\",\n",
    "        api_key=API_KEY, # Use your API Key\n",
    "        temperature=temperature,\n",
    "        max_output_tokens=3024)\n",
    "\n",
    "dspy.settings.configure(lm=model, max_tokens=3024) # this is used to set the model to be used globally across all DSPy tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Signatures\n",
    "Signatures in DSPy are like the blueprints of desired behavior. They define the input/output relationship of a task in natural language, abstracting away the complexities of how to achieve it.\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sig_1 = dspy.Signature(\"question -> answer\")\n",
    "sig_2 = dspy.Signature(\"document -> summary\")\n",
    "sig_3 = dspy.Signature(\"question, context -> answer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StringSignature(question, context -> answer\n",
       "    instructions='Given the fields `question`, `context`, produce the fields `answer`.'\n",
       "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
       "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
       "    answer = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Answer:', 'desc': '${answer}'})\n",
       ")"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here are the components of a signature:\n",
    "sig_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown above, the DSPy signature consists of three main components:  instructions, inputs, and outputs. You can use the built in signature or write your own signature as shown in the ExtractActionItems example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExtractActionItems(dspy.Signature):\n",
    "    \"\"\"\n",
    "    Extract the action items from the provided text and return them as a Python list.    \n",
    "    \"\"\"\n",
    "    Text = dspy.InputField(desc=\"Text\")\n",
    "    Extracted_action_items = dspy.OutputField(desc=\"Python list of action items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add as many signature as you wish and customize them based on your problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Modules\n",
    "DSPy modules are essential components designed to simplify and enhance the process of programming large language models (LLMs). These modules provide pre-built functionalities that can be combined to handle complex tasks efficiently. Let's delve into the various types of DSPy modules, their specific uses, and practical examples to illustrate their application.\n",
    "Types of DSPy Modules\n",
    "Predict Module\n",
    "ChainOfThought (CoT) Module\n",
    "ProgramOfThought (PoT) Module\n",
    "Task-Specific Modules\n",
    "\n",
    "There are other advanced types of DSPy modules such as ReAct, MultiChainComparison and Majority. I will leave these to another blog. Stay Tuned! \n",
    "### 1. Predict Module:\n",
    "Purpose: The Predict module is the foundational component that leverages a given Signature to generate prompts and produce results. It serves as a straightforward bridge between the Signature and the final output.\n",
    "Usage:\n",
    "Extracting Specific Information: The Predict module can be used to extract action items from a text, summarize content, or generate specific outputs based on predefined Signatures.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Extract the action items from the provided text and return them as a Python list.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Text: Text\n",
      "Extracted Action Items: Python list of action items\n",
      "\n",
      "---\n",
      "\n",
      "Text: In our meeting today, we discussed the progress of Project A. It was noted that there are several blockers that need attention. John will follow up with the IT department to resolve the server issues. Sarah will prepare the financial report for the next quarter by Friday. Additionally, we need to organize a team-building activity next month to boost morale. Lastly, review the new project proposals by next Wednesday.\n",
      "Extracted Action Items:\n"
     ]
    }
   ],
   "source": [
    "# Defining and using the ExtractActionItems Signature\n",
    "class ExtractActionItems(dspy.Signature):\n",
    "    \"\"\"\n",
    "    Extract the action items from the provided text and return them as a Python list.    \n",
    "    \"\"\"\n",
    "    Text = dspy.InputField(desc=\"Text\")\n",
    "    Extracted_action_items = dspy.OutputField(desc=\"Python list of action items\")\n",
    " \n",
    "# Extracting action items\n",
    "Text = \"In our meeting today, we discussed the progress of Project A. It was noted that there are several blockers that need attention. John will follow up with the IT department to resolve the server issues. Sarah will prepare the financial report for the next quarter by Friday. Additionally, we need to organize a team-building activity next month to boost morale. Lastly, review the new project proposals by next Wednesday.\"\n",
    "action_items_extractor = dspy.Predict(ExtractActionItems)\n",
    "response = action_items_extractor(Text=Text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets inspect the output of this program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Extract the action items from the provided text and return them as a Python list.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Text: Text\n",
      "Extracted Action Items: Python list of action items\n",
      "\n",
      "---\n",
      "\n",
      "Text: In our meeting today, we discussed the progress of Project A. It was noted that there are several blockers that need attention. John will follow up with the IT department to resolve the server issues. Sarah will prepare the financial report for the next quarter by Friday. Additionally, we need to organize a team-building activity next month to boost morale. Lastly, review the new project proposals by next Wednesday.\n",
      "Extracted Action Items:- John will follow up with the IT department to resolve the server issues.\n",
      "- Sarah will prepare the financial report for the next quarter by Friday.\n",
      "- Organize a team-building activity next month to boost morale.\n",
      "- Review the new project proposals by next Wednesday.\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\n\\n\\nExtract the action items from the provided text and return them as a Python list.\\n\\n---\\n\\nFollow the following format.\\n\\nText: Text\\nExtracted Action Items: Python list of action items\\n\\n---\\n\\nText: In our meeting today, we discussed the progress of Project A. It was noted that there are several blockers that need attention. John will follow up with the IT department to resolve the server issues. Sarah will prepare the financial report for the next quarter by Friday. Additionally, we need to organize a team-building activity next month to boost morale. Lastly, review the new project proposals by next Wednesday.\\nExtracted Action Items:\\x1b[32m- John will follow up with the IT department to resolve the server issues.\\n- Sarah will prepare the financial report for the next quarter by Friday.\\n- Organize a team-building activity next month to boost morale.\\n- Review the new project proposals by next Wednesday.\\x1b[0m\\n\\n\\n'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.inspect_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how the LLM was unable to produce the output in the correct format. In a few minutes, we will see how DSPy can optimize the program to improve the output and align it with specific metrics to meet certain expectations or performance criteria."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. ChainOfThought (CoT) Module\n",
    "Purpose: The ChainOfThought module is designed for tasks requiring logical reasoning or multi-step processing. It structures prompts to guide the model through a step-by-step reasoning process, improving the depth and accuracy of responses.\n",
    "Usage:\n",
    "Logical Reasoning: Ideal for tasks that need to follow a logical sequence of steps.\n",
    "Multi-Step Processing: Useful for breaking down complex tasks into manageable steps.\n",
    "\n",
    "Example:\n",
    "For this example, we will define the signature directly within the definition of the Chain of Thought (CoT) program. This is an alternative to explicitly defining the signature separately. Note that the words used for the inputs and outputs are crucial as they guide the LLM to produce reasonable answers. For example, if you have a question-answer problem, your input should be labeled as 'question' and your output as 'answer,' as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `question`, produce the fields `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    }
   ],
   "source": [
    "QA_program = dspy.ChainOfThought('question -> answer')\n",
    "question =\"If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\"\n",
    "\n",
    "response = QA_program(question=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let inspect our output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Given the fields `question`, produce the fields `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "Reasoning: Let's think step by step in order toQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "Reasoning: Let's think step by step in order to produce the answer. First, we start with 10 oranges. Then, we eat 3 oranges, which leaves us with 10 - 3 = 7 oranges. Finally, we buy 7 more oranges, which gives us 7 + 7 = 14 oranges.\n",
      "Answer: 14\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"\\n\\n\\nGiven the fields `question`, produce the fields `answer`.\\n\\n---\\n\\nFollow the following format.\\n\\nQuestion: ${question}\\nReasoning: Let's think step by step in order to ${produce the answer}. We ...\\nAnswer: ${answer}\\n\\n---\\n\\nQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\\nReasoning: Let's think step by step in order to\\x1b[32mQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\\nReasoning: Let's think step by step in order to produce the answer. First, we start with 10 oranges. Then, we eat 3 oranges, which leaves us with 10 - 3 = 7 oranges. Finally, we buy 7 more oranges, which gives us 7 + 7 = 14 oranges.\\nAnswer: 14\\x1b[0m\\n\\n\\n\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.inspect_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how the chain of thought reasoning helped the LLM to logically deduce and find the correct answer. The CoT module not only returns the requested answer but also provides the reasoning behind each step, enhancing transparency and understanding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. ProgramOfThought (PoT) Module\n",
    "Purpose: The ProgramOfThought module extends the capabilities of the CoT module by allowing for more complex, programmable thought processes. It's suited for tasks that require a higher level of abstraction and control over the reasoning process.\n",
    "Usage:\n",
    "Complex Programming Tasks: Suitable for scenarios where the model needs to handle intricate logic and reasoning.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  You will be given `question` and you will respond with `answer`.\n",
      "Generating executable Python code that programmatically computes the correct `answer`.\n",
      "After you're done with the computation, make sure the last line in your code evaluates to the correct value for `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "Reasoning: Let's think step by step in order to ${produce the generated_code}. We ...\n",
      "Code: python code that answers the question\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "Reasoning: Let's think step by step in order to\n",
      "prompt: /n  You will be given `question` and you will respond with `answer`.\n",
      "Generating executable Python code that programmatically computes the correct `answer`.\n",
      "After you're done with the computation, make sure the last line in your code evaluates to the correct value for `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "Reasoning: Let's think step by step in order to ${produce the generated_code}. We ...\n",
      "Code: python code that answers the question\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "Reasoning: Let's think step by step in order to ```python oranges_initial = 10 oranges_eaten = 3 oranges_bought = 7 oranges_now = oranges_initial - oranges_eaten + oranges_bought answer = oranges_now ```\n",
      "Code:\n",
      "prompt: /n  Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "\n",
      "Code: python code that answers the question\n",
      "\n",
      "Code Output: output of previously-generated python code\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "\n",
      "Code:\n",
      "oranges_initial = 10\n",
      "oranges_eaten = 3\n",
      "oranges_bought = 7\n",
      "oranges_now = oranges_initial - oranges_eaten + oranges_bought\n",
      "answer = oranges_now\n",
      "answer\n",
      "\n",
      "Code Output: 14\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    }
   ],
   "source": [
    "# Define a signature for question answering\n",
    "generate_answer_signature = dspy.Signature(\"question -> answer\")\n",
    "\n",
    "# Pass signature to ProgramOfThought Module\n",
    "pot = dspy.ProgramOfThought(generate_answer_signature)\n",
    "\n",
    "# Call the ProgramOfThought module\n",
    "question = \"If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\"\n",
    "result = pot(question=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let inspect our output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Given the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "\n",
      "Code: python code that answers the question\n",
      "\n",
      "Code Output: output of previously-generated python code\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Question: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "\n",
      "Code:\n",
      "oranges_initial = 10\n",
      "oranges_eaten = 3\n",
      "oranges_bought = 7\n",
      "oranges_now = oranges_initial - oranges_eaten + oranges_bought\n",
      "answer = oranges_now\n",
      "answer\n",
      "\n",
      "Code Output: 14\n",
      "\n",
      "Reasoning: Let's think step by step in order toQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\n",
      "\n",
      "Code:\n",
      "oranges_initial = 10\n",
      "oranges_eaten = 3\n",
      "oranges_bought = 7\n",
      "oranges_now = oranges_initial - oranges_eaten + oranges_bought\n",
      "answer = oranges_now\n",
      "answer\n",
      "\n",
      "Code Output: 14\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the answer.\n",
      "1. We start with 10 oranges.\n",
      "2. We eat 3 oranges, so we have 10 - 3 = 7 oranges left.\n",
      "3. We buy 7 more oranges, so we have 7 + 7 = 14 oranges now.\n",
      "\n",
      "Answer: 14\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"\\n\\n\\nGiven the final code `question`, `final_generated_code`, `code_output`, provide the final `answer`.\\n\\n---\\n\\nFollow the following format.\\n\\nQuestion: ${question}\\n\\nCode: python code that answers the question\\n\\nCode Output: output of previously-generated python code\\n\\nReasoning: Let's think step by step in order to ${produce the answer}. We ...\\n\\nAnswer: ${answer}\\n\\n---\\n\\nQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\\n\\nCode:\\noranges_initial = 10\\noranges_eaten = 3\\noranges_bought = 7\\noranges_now = oranges_initial - oranges_eaten + oranges_bought\\nanswer = oranges_now\\nanswer\\n\\nCode Output: 14\\n\\nReasoning: Let's think step by step in order to\\x1b[32mQuestion: If you have 10 oranges and you eat 3, then buy 7 more, how many oranges do you have now?\\n\\nCode:\\noranges_initial = 10\\noranges_eaten = 3\\noranges_bought = 7\\noranges_now = oranges_initial - oranges_eaten + oranges_bought\\nanswer = oranges_now\\nanswer\\n\\nCode Output: 14\\n\\nReasoning: Let's think step by step in order to produce the answer.\\n1. We start with 10 oranges.\\n2. We eat 3 oranges, so we have 10 - 3 = 7 oranges left.\\n3. We buy 7 more oranges, so we have 7 + 7 = 14 oranges now.\\n\\nAnswer: 14\\x1b[0m\\n\\n\\n\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.inspect_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how the LLM produced the answer, the code used to generate the answer, and the reasoning employed to reach the correct answer.\n",
    "\n",
    "### 4. Task-Specific Modules\n",
    "Purpose: These modules are tailored for specific tasks and applications. They leverage the strengths of DSPy to provide optimized solutions for particular use cases.\n",
    "Usage:\n",
    "Custom Tasks: Create modules for unique tasks such as document classification, sentiment analysis, or any other specific application.\n",
    "\n",
    "Example: To avoid repetition, refer to the use case in the last section of the blog. The use case demonstrates how to build a customized module tailored specifically for the task.\n",
    "DSPy modules offer a powerful and flexible way to handle a variety of tasks in AI programming. By leveraging these modules, developers can simplify the process of interacting with large language models, making it more efficient and effective. By embracing the modular approach of DSPy, you're not just writing prompts; you're programming intelligent, adaptable systems capable of tackling intricate tasks with ease."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Teleprompters (Optimizers)\n",
    "Teleprompters are the optimizers in DSPy, guiding the learning and optimization of modules. They improve the performance of LLM pipelines by refining prompts and learning from data. Here's a look at the different types of DSPy optimizers and how to effectively use them.\n",
    "Types of DSPy Optimizers\n",
    "### 1. Automatic Few-Shot Learning Optimizers\n",
    "These optimizers extend the signature by generating and including optimized examples within the prompt, implementing few-shot learning.\n",
    "LabeledFewShot: Constructs few-shot examples from labeled input-output data points. This optimizer uses existing labeled examples to create prompts that help the model understand the task with minimal examples.\n",
    "BootstrapFewShot: Uses a teacher module to generate complete demonstrations for every stage of your program. This optimizer generates examples using a more knowledgeable model (teacher) to guide the less knowledgeable one (student).\n",
    "BootstrapFewShotWithRandomSearch: Extends BootstrapFewShot by applying random search over generated demonstrations. This optimizer adds a layer of random search to find the best possible few-shot examples, improving the robustness of the generated prompts.\n",
    "BootstrapFewShotWithOptuna: Uses Optuna optimization across demonstration sets. This optimizer integrates Optuna, a hyperparameter optimization framework, to enhance the selection of few-shot examples.\n",
    "KNNFewShot: Selects demonstrations through the k-Nearest Neighbors algorithm. This optimizer uses the k-Nearest Neighbors algorithm to find the most relevant examples from the dataset to use as few-shot demonstrations.\n",
    "\n",
    "### 2. Automatic Instruction Optimization Optimizers\n",
    "These optimizers generate optimal instructions for the prompt.\n",
    "COPRO: Generates and refines new instructions using coordinate ascent. COPRO uses a method called coordinate ascent to iteratively improve the instructions provided to the model.\n",
    "MIPRO: Generates instructions and few-shot examples using Bayesian Optimization. MIPRO employs Bayesian Optimization to fine-tune both the instructions and the few-shot examples, balancing exploration and exploitation to find the best configurations.\n",
    "\n",
    "### 3. Automatic Finetuning Optimizers\n",
    "BootstrapFinetune: Distills a prompt-based DSPy program into weight updates for smaller LMs. This optimizer takes a program optimized for a large model and fine-tunes it to work effectively on a smaller model by updating the model weights.\n",
    "### 4. Program Transformation Optimizers\n",
    "Ensemble: Combines a set of DSPy programs into a single program.The Ensemble optimizer aggregates multiple DSPy programs to create a more robust and reliable single program, leveraging the strengths of each.\n",
    "DSPy optimizer is a topic worth exploring in depth. MIPRO is a very interesting tool that can significantly improve the performance of your LLM. I'll address MIPRO and other optimizers in a separate blog, but for now, to get you started, I'll show you how to use BootstrapFewShot. In the upcoming use case example, I'll demonstrate how to use this optimizer and how it can enhance the outcome of your LLM.\n",
    "\n",
    "## DSPy Workflow\n",
    "The DSPy framework simplifies the process of building and optimizing language model pipelines. By breaking down the development process into clear, manageable steps, DSPy ensures that you can focus on defining the tasks and metrics that matter most. Here's a step-by-step guide to the DSPy workflow:\n",
    "- Define the Task: Outline the task and metrics to optimize.\n",
    "- Assemble Examples: Collect input examples, with or without labels.\n",
    "- Construct the Pipeline: Choose built-in layers (modules) and assign input/output specifications.\n",
    "- Optimize: Use a DSPy optimizer to transform the code into high-quality instructions and generate automatic few-shot examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Case: Structured Feedback Summarization with DSPy\n",
    "Scenario\n",
    "Imagine you're receiving tons of unstructured feedback from users about your app. Some love the new features, others are frustrated with bugs, and a few offer suggestions for improvements. You need a way to make sense of all this data quickly and efficiently. Lets explore how we can use DSPy, our new best friend for transforming chaos into clarity.\n",
    "### 1. Setting Up the Environment:\n",
    "Here, we set up DSPy and configure our language model (we use Google's Gemini model) with the API key and some settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy\n",
    "from dspy.teleprompt import BootstrapFewShot\n",
    "from dspy.evaluate.evaluate import Evaluate\n",
    "from rouge_score import rouge_scorer\n",
    "from typing import List, Dict, Any\n",
    "temperature=0 # set your temprature here\n",
    "model = dspy.Google(\"models/gemini-1.0-pro\",\n",
    "        api_key=API_KEY, # Use your API Key\n",
    "        temperature=temperature,\n",
    "        max_output_tokens=3024)\n",
    "# this is used to set the model to be used globally across all DSPy tasks\n",
    "dspy.settings.configure(lm=model, max_tokens=3024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Define the Task with Signatures\n",
    "Think of signatures as blueprints. They outline what you want to achieve without worrying about the nitty-gritty details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedbackSummary(dspy.Signature):\n",
    "    feedback_text = dspy.InputField()\n",
    "    sentiment = dspy.OutputField()\n",
    "    topics = dspy.OutputField()\n",
    "    action_items = dspy.OutputField()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, FeedbackSummary is a blueprint with an input for feedback_text and outputs for sentiment, topics, and action_items.\n",
    "\n",
    "### 2. Create Modules for Processing\n",
    "Modules are like little workers following the blueprint to get the job done. In this case, we create a module to process the feedback and generate structured summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SummarizeFeedback(dspy.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.summary = dspy.ChainOfThought(FeedbackSummary)\n",
    "\n",
    "    def forward(self, feedback_text):\n",
    "        # You can do sort of post processing you need her to return the output you need from you program. \n",
    "        # I am just going to return the output of the summary program \n",
    "        return self.summary(feedback_text=feedback_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our SummarizeFeedback module takes feedback text and generates structured outputs based on our blueprint.\n",
    "\n",
    "### 3. Add Training and Validation Examples\n",
    "Here, I added some training and validation examples to teach the model what good summaries look like. These examples serve as a guide, showing the model the desired output format and content. By providing feedback texts along with the expected summary components, you help the model learn the desired structure and details. The examples include feedback texts with corresponding sentiments, key topics, and action items. The full dataset can be found in the notebook on my repository here: [[link to repo](https://github.com/rawanalkurd/Generative-AI-DSPy)]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training examples\n",
    "def get_examples() -> list[dspy.Example]:\n",
    "    return [\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The new feature is great, but it needs more documentation.\",\n",
    "            sentiment=\"positive\",\n",
    "            topics=\"new feature documentation\",\n",
    "            action_items=\"improve documentation\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The app is user-friendly, but it crashes frequently.\",\n",
    "            sentiment=\"negative\",\n",
    "            topics=\"usability stability\",\n",
    "            action_items=\"fix crash issues\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"Customer support was very helpful, but the response time was slow.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"customer support response time\",\n",
    "            action_items=\"reduce response time\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"I love the design of the app, but there are some bugs that need to be addressed.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"design bugs\",\n",
    "            action_items=\"fix bugs\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The performance has improved significantly, but the UI could be more intuitive.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"performance UI\",\n",
    "            action_items=\"improve UI\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The pricing is reasonable, but the free trial period is too short.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"pricing trial period\",\n",
    "            action_items=\"extend trial period\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The installation process was straightforward, but the app is slow.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"installation performance\",\n",
    "            action_items=\"improve performance\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The customer support solved my issue quickly, but the initial wait time was long.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"customer support wait time\",\n",
    "            action_items=\"reduce wait time\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The recent update added useful features, but it also introduced new bugs.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"update bugs\",\n",
    "            action_items=\"fix new bugs\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The app's search functionality is excellent, but it needs more filters.\",\n",
    "            sentiment=\"positive\",\n",
    "            topics=\"search functionality filters\",\n",
    "            action_items=\"add more filters\"\n",
    "        ).with_inputs(\"feedback_text\")\n",
    "    ]\n",
    "\n",
    "# Validation examples\n",
    "def get_validation_examples() -> list[dspy.Example]:\n",
    "    return [\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The software is fast, but the interface is not intuitive.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"performance interface\",\n",
    "            action_items=\"improve interface\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"I appreciate the quick updates, but the app is still buggy.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"updates bugs\",\n",
    "            action_items=\"fix bugs\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The customer service is excellent, but the product quality needs improvement.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"customer service product quality\",\n",
    "            action_items=\"improve product quality\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The app has many features, but it's overwhelming to new users.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"features usability\",\n",
    "            action_items=\"improve usability for new users\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The recent changes are beneficial, but the app's speed has decreased.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"changes speed\",\n",
    "            action_items=\"improve speed\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The design is appealing, but the app crashes on older devices.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"design stability\",\n",
    "            action_items=\"fix crashes on older devices\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The tutorials are helpful, but there should be more advanced guides.\",\n",
    "            sentiment=\"positive\",\n",
    "            topics=\"tutorials guides\",\n",
    "            action_items=\"add advanced guides\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The app integrates well with other tools, but the synchronization is slow.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"integration synchronization\",\n",
    "            action_items=\"improve synchronization speed\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The pricing is competitive, but the subscription management is confusing.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"pricing subscription management\",\n",
    "            action_items=\"simplify subscription management\"\n",
    "        ).with_inputs(\"feedback_text\"),\n",
    "        dspy.Example(\n",
    "            feedback_text=\"The app is intuitive, but it lacks support for some file formats.\",\n",
    "            sentiment=\"mixed\",\n",
    "            topics=\"usability file formats\",\n",
    "            action_items=\"add support for more file formats\"\n",
    "        ).with_inputs(\"feedback_text\")\n",
    "    ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Evaluating Model Performance: \n",
    "Next, we set up a metric to evaluate the performance of our model. This function is crucial because it checks whether the generated summary includes all necessary components: sentiment, topics, and action items. It also assesses how well the generated output matches the expected output. I chose to use ROUGE scores for this comparison. There are many other metrics you can choose from, or you can create your own metric depending on your needs and use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rouge_score import rouge_scorer\n",
    "from typing import Dict, Any\n",
    "\n",
    "def calculate_rouge_score(prediction: str, reference: str) -> float:\n",
    "    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)\n",
    "    scores = scorer.score(reference, prediction)\n",
    "    return (scores['rouge1'].fmeasure + scores['rouge2'].fmeasure + scores['rougeL'].fmeasure) / 3\n",
    "\n",
    "def evaluate_summary(example: Dict[str, Any], pred: Dict[str, Any], trace=None) -> float:\n",
    "    if not pred or not all(key in pred for key in [\"sentiment\", \"topics\", \"action_items\"]):\n",
    "        return 0.0\n",
    "    \n",
    "    expected_sentiment = example.get(\"sentiment\")\n",
    "    expected_topics = example.get(\"topics\", [])\n",
    "    expected_action_items = example.get(\"action_items\", [])\n",
    "    \n",
    "    predicted_sentiment = pred.get(\"sentiment\")\n",
    "    predicted_topics = pred.get(\"topics\", [])\n",
    "    predicted_action_items = pred.get(\"action_items\", [])\n",
    "    \n",
    "    sentiment_match = 1.0 if predicted_sentiment == expected_sentiment else 0.0\n",
    "    \n",
    "    topics_rouge_score = calculate_rouge_score(predicted_topics, expected_topics)\n",
    "    action_items_rouge_score = calculate_rouge_score(predicted_action_items, expected_action_items)\n",
    "    \n",
    "    return 0.2 * sentiment_match + 0.4 * topics_rouge_score + 0.4 * action_items_rouge_score\n",
    "\n",
    "def evaluate_model(model, examples):\n",
    "    from dspy.evaluate.evaluate import Evaluate\n",
    "    evaluator = Evaluate(metric=evaluate_summary, devset=examples, display_progress=True, display_table=1)\n",
    "    evaluation = evaluator(model)\n",
    "    return evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Initial Model Performance: \n",
    "Before we start optimizing our LLM performance, we need baseline performance of our summarizer before optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The software is fast, but the interface is not intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.2111111111111111 / 1  (21.1):  10%|█         | 1/10 [00:02<00:25,  2.85s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I appreciate the quick updates, but the app is still buggy.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.45777777777777784 / 2  (22.9):  20%|██        | 2/10 [00:04<00:17,  2.14s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The customer service is excellent, but the product quality needs improvement.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9228267347807578 / 3  (30.8):  30%|███       | 3/10 [00:06<00:15,  2.19s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app has many features, but it's overwhelming to new users.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9846449165989396 / 4  (24.6):  40%|████      | 4/10 [00:09<00:13,  2.27s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The recent changes are beneficial, but the app's speed has decreased.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9846449165989396 / 5  (19.7):  50%|█████     | 5/10 [00:11<00:11,  2.34s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The design is appealing, but the app crashes on older devices.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.2017877737417968 / 6  (20.0):  60%|██████    | 6/10 [00:13<00:09,  2.30s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The tutorials are helpful, but there should be more advanced guides.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.620835392789416 / 7  (23.2):  70%|███████   | 7/10 [00:16<00:06,  2.32s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app integrates well with other tools, but the synchronization is slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.020835392789416 / 8  (25.3):  80%|████████  | 8/10 [00:18<00:04,  2.30s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The pricing is competitive, but the subscription management is confusing.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.5124593244133475 / 9  (27.9):  90%|█████████ | 9/10 [00:20<00:02,  2.12s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is intuitive, but it lacks support for some file formats.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.82515773711176 / 10  (28.3): 100%|██████████| 10/10 [00:22<00:00,  2.24s/it]\n",
      "  0%|          | 0/10 [00:00<?, ?it/s]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The software is fast, but the interface is not intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.2111111111111111 / 1  (21.1):  10%|█         | 1/10 [00:02<00:19,  2.15s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I appreciate the quick updates, but the app is still buggy.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.45777777777777784 / 2  (22.9):  20%|██        | 2/10 [00:03<00:15,  1.92s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The customer service is excellent, but the product quality needs improvement.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9228267347807578 / 3  (30.8):  30%|███       | 3/10 [00:06<00:14,  2.12s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app has many features, but it's overwhelming to new users.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9846449165989396 / 4  (24.6):  40%|████      | 4/10 [00:08<00:13,  2.17s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The recent changes are beneficial, but the app's speed has decreased.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.9846449165989396 / 5  (19.7):  50%|█████     | 5/10 [00:10<00:11,  2.28s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The design is appealing, but the app crashes on older devices.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.203904175858199 / 6  (20.1):  60%|██████    | 6/10 [00:13<00:09,  2.29s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The tutorials are helpful, but there should be more advanced guides.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.622951794905818 / 7  (23.2):  70%|███████   | 7/10 [00:15<00:07,  2.36s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app integrates well with other tools, but the synchronization is slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.022951794905818 / 8  (25.3):  80%|████████  | 8/10 [00:18<00:04,  2.41s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The pricing is competitive, but the subscription management is confusing.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.5145757265297495 / 9  (27.9):  90%|█████████ | 9/10 [00:19<00:02,  2.18s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is intuitive, but it lacks support for some file formats.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.8272741392281624 / 10  (28.3): 100%|██████████| 10/10 [00:22<00:00,  2.22s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_96ff2 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_96ff2 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_96ff2_row0_col0, #T_96ff2_row0_col1, #T_96ff2_row0_col2, #T_96ff2_row0_col3, #T_96ff2_row0_col4, #T_96ff2_row0_col5, #T_96ff2_row0_col6, #T_96ff2_row0_col7, #T_96ff2_row0_col8 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_96ff2\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_96ff2_level0_col0\" class=\"col_heading level0 col0\" >feedback_text</th>\n",
       "      <th id=\"T_96ff2_level0_col1\" class=\"col_heading level0 col1\" >example_sentiment</th>\n",
       "      <th id=\"T_96ff2_level0_col2\" class=\"col_heading level0 col2\" >example_topics</th>\n",
       "      <th id=\"T_96ff2_level0_col3\" class=\"col_heading level0 col3\" >example_action_items</th>\n",
       "      <th id=\"T_96ff2_level0_col4\" class=\"col_heading level0 col4\" >rationale</th>\n",
       "      <th id=\"T_96ff2_level0_col5\" class=\"col_heading level0 col5\" >pred_sentiment</th>\n",
       "      <th id=\"T_96ff2_level0_col6\" class=\"col_heading level0 col6\" >pred_topics</th>\n",
       "      <th id=\"T_96ff2_level0_col7\" class=\"col_heading level0 col7\" >pred_action_items</th>\n",
       "      <th id=\"T_96ff2_level0_col8\" class=\"col_heading level0 col8\" >evaluate_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_96ff2_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_96ff2_row0_col0\" class=\"data row0 col0\" >The software is fast, but the interface is not intuitive.</td>\n",
       "      <td id=\"T_96ff2_row0_col1\" class=\"data row0 col1\" >mixed</td>\n",
       "      <td id=\"T_96ff2_row0_col2\" class=\"data row0 col2\" >performance interface</td>\n",
       "      <td id=\"T_96ff2_row0_col3\" class=\"data row0 col3\" >improve interface</td>\n",
       "      <td id=\"T_96ff2_row0_col4\" class=\"data row0 col4\" >Feedback Text: The software is fast, but the interface is not intuitive. Reasoning: Let's think step by step in order to improve the user experience....</td>\n",
       "      <td id=\"T_96ff2_row0_col5\" class=\"data row0 col5\" >Mixed</td>\n",
       "      <td id=\"T_96ff2_row0_col6\" class=\"data row0 col6\" >Software performance, User interface</td>\n",
       "      <td id=\"T_96ff2_row0_col7\" class=\"data row0 col7\" >- Identify areas where the interface is not intuitive.\n",
       "- Make the interface more user-friendly.</td>\n",
       "      <td id=\"T_96ff2_row0_col8\" class=\"data row0 col8\" >✔️ [0.2111111111111111]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x28652787460>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 9 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial performance: 28.27\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|█         | 1/10 [00:01<00:11,  1.29s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|██        | 2/10 [00:03<00:15,  1.90s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 30%|███       | 3/10 [00:04<00:10,  1.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|████      | 4/10 [00:05<00:07,  1.30s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|█████     | 5/10 [00:06<00:06,  1.30s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The software is fast, but the interface is not intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.4355555555555556 / 1  (43.6):  10%|█         | 1/10 [00:01<00:11,  1.27s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I appreciate the quick updates, but the app is still buggy.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.4355555555555557 / 2  (71.8):  20%|██        | 2/10 [00:02<00:09,  1.23s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The customer service is excellent, but the product quality needs improvement.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.4355555555555557 / 3  (81.2):  30%|███       | 3/10 [00:03<00:08,  1.22s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app has many features, but it's overwhelming to new users.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.768888888888889 / 4  (69.2):  40%|████      | 4/10 [00:04<00:06,  1.15s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The recent changes are beneficial, but the app's speed has decreased.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 3.377777777777778 / 5  (67.6):  50%|█████     | 5/10 [00:06<00:06,  1.22s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The design is appealing, but the app crashes on older devices.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 4.377777777777778 / 6  (73.0):  60%|██████    | 6/10 [00:07<00:05,  1.31s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The tutorials are helpful, but there should be more advanced guides.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 4.996825396825397 / 7  (71.4):  70%|███████   | 7/10 [00:09<00:04,  1.36s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app integrates well with other tools, but the synchronization is slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 5.899047619047619 / 8  (73.7):  80%|████████  | 8/10 [00:10<00:02,  1.28s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The pricing is competitive, but the subscription management is confusing.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 6.743492063492063 / 9  (74.9):  90%|█████████ | 9/10 [00:11<00:01,  1.19s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is intuitive, but it lacks support for some file formats.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 7.251428571428571 / 10  (72.5): 100%|██████████| 10/10 [00:12<00:00,  1.24s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_ee40a th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_ee40a td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_ee40a_row0_col0, #T_ee40a_row0_col1, #T_ee40a_row0_col2, #T_ee40a_row0_col3, #T_ee40a_row0_col4, #T_ee40a_row0_col5, #T_ee40a_row0_col6, #T_ee40a_row0_col7, #T_ee40a_row0_col8 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_ee40a\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_ee40a_level0_col0\" class=\"col_heading level0 col0\" >feedback_text</th>\n",
       "      <th id=\"T_ee40a_level0_col1\" class=\"col_heading level0 col1\" >example_sentiment</th>\n",
       "      <th id=\"T_ee40a_level0_col2\" class=\"col_heading level0 col2\" >example_topics</th>\n",
       "      <th id=\"T_ee40a_level0_col3\" class=\"col_heading level0 col3\" >example_action_items</th>\n",
       "      <th id=\"T_ee40a_level0_col4\" class=\"col_heading level0 col4\" >rationale</th>\n",
       "      <th id=\"T_ee40a_level0_col5\" class=\"col_heading level0 col5\" >pred_sentiment</th>\n",
       "      <th id=\"T_ee40a_level0_col6\" class=\"col_heading level0 col6\" >pred_topics</th>\n",
       "      <th id=\"T_ee40a_level0_col7\" class=\"col_heading level0 col7\" >pred_action_items</th>\n",
       "      <th id=\"T_ee40a_level0_col8\" class=\"col_heading level0 col8\" >evaluate_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_ee40a_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_ee40a_row0_col0\" class=\"data row0 col0\" >The software is fast, but the interface is not intuitive.</td>\n",
       "      <td id=\"T_ee40a_row0_col1\" class=\"data row0 col1\" >mixed</td>\n",
       "      <td id=\"T_ee40a_row0_col2\" class=\"data row0 col2\" >performance interface</td>\n",
       "      <td id=\"T_ee40a_row0_col3\" class=\"data row0 col3\" >improve interface</td>\n",
       "      <td id=\"T_ee40a_row0_col4\" class=\"data row0 col4\" >produce the action_items}. We need to improve the intuitiveness of the interface.</td>\n",
       "      <td id=\"T_ee40a_row0_col5\" class=\"data row0 col5\" >positive</td>\n",
       "      <td id=\"T_ee40a_row0_col6\" class=\"data row0 col6\" >speed interface</td>\n",
       "      <td id=\"T_ee40a_row0_col7\" class=\"data row0 col7\" >improve interface intuitiveness</td>\n",
       "      <td id=\"T_ee40a_row0_col8\" class=\"data row0 col8\" >✔️ [0.4355555555555556]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x28654419850>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 9 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimized performance: 72.51\n",
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Prediction(\n",
      "    rationale='produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.',\n",
      "    sentiment='mixed',\n",
      "    topics='user-friendliness stability',\n",
      "    action_items='fix crashes'\n",
      ")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_f777e th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_f777e td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_f777e_row0_col0, #T_f777e_row0_col1, #T_f777e_row0_col2, #T_f777e_row0_col3, #T_f777e_row0_col4, #T_f777e_row0_col5, #T_f777e_row0_col6, #T_f777e_row0_col7, #T_f777e_row0_col8 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_f777e\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_f777e_level0_col0\" class=\"col_heading level0 col0\" >feedback_text</th>\n",
       "      <th id=\"T_f777e_level0_col1\" class=\"col_heading level0 col1\" >example_sentiment</th>\n",
       "      <th id=\"T_f777e_level0_col2\" class=\"col_heading level0 col2\" >example_topics</th>\n",
       "      <th id=\"T_f777e_level0_col3\" class=\"col_heading level0 col3\" >example_action_items</th>\n",
       "      <th id=\"T_f777e_level0_col4\" class=\"col_heading level0 col4\" >rationale</th>\n",
       "      <th id=\"T_f777e_level0_col5\" class=\"col_heading level0 col5\" >pred_sentiment</th>\n",
       "      <th id=\"T_f777e_level0_col6\" class=\"col_heading level0 col6\" >pred_topics</th>\n",
       "      <th id=\"T_f777e_level0_col7\" class=\"col_heading level0 col7\" >pred_action_items</th>\n",
       "      <th id=\"T_f777e_level0_col8\" class=\"col_heading level0 col8\" >evaluate_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_f777e_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_f777e_row0_col0\" class=\"data row0 col0\" >The software is fast, but the interface is not intuitive.</td>\n",
       "      <td id=\"T_f777e_row0_col1\" class=\"data row0 col1\" >mixed</td>\n",
       "      <td id=\"T_f777e_row0_col2\" class=\"data row0 col2\" >performance interface</td>\n",
       "      <td id=\"T_f777e_row0_col3\" class=\"data row0 col3\" >improve interface</td>\n",
       "      <td id=\"T_f777e_row0_col4\" class=\"data row0 col4\" >Feedback Text: The software is fast, but the interface is not intuitive. Reasoning: Let's think step by step in order to improve the user experience....</td>\n",
       "      <td id=\"T_f777e_row0_col5\" class=\"data row0 col5\" >Mixed</td>\n",
       "      <td id=\"T_f777e_row0_col6\" class=\"data row0 col6\" >Software performance, User interface</td>\n",
       "      <td id=\"T_f777e_row0_col7\" class=\"data row0 col7\" >- Identify areas where the interface is not intuitive.\n",
       "- Make the interface more user-friendly.</td>\n",
       "      <td id=\"T_f777e_row0_col8\" class=\"data row0 col8\" >✔️ [0.2111111111111111]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x28652743d60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 9 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial performance: 28.25\n"
     ]
    }
   ],
   "source": [
    "initial_summarizer = SummarizeFeedback()\n",
    "initial_performance = evaluate_model(initial_summarizer, get_validation_examples())\n",
    "print(f\"Initial performance: {initial_performance:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The initial summarizer without optimization scored 28.30, which was a decent start but needed improvement.\n",
    "\n",
    "### 6. Optimizing the Model\n",
    "Optimizers are like personal trainers for your modules, helping them improve at their tasks by learning from examples. I decided to use the BootstrapFewShot optimizer to enhance the model's performance by generating better training examples, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|█         | 1/10 [00:01<00:10,  1.15s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|██        | 2/10 [00:03<00:12,  1.58s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 30%|███       | 3/10 [00:03<00:08,  1.28s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "Sentiment: mixed\n",
      "Topics: performance UI\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|████      | 4/10 [00:05<00:07,  1.23s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The installation process was straightforward, but the app is slow.\n",
      "Sentiment: mixed\n",
      "Topics: installation performance\n",
      "Action Items: improve performance\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app's search functionality is excellent, but it needs more filters.\n",
      "Sentiment: positive\n",
      "Topics: search functionality filters\n",
      "Action Items: add more filters\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "Sentiment: positive\n",
      "Topics: new feature documentation\n",
      "Action Items: improve documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "Sentiment: mixed\n",
      "Topics: customer support response time\n",
      "Action Items: reduce response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|█████     | 5/10 [00:06<00:06,  1.26s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The software is fast, but the interface is not intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 0.4355555555555556 / 1  (43.6):  10%|█         | 1/10 [00:01<00:13,  1.46s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I appreciate the quick updates, but the app is still buggy.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 1.4355555555555557 / 2  (71.8):  20%|██        | 2/10 [00:02<00:10,  1.28s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The customer service is excellent, but the product quality needs improvement.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.4355555555555557 / 3  (81.2):  30%|███       | 3/10 [00:03<00:08,  1.18s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app has many features, but it's overwhelming to new users.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 2.768888888888889 / 4  (69.2):  40%|████      | 4/10 [00:04<00:07,  1.19s/it] prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The recent changes are beneficial, but the app's speed has decreased.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 3.377777777777778 / 5  (67.6):  50%|█████     | 5/10 [00:06<00:06,  1.33s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The design is appealing, but the app crashes on older devices.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 4.377777777777778 / 6  (73.0):  60%|██████    | 6/10 [00:07<00:05,  1.34s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The tutorials are helpful, but there should be more advanced guides.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 4.996825396825397 / 7  (71.4):  70%|███████   | 7/10 [00:09<00:04,  1.40s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app integrates well with other tools, but the synchronization is slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 5.899047619047619 / 8  (73.7):  80%|████████  | 8/10 [00:10<00:02,  1.26s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The pricing is competitive, but the subscription management is confusing.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 6.743492063492063 / 9  (74.9):  90%|█████████ | 9/10 [00:11<00:01,  1.29s/it]prompt: /n  Given the fields `feedback_text`, produce the fields `sentiment`, `topics`, `action_items`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Feedback Text: ${feedback_text}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the action_items}. We ...\n",
      "\n",
      "Sentiment: ${sentiment}\n",
      "\n",
      "Topics: ${topics}\n",
      "\n",
      "Action Items: ${action_items}\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The new feature is great, but it needs more documentation.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to provide more documentation for the new feature.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: new feature documentation\n",
      "\n",
      "Action Items: provide more documentation\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is user-friendly, but it crashes frequently.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items. We need to identify the issues and their causes. The feedback mentions that the app is user-friendly, which is a positive aspect. However, it also mentions that the app crashes frequently, which is a negative aspect. To improve the app, we need to address the issue of frequent crashes.\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: user-friendliness stability\n",
      "\n",
      "Action Items: fix crashes\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: Customer support was very helpful, but the response time was slow.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve response time. We should ...\n",
      "\n",
      "Sentiment: mixed\n",
      "\n",
      "Topics: customer support response time\n",
      "\n",
      "Action Items: improve response time\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: I love the design of the app, but there are some bugs that need to be addressed.\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the action_items}. We need to identify the bugs and fix them.\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: design bugs\n",
      "\n",
      "Action Items: fix bugs\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The performance has improved significantly, but the UI could be more intuitive.\n",
      "\n",
      "Reasoning: Let's think step by step in order to improve the UI. We should ...\n",
      "\n",
      "Sentiment: positive\n",
      "\n",
      "Topics: performance UI\n",
      "\n",
      "Action Items: improve UI\n",
      "\n",
      "---\n",
      "\n",
      "Feedback Text: The app is intuitive, but it lacks support for some file formats.\n",
      "\n",
      "Reasoning: Let's think step by step in order to\n",
      "Average Metric: 7.251428571428571 / 10  (72.5): 100%|██████████| 10/10 [00:13<00:00,  1.33s/it]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_f46d2 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_f46d2 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_f46d2_row0_col0, #T_f46d2_row0_col1, #T_f46d2_row0_col2, #T_f46d2_row0_col3, #T_f46d2_row0_col4, #T_f46d2_row0_col5, #T_f46d2_row0_col6, #T_f46d2_row0_col7, #T_f46d2_row0_col8 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_f46d2\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_f46d2_level0_col0\" class=\"col_heading level0 col0\" >feedback_text</th>\n",
       "      <th id=\"T_f46d2_level0_col1\" class=\"col_heading level0 col1\" >example_sentiment</th>\n",
       "      <th id=\"T_f46d2_level0_col2\" class=\"col_heading level0 col2\" >example_topics</th>\n",
       "      <th id=\"T_f46d2_level0_col3\" class=\"col_heading level0 col3\" >example_action_items</th>\n",
       "      <th id=\"T_f46d2_level0_col4\" class=\"col_heading level0 col4\" >rationale</th>\n",
       "      <th id=\"T_f46d2_level0_col5\" class=\"col_heading level0 col5\" >pred_sentiment</th>\n",
       "      <th id=\"T_f46d2_level0_col6\" class=\"col_heading level0 col6\" >pred_topics</th>\n",
       "      <th id=\"T_f46d2_level0_col7\" class=\"col_heading level0 col7\" >pred_action_items</th>\n",
       "      <th id=\"T_f46d2_level0_col8\" class=\"col_heading level0 col8\" >evaluate_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_f46d2_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_f46d2_row0_col0\" class=\"data row0 col0\" >The software is fast, but the interface is not intuitive.</td>\n",
       "      <td id=\"T_f46d2_row0_col1\" class=\"data row0 col1\" >mixed</td>\n",
       "      <td id=\"T_f46d2_row0_col2\" class=\"data row0 col2\" >performance interface</td>\n",
       "      <td id=\"T_f46d2_row0_col3\" class=\"data row0 col3\" >improve interface</td>\n",
       "      <td id=\"T_f46d2_row0_col4\" class=\"data row0 col4\" >produce the action_items}. We need to improve the intuitiveness of the interface.</td>\n",
       "      <td id=\"T_f46d2_row0_col5\" class=\"data row0 col5\" >positive</td>\n",
       "      <td id=\"T_f46d2_row0_col6\" class=\"data row0 col6\" >speed interface</td>\n",
       "      <td id=\"T_f46d2_row0_col7\" class=\"data row0 col7\" >improve interface intuitiveness</td>\n",
       "      <td id=\"T_f46d2_row0_col8\" class=\"data row0 col8\" >✔️ [0.4355555555555556]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x28654460a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <div style='\n",
       "                    text-align: center;\n",
       "                    font-size: 16px;\n",
       "                    font-weight: bold;\n",
       "                    color: #555;\n",
       "                    margin: 10px 0;'>\n",
       "                    ... 9 more rows not displayed ...\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimized performance: 72.51\n"
     ]
    }
   ],
   "source": [
    "optimizer = BootstrapFewShot(metric=evaluate_summary, max_bootstrapped_demos=5, max_labeled_demos=5)\n",
    "compiled_summarizer = optimizer.compile(SummarizeFeedback(), trainset=get_examples())\n",
    "optimized_performance = evaluate_model(compiled_summarizer, get_validation_examples())\n",
    "print(f\"Optimized performance: {optimized_performance:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After using the BootstrapFewShot optimizer, our model's performance shot up to 72.51. That's a huge jump and shows that the optimizer really helped the model get better at picking up the right sentiment, key topics, and action items from the feedback.\n",
    "\n",
    "### Key Takeaways\n",
    "- Big Improvement: Going from a score of 28.30 to 72.51 is a significant boost. It means our model is now much more accurate and reliable.\n",
    "- Effective Optimization: The optimizer played a crucial role here. By generating and including high-quality examples, it fine-tuned our model to perform much better.\n",
    "- Better Summarization: Now, our model can provide more accurate and detailed summaries of customer feedback, which can be extremely useful for making informed decisions.\n",
    "- Cost-Effective Enhancement: This improvement was achieved without fine-tuning the model, simply by tweaking the prompt. This is a big win in terms of cost efficiency.\n",
    "- Modularization with DSPy: DSPy helped to modularize the problem, making it easy to optimize and manage different components of the task.\n",
    "\n",
    "In simpler terms, we started with a model that was just okay, but after some smart optimization, it became really good at its job. This kind of improvement is exactly what we want when dealing with complex tasks like feedback summarization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stay tuned for more blog posts where we'll explore DSPy in greater depth and uncover its features. For more details on DSPy, visit their official documentation: https://dspy-docs.vercel.app/\n",
    "You can access the full source code on my GitHub here: https://github.com/rawanalkurd/Generative-AI-DSPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SGPT",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}