prompting_capabilities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5d7f6058-d0d1-44af-b43a-cb4b06df03d8",
   "metadata": {},
   "source": [
    "# Prompting Capabilities \n",
    "\n",
    "When you first start using Mistral models, your first interaction will revolve around prompts. The art of crafting effective prompts is essential for generating desirable responses from Mistral models or other LLMs. This guide will walk you through example prompts showing four different prompting capabilities. \n",
    "\n",
    "- Classification \n",
    "- Summarization \n",
    "- Personalization\n",
    "- Evaluation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daa37d97-10e4-425d-b852-15bd043662b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install mistralai==0.0.12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ae93e75a-fd20-4ca6-9eff-bef7d44ab3a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mistralai.client import MistralClient\n",
    "from mistralai.models.chat_completion import ChatMessage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "99ff9378-c059-4063-a8a0-9ec1d1186954",
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = \"TYPE YOUR API KEY\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e030d9c2-1ecb-4bf0-864d-9f96b41bd016",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_mistral(user_message, model=\"mistral-medium\"):\n",
    "    client = MistralClient(api_key=api_key)\n",
    "    messages = [\n",
    "        ChatMessage(role=\"user\", content=user_message)\n",
    "    ]\n",
    "    chat_response = client.chat(\n",
    "        model=model,\n",
    "        messages=messages\n",
    "    )\n",
    "    return (chat_response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b98a29ef-4764-40b7-aed8-0e5d0502f985",
   "metadata": {},
   "source": [
    "## Classification\n",
    "Mistral models can easily categorize text into distinct classes. In this example prompt, we can define a list of predefined categories and ask Mistral models to classify user inquiry. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d71e9d1c-ca45-4d19-882c-07e077ea19ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def user_message(inquiry):\n",
    "    user_message = (\n",
    "        f\"\"\"\n",
    "        You are a bank customer service bot. Your task is to assess customer intent \n",
    "        and categorize customer inquiry after <<<>>> into one of the following predefined categories:\n",
    "        \n",
    "        card arrival\n",
    "        change pin\n",
    "        exchange rate\n",
    "        country support \n",
    "        cancel transfer\n",
    "        charge dispute\n",
    "        \n",
    "        If the text doesn't fit into any of the above categories, classify it as:\n",
    "        customer service\n",
    "        \n",
    "        You will only respond with the predefined category. Do not include the word \"Category\". Do not provide explanations or notes. \n",
    "        \n",
    "        ####\n",
    "        Here are some examples:\n",
    "        \n",
    "        Inquiry: How do I know if I will get my card, or if it is lost? I am concerned about the delivery process and would like to ensure that I will receive my card as expected. Could you please provide information about the tracking process for my card, or confirm if there are any indicators to identify if the card has been lost during delivery?\n",
    "        Category: card arrival\n",
    "        Inquiry: I am planning an international trip to Paris and would like to inquire about the current exchange rates for Euros as well as any associated fees for foreign transactions.\n",
    "        Category: exchange rate \n",
    "        Inquiry: What countries are getting support? I will be traveling and living abroad for an extended period of time, specifically in France and Germany, and would appreciate any information regarding compatibility and functionality in these regions.\n",
    "        Category: country support\n",
    "        Inquiry: Can I get help starting my computer? I am having difficulty starting my computer, and would appreciate your expertise in helping me troubleshoot the issue. \n",
    "        Category: customer service\n",
    "        ###\n",
    "    \n",
    "        <<<\n",
    "        Inquiry: {inquiry}\n",
    "        >>>\n",
    "        \"\"\"\n",
    "    )\n",
    "    return user_message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8b208e1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8863460a-7670-4b6f-b511-cf1df7693cfa",
   "metadata": {},
   "source": [
    "### Strategies we used: \n",
    "\n",
    "- **Few shot learning**: Few-shot learning or in-context learning is when we give a few examples in the prompts, and the LLM can generate corresponding output based on the example demonstrations. Few-shot learning can often improve model performance especially when the task is difficult or when we want the model to respond in a specific manner. \n",
    "- **Delimiter**: Delimiters like ### <<< >>> specify the boundary between different sections of the text. In our example, we used ### to indicate examples and <<<>>> to indicate customer inquiry. \n",
    "- **Role playing**: Providing LLM a role (e.g., \"You are a bank customer service bot.\") adds personal context to the model and often leads to better performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9d8a83cc-31e6-4d4b-b252-93d9133ecbf5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "country support\n"
     ]
    }
   ],
   "source": [
    "print(run_mistral(user_message(\n",
    "    \"I am inquiring about the availability of your cards in the EU, as I am a resident of France and am interested in using your cards. \"\n",
    ")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d6eca06b-7b1d-4663-9a68-2a23116f8c6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "customer service\n"
     ]
    }
   ],
   "source": [
    "print(run_mistral(user_message(\"What's the weather today?\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f351f00-683a-4244-974f-5c719812141f",
   "metadata": {},
   "source": [
    "## Summarization\n",
    "\n",
    "Summarization is a common task for LLMs due to their natural language understanding and generation capabilities. Here is an example prompt we can use to generate interesting questions about an essay and summarize the essay.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0bbab492-1d18-4832-86a9-2fba645e0e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "response = requests.get('https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt')\n",
    "essay = response.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "eaede63d-7392-4f1c-8a87-507ee31fe246",
   "metadata": {},
   "outputs": [],
   "source": [
    "message = f\"\"\"\n",
    "You are a commentator. Your task is to write a report on an essay. \n",
    "When presented with the essay, come up with interesting questions to ask, and answer each question. \n",
    "Afterward, combine all the information and write a report in the markdown format. \n",
    "\n",
    "# Essay: \n",
    "{essay}\n",
    "\n",
    "# Instructions: \n",
    "## Summarize:\n",
    "In clear and concise language, summarize the key points and themes presented in the essay.\n",
    "\n",
    "## Interesting Questions: \n",
    "Generate three distinct and thought-provoking questions that can be asked about the content of the essay. For each question:\n",
    "- After \"Q: \", describe the problem \n",
    "- After \"A: \", provide a detailed explanation of the problem addressed in the question.\n",
    "- Enclose the ultimate answer in <>.\n",
    "\n",
    "## Write a report \n",
    "Using the essay summary and the answers to the interesting questions, create a comprehensive report in Markdown format. \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5505b0a5-411b-4804-aaef-ccecfa3d07be",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Summary:**\n",
      "\n",
      "The essay is a reflection on the author's experiences and the lessons learned from working on different projects throughout their life. The author discusses their time spent writing short stories, programming, studying philosophy, and working in AI. They also touch on their brief foray into the world of art school and their eventual return to programming and the creation of Y Combinator. Throughout the essay, the author emphasizes the importance of focusing on projects that are interesting and meaningful, rather than chasing prestige.\n",
      "\n",
      "**Interesting Questions:**\n",
      "\n",
      "Q: What role did Lisp play in the author's life and career?\n",
      "A: Lisp was a significant part of the author's life and career. They first learned Lisp in college, and it expanded their concept of a program and what was possible with code. The author also wrote a book about Lisp hacking, and they used Lisp to build the code editor for users to define their own page styles in Viaweb. Lisp was also the language used to write the Y Combinator application, and the author's experience with Lisp was a major factor in their decision to start Y Combinator.\n",
      "\n",
      "Q: What lessons did the author learn from their experiences in art school?\n",
      "A: The author attended two different art schools, RISD and the Accademia di Belli Arti in Florence. They learned that art school did not bear the same relationship to art that medical school bore to medicine. The author also discovered that painting was a post-rigorous field at RISD, and that many students were more focused on creating a signature style than on learning to draw or paint. Despite this, the author learned a lot in the color class at RISD and enjoyed the presence of Idelle and Julian Weber in New York.\n",
      "\n",
      "Q: How did the author's experiences in AI influence their decision to start Y Combinator?\n",
      "A: The author spent several years working in AI, but eventually realized that the type of AI being practiced at the time was a hoax. They realized that there was an unbridgeable gap between what AI programs could do and actually understanding natural language. This realization led the author to look for other projects to work on, and they eventually decided to focus on Lisp. The author's experience with AI also taught them that the low end eats the high end, which they used to their advantage when starting Y Combinator.\n",
      "\n",
      "**Report:**\n",
      "\n",
      "The essay, \"What I Worked On,\" is a reflection on the author's experiences and the lessons learned from working on different projects throughout their life. The author discusses their time spent writing short stories, programming, studying philosophy, and working in AI. They also touch on their brief foray into the world of art school and their eventual return to programming and the creation of Y Combinator.\n",
      "\n",
      "One of the most significant experiences in the author's life was their time spent programming in Lisp. They first learned Lisp in college, and it expanded their concept of a program and what was possible with code. The author also wrote a book about Lisp hacking, and they used Lisp to build the code editor for users to define their own page styles in Viaweb. Lisp was also the language used to write the Y Combinator application, and the author's experience with Lisp was a major factor in their decision to start Y Combinator.\n",
      "\n",
      "The author's experiences in art school were also influential, although not in the way they had expected. They attended two different art schools, RISD and the Accademia di Belli Arti in Florence. They learned that art school did not bear the same relationship to art that medical school bore to medicine. The author also discovered that painting was a post-rigorous field at RISD, and that many students were more focused on creating a signature style than on learning to draw or paint. Despite this, the author learned a lot in the color class at RISD and enjoyed the presence of Idelle and Julian Weber in New York.\n",
      "\n",
      "The author's experiences in AI also played a role in their decision to start Y Combinator. They spent several years working in AI, but eventually realized that the type of AI being practiced at the time was a hoax. They realized that there was an unbridgeable gap between what AI programs could do and actually understanding natural language. This realization led the author to look for other projects to work on, and they eventually decided to focus on Lisp. The author's experience with AI also taught them that the low end eats the high end, which they used to their advantage when starting Y Combinator.\n",
      "\n",
      "Throughout the essay, the author emphasizes the importance of focusing on projects that are interesting and meaningful, rather than chasing prestige. They write, \"I've worked on several different things, but to the extent there was a turning point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\" This focus on meaningful work is a recurring theme in the essay, and it reflects the author's belief that the most important thing is to do work that matters to you, rather than work that will impress others.\n",
      "\n",
      "Overall, the essay is a thoughtful reflection on the author's experiences and the lessons they learned from working on different projects throughout their life. It is a reminder that the most important thing is to do work that is meaningful and interesting, rather than chasing prestige or trying to fit into a certain mold. By focusing on projects that matter to them, the author was able to make a significant impact in their field and create a company that has helped countless startups get off the ground.\n",
      "\n",
      "References:\n",
      "\n",
      "[1] Graham, P. (n.d.). What I Worked On. Retrieved February 16, 2023, from <https://www.paulgraham.com/workedon.html>\n",
      "\n",
      "[2] ANSI Common Lisp. (n.d.). Retrieved February 16, 2023, from <https://www.paulgraham.com/acl.html>\n",
      "\n",
      "[3] Y Combinator. (n.d.). Retrieved February 16, 2023, from <https://www.ycombinator.com/>\n",
      "\n",
      "[4] Lisp Hackers. (n.d.). Retrieved February 16, 2023, from <https://www.lisp.org/>\n",
      "\n",
      "[5] Artificial Intelligence. (n.d.). Retrieved February 16, 2023, from <https://en.wikipedia.org/wiki/Artificial_intelligence>\n"
     ]
    }
   ],
   "source": [
    "print(run_mistral(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3c1535e-1156-46bc-8e28-5f71846dbe65",
   "metadata": {},
   "source": [
    "## Strategies we used: \n",
    "\n",
    "- **Step-by-step instructions**: This strategy is inspired by the chain-of-thought prompting that enables LLMs to use a series of intermediate reasoning steps to tackle complex tasks. It's often easier to solve complex problems when we decompose them into simpler and small steps and it's easier for us to debug and inspect the model behavior.  In our example, we break down the task into three steps: summarize, generate interesting questions, and write a report. This helps the language to think in each step and generate a more comprehensive final report. \n",
    "- **Example generation**: We can ask LLMs to automatically guide the reasoning and understanding process by generating examples with the explanations and steps. In this example, we ask the LLM to generate three questions and provide detailed explanations for each question. \n",
    "- **Output formatting**: We can ask LLMs to output in a certain format by directly asking \"write a report in the Markdown format\".\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a9cb95-bb08-4929-b16c-eb19877f3c01",
   "metadata": {},
   "source": [
    "## Personlization\n",
    "\n",
    "LLMs excel at personalization tasks as they can deliver content that aligns closely with individual users. In this example, we create personalized email responses to address customer questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9cf048b4-3e33-4753-af97-25b73c51ee6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "email = \"\"\"\n",
    "Dear mortgage lender, \n",
    "\n",
    "What's your 30-year fixed-rate APR, how is it compared to the 15-year fixed rate?\n",
    "\n",
    "Regards,\n",
    "Anna\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "36de7c1e-60c2-4f35-a51a-115b12d65bb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "message = f\"\"\"\n",
    "\n",
    "You are a mortgage lender customer service bot, and your task is to create personalized email responses to address customer questions.\n",
    "Answer the customer's inquiry using the provided facts below. Ensure that your response is clear, concise, and \n",
    "directly addresses the customer's question. Address the customer in a friendly and professional manner. Sign the email with \n",
    "\"Lender Customer Support.\"   \n",
    "\n",
    "\n",
    "      \n",
    "# Facts\n",
    "30-year fixed-rate: interest rate 6.403%, APR 6.484%\n",
    "20-year fixed-rate: interest rate 6.329%, APR 6.429%\n",
    "15-year fixed-rate: interest rate 5.705%, APR 5.848%\n",
    "10-year fixed-rate: interest rate 5.500%, APR 5.720%\n",
    "7-year ARM: interest rate 7.011%, APR 7.660%\n",
    "5-year ARM: interest rate 6.880%, APR 7.754%\n",
    "3-year ARM: interest rate 6.125%, APR 7.204%\n",
    "30-year fixed-rate FHA: interest rate 5.527%, APR 6.316%\n",
    "30-year fixed-rate VA: interest rate 5.684%, APR 6.062%\n",
    "\n",
    "# Email\n",
    "{email}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "aaf50774-0f91-4e0e-86c9-1525f6045ebb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dear Anna,\n",
      "\n",
      "Thank you for your inquiry regarding our mortgage rates.\n",
      "\n",
      "Our current 30-year fixed-rate interest rate is 6.403% with an APR of 6.484%. In comparison, our 15-year fixed-rate interest rate is 5.705% with an APR of 5.848%.\n",
      "\n",
      "Please note that the APR, or Annual Percentage Rate, is a broader measure of the cost of a mortgage loan that includes not only the interest rate but also other costs associated with the loan, such as points, fees, and mortgage insurance.\n",
      "\n",
      "I hope this information is helpful. If you have any further questions, please don't hesitate to contact us.\n",
      "\n",
      "Best regards,\n",
      "Lender Customer Support\n"
     ]
    }
   ],
   "source": [
    "print(run_mistral(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af53ac9-8ef1-4840-8b23-779e1d59204d",
   "metadata": {},
   "source": [
    "### Strategies we used: \n",
    "- Providing facts: Incorporating facts into prompts can be useful for developing customer support bots. It’s important to use clear and concise language when presenting these facts. This can help the LLM to provide accurate and quick responses to customer queries.\n",
    "\n",
    "## Evaluation\n",
    "\n",
    "There are many ways to evaluate LLM outputs. Here are three approaches for your reference: include a confidence score, introduce an evaluation step, or employ another LLM for evaluation.\n",
    "\n",
    "\n",
    "\n",
    "## Include a confidence score\n",
    "We can include a confidence score along with the generated output in the prompt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8bf511a7-1828-465c-92ed-a701e21b23da",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_mistral(user_message, model=\"mistral-medium\"):\n",
    "    client = MistralClient(api_key=api_key)\n",
    "    messages = [\n",
    "        ChatMessage(role=\"user\", content=user_message)\n",
    "    ]\n",
    "    chat_response = client.chat(\n",
    "        model=model,\n",
    "        messages=messages,\n",
    "        temperature=1\n",
    "    )\n",
    "    return (chat_response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8087cae2-9b3b-407c-b808-31fd765fd6f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "message = f\"\"\"\n",
    "You are a summarization system that can provide summaries with associated confidence scores.\n",
    "In clear and concise language, provide three short summaries of the following essay, along with their confidence scores. \n",
    "You will only respond with a JSON object with the key Summary and Confidence. Do not provide explanations. \n",
    "\n",
    "# Essay: \n",
    "{essay}\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1c7bfcee-ff8b-4c14-b9e8-7e00d3d389ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Summary1\": \"The author discusses their experiences working on writing, programming, and art projects before and during college. They share anecdotes about their early attempts at programming on an IBM 1401, their love for the language Lisp, and their disillusionment with the field of philosophy. The author also reflects on their decision to study artificial intelligence in college and their eventual realization that the field was not what they had expected it to be.\",\n",
      "  \"Confidence1\": 0.85,\n",
      "  \"Summary2\": \"The author talks about their journey from writing short stories to programming and eventually studying AI in college. They discuss their initial disappointment with AI and how they switched to studying Lisp instead. They also touch upon their experiences in art school and their eventual decision to become an artist.\",\n",
      "  \"Confidence2\": 0.8,\n",
      "  \"Summary3\": \"The author reflects on their experiences working on various projects, including writing, programming, and art. They discuss their early attempts at programming on an IBM 1401, their passion for the language Lisp, and their disappointment with the field of philosophy in college. The author also shares their experiences studying AI and attending art school, and expresses their desire to build things that will last.\",\n",
      "  \"Confidence3\": 0.75\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(run_mistral(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8649ad18-10c0-4fb0-92cc-052e2ab75342",
   "metadata": {},
   "source": [
    "### Strategies we used: \n",
    "- JSON output: For facilitating downstream tasks, JSON format output is frequently preferred. We can specify in the prompt that “You will only respond with a JSON object with the key Summary and Confidence.” Specifying these keys within the JSON object is beneficial for clarity and consistency.\n",
    "- Higher Temperature: In this example, we increase the temperature score to encourage the model to be more creative and output three generated summaries that are different from each other.  \n",
    "\n",
    "\n",
    "## Introduce an evaluation step \n",
    "We can also add a second step in the prompt for evaluation. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6300a485-43be-4bc4-9088-9e6c2a365b6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary 1:\n",
      "The author shared their experiences in programming, writing, and art school. They started programming on an IBM 1401 in 9th grade and began writing short stories. They later switched to AI in college, but became disillusioned with it and turned to Lisp programming. After studying at an art school in Florence, they attended RISD where they learned about color and painting. However, they were disappointed with the painting department and eventually left to start their own software company, Viaweb, which was later acquired by Yahoo.\n",
      "\n",
      "Summary 2:\n",
      "The author describes their experiences in programming, AI, and art school. They started programming on an IBM 1401 in 9th grade and wrote short stories. After discovering Lisp programming in college, they became interested in AI but eventually became disillusioned with it. They then attended an art school in Florence and later RISD, where they were disappointed with the painting department. The author eventually left to start their software company, Viaweb, which was later acquired by Yahoo.\n",
      "\n",
      "Summary 3:\n",
      "The author reflects on their journey from programming on an IBM 1401 in 9th grade to starting their own software company, Viaweb, which was later acquired by Yahoo. They discuss their experiences in writing short stories, studying AI in college, and becoming disillusioned with it. They then attended an art school in Florence and later RISD, where they learned about color and painting. Eventually, they left RISD to start Viaweb.\n",
      "\n",
      "Evaluation:\n",
      "In my opinion, Summary 3 is the best summary as it captures the author's journey from programming to starting their own software company, which is the central theme of the essay. Summary 2 is also a good summary, but it focuses more on the author's experiences in AI and art school. Summary 1, on the other hand, is too detailed and includes unnecessary information about the author's experience with color and painting at RISD.\n",
      "\n",
      "Overall, Summary 3 effectively conveys the main points of the essay while remaining concise and clear, making it the best summary in my opinion.\n"
     ]
    }
   ],
   "source": [
    "message = f\"\"\"\n",
    "You are given an essay text and need to provide summaries and evaluate them.\n",
    "\n",
    "# Essay: \n",
    "{essay}\n",
    "\n",
    "Step 1: In this step, provide three short summaries of the given essay. Each summary should be clear, concise, and capture the key points of the speech. Aim for around 2-3 sentences for each summary.\n",
    "Step 2: Evaluate the three summaries from Step 1 and rate which one you believe is the best. Explain your choice by pointing out specific reasons such as clarity, completeness, and relevance to the speech content.\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "print(run_mistral(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba076c25-9536-4ba1-a730-0a3ce6fe24f0",
   "metadata": {},
   "source": [
    "## Employ another LLM for evaluation\n",
    "In production systems, it is common to employ another LLM for evaluation so that the evaluation step can be separate from the generation step.  \n",
    "\n",
    "- Step 1: use the first LLM to generate three summaries \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d5265085-77fd-433a-8d9d-d89ffac76543",
   "metadata": {},
   "outputs": [],
   "source": [
    "message = f\"\"\"\n",
    "Provide three short summaries of the given essay. Each summary should be clear, concise, and capture the key points of the essay.\n",
    "Aim for around 2-3 sentences for each summary.\n",
    "\n",
    "# essay: \n",
    "{essay}\n",
    "\n",
    "\"\"\"\n",
    "summaries = run_mistral(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4372a9fe-d596-4f36-86c6-3e3c01c74c38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary 1:\n",
      "The essay describes the author's experiences before college, focusing on their work with writing and programming. They wrote short stories as a beginner writer and experimented with programming on an IBM 1401 used for data processing. The author struggled to find a purpose for the 1401 and eventually shifted their focus to microcomputers, which allowed for direct user input and a more interactive programming experience.\n",
      "\n",
      "Summary 2:\n",
      "The author reflects on their early programming experiences, starting with writing simple programs on punch cards for an IBM 1401 and progressing to microcomputers. They learned to program in Fortran and later Lisp, eventually settling on the latter as their preferred language. The author's interest in artificial intelligence led them to write a book about Lisp, called \"On Lisp\". They also experimented with art classes at Harvard and discovered a passion for painting.\n",
      "\n",
      "Summary 3:\n",
      "The essay covers the author's background in writing and programming before college. They wrote short stories without much success and discovered programming on an IBM 1401 at their school. When microcomputers became available, they found them more engaging due to their interactive nature. The author discovered their passion for Lisp and wrote a book about it while attending Harvard for a Ph.D. in computer science. Despite enjoying Lisp hacking, the author also took art classes at Harvard, revealing an unexpected love for painting.\n"
     ]
    }
   ],
   "source": [
    "print(summaries)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58fd5f25-ab9a-4547-83ab-48638517b7ef",
   "metadata": {},
   "source": [
    "- Step 2: use another LLM to rate the generated summaries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b755adc8-c277-4421-8925-3d70e5ee39e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best summary is Summary 3.\n",
      "\n",
      "It provides a clear and concise overview of the essay, highlighting the author's experiences with writing and programming before college. It mentions the author's failed attempts at short story writing, their discovery of programming on an IBM 1401, and their shift to microcomputers. It also notes the author's interest in Lisp and their decision to write a book about it while attending Harvard for a Ph.D. in computer science. Finally, it acknowledges the author's unexpected passion for painting, discovered through art classes at Harvard.\n",
      "\n",
      "Summary 1 is also fairly good, but it omits some important details, such as the author's interest in Lisp and their book about it. It also does not mention the author's passion for painting.\n",
      "\n",
      "Summary 2 is the least accurate, as it omits many of the key details from the essay. It mentions the author's experiences with Fortran and Lisp, but does not provide any context or explanation for how they discovered these languages. It also fails to mention the author's short story writing, their discovery of programming on an IBM 1401, their shift to microcomputers, or their passion for painting.\n"
     ]
    }
   ],
   "source": [
    "message = f\"\"\"\n",
    "You are given an essay and three summaries of the essay. Evaluate the three summaries and rate which one you believe is the best. \n",
    "Explain your choice by pointing out specific reasons such as clarity, completeness, and relevance to the essay content.\n",
    "\n",
    "# Essay: \n",
    "{essay}\n",
    "\n",
    "# Summaries \n",
    "{summaries}\n",
    "\n",
    "\"\"\"\n",
    "print(run_mistral(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e96d61a-9dbc-4f8c-9d80-d2260c5f3e61",
   "metadata": {},
   "source": [
    "### Strategies we used: \n",
    "- **LLM chaining**: In this example, we chain two LLMs in a sequence, where the output from the first LLM serves as the input for the second LLM. The method of chaining LLMs can be adapted to suit your specific use cases. For instance, you might choose to employ three LLMs in a chain, where the output of two LLMs is funneled into the third LLM. While LLM chaining offers flexibility, it's important to consider that it may result in additional API calls and potentially increased costs.\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:light"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}