examples/classify-recipes/3-evaluate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have a model in `./models/run1/merged` that was trained on GPT-4's outputs to classify recipes. I need to figure out whether it does a good job at classifying recipes. I'll install dependencies first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "%pip install vllm==0.1.3 pandas==2.0.3 joblib==1.3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember I got a \"test.jsonl\" file from OpenPipe back in [./1-generate-data.ipynb](./1-generate-data.ipynb)? That's data from our dataset that we didn't use in training, so we can use it to check our model's performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "test_data = pd.read_json(\"./data/test.jsonl\", lines=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the training process Axolotl transformed our data into an instruction/response format known as the \"Alpaca format\" based on [the project that introduced it](https://github.com/tatsu-lab/stanford_alpaca). I need to transform my test data into the same format for best results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample prompt:\n",
      "--------------\n",
      "### Instruction:\n",
      "[{\"role\":\"system\",\"content\":\"Your goal is to classify a recipe along several dimensions.Pay attention to the instructions.\"},{\"role\":\"user\",\"content\":\"Pan Gravy\\n\\nIngredients:\\n- 1/3 cup all purpose flour\\n- 1/3 cup turkey drippings\\n- 3 cup water or broth\\n- 1/8 to 1/4 teaspoon salt\\n- 1/8 tsp pepper\\n\\nDirections:\\n- In a skillet or roasting pan, add flour to drippings; blend well.\\n- Cook over medium heat 2 to 3 minutes until smooth and light brown, stirring constantly.\\n- Add water; cook until mixture boils and thickens, stirring constantly.\\n- Stir in salt and pepper.\\n- *Flour and drippings can be decreased to 1/4 cup each for thinner gravy.\\n- *\"}]\n",
      "\n",
      "### Response:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from axolotl.prompters import UnpromptedPrompter\n",
    "\n",
    "prompter = UnpromptedPrompter()\n",
    "\n",
    "\n",
    "def format_prompt(input: str) -> str:\n",
    "    return next(prompter.build_prompt(input))\n",
    "\n",
    "\n",
    "prompts = test_data[\"instruction\"].apply(format_prompt)\n",
    "\n",
    "print(f\"Sample prompt:\\n--------------\\n{prompts[0]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next up, I'll use [vLLM](https://vllm.readthedocs.io/en/latest/) to efficiently process all the prompts in our test data with our own model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 08-28 00:26:23 llm_engine.py:70] Initializing an LLM engine with config: model='./models/run1/merged', tokenizer='./models/run1/merged', tokenizer_mode=auto, trust_remote_code=False, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)\n",
      "INFO 08-28 00:27:26 llm_engine.py:196] # GPU blocks: 3419, # CPU blocks: 512\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processed prompts: 100%|██████████| 500/500 [00:37<00:00, 13.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample output:\n",
      "--------------\n",
      "{\"role\":\"assistant\",\"content\":null,\"function_call\":{\"name\":\"classify\",\"arguments\":\"{\\n\\\"has_non_fish_meat\\\": true,\\n\\\"requires_oven\\\": false,\\n\\\"requires_stove\\\": true,\\n\\\"cook_time_over_30_mins\\\": false,\\n\\\"main_dish\\\": false\\n}\"}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from vllm import LLM, SamplingParams\n",
    "\n",
    "llm = LLM(model=\"./models/run1/merged\", max_num_batched_tokens=4096)\n",
    "\n",
    "sampling_params = SamplingParams(\n",
    "    # 120 should be fine for the work we're doing here.\n",
    "    max_tokens=120,\n",
    "    # This is a deterministic task so temperature=0 is best.\n",
    "    temperature=0,\n",
    ")\n",
    "\n",
    "my_outputs = llm.generate(prompts, sampling_params=sampling_params)\n",
    "my_outputs = [o.outputs[0].text for o in my_outputs]\n",
    "\n",
    "test_data[\"my_outputs\"] = my_outputs\n",
    "\n",
    "print(f\"Sample output:\\n--------------\\n{my_outputs[0]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, we have our outputs! There are 5 categories we classify each recipe on, so let's check what percentage of the time our model's output matches GPT-4's. I'll write a quick eval function for that:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall accuracy: 0.95\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def parse_fn_call(str):\n",
    "    \"\"\"Parse the function call arguments from the response\"\"\"\n",
    "    response_dict = json.loads(str)\n",
    "    args_dict = json.loads(response_dict[\"function_call\"][\"arguments\"])\n",
    "\n",
    "    return args_dict\n",
    "\n",
    "\n",
    "test_data[\"output_parsed\"] = test_data[\"output\"].apply(parse_fn_call)\n",
    "test_data[\"my_outputs_parsed\"] = test_data[\"my_outputs\"].apply(parse_fn_call)\n",
    "\n",
    "\n",
    "def calculate_accuracy(row, labels_col):\n",
    "    \"\"\"Calculate the fraction of my model's outputs that match the reference outputs\"\"\"\n",
    "    true_outputs = row[\"output_parsed\"]\n",
    "    labels_outputs = row[labels_col]\n",
    "\n",
    "    # print(f\"true_outputs: {true_outputs}\")\n",
    "    # print(f\"my_outputs: {row[labels_col]}\")\n",
    "\n",
    "    num_matching_outputs = 0\n",
    "    for key in true_outputs.keys():\n",
    "        if key in labels_outputs and true_outputs[key] == labels_outputs[key]:\n",
    "            num_matching_outputs += 1\n",
    "\n",
    "    return num_matching_outputs / len(true_outputs)\n",
    "\n",
    "\n",
    "test_data[\"accuracy\"] = test_data.apply(\n",
    "    calculate_accuracy, axis=1, labels_col=\"my_outputs_parsed\"\n",
    ")\n",
    "\n",
    "print(f\"Overall accuracy: {test_data['accuracy'].mean():.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "95% seems good! However, we don't have much to compare it to. Let's see how GPT-3.5 would do on the same task as a baseline. We'll use the same prompt we used with GPT-4 to generate the labels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample recipe:\n",
      "--------------\n",
      "Pan Gravy\n",
      "\n",
      "Ingredients:\n",
      "- 1/3 cup all purpose flour\n",
      "- 1/3 cup turkey drippings\n",
      "- 3 cup water or broth\n",
      "- 1/8 to 1/4 teaspoon salt\n",
      "- 1/8 tsp pepper\n",
      "\n",
      "Directions:\n",
      "- In a skillet or roasting pan, add flour to drippings; blend well.\n",
      "- Cook over medium heat 2 to 3 minutes until smooth and light brown, stirring constantly.\n",
      "- Add water; cook until mixture boils and thickens, stirring constantly.\n",
      "- Stir in salt and pepper.\n",
      "- *Flour and drippings can be decreased to 1/4 cup each for thinner gravy.\n",
      "- *\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def extract_recipe(row):\n",
    "    \"\"\"Extract the recipe from the instruction\"\"\"\n",
    "    return json.loads(row[\"instruction\"])[1][\"content\"]\n",
    "\n",
    "\n",
    "recipes = test_data.apply(extract_recipe, axis=1)\n",
    "print(f\"Sample recipe:\\n--------------\\n{recipes[0]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifying first recipe:\n",
      "------------------\n",
      "{'has_non_fish_meat': False, 'requires_oven': False, 'requires_stove': True, 'cook_time_over_30_mins': False, 'main_dish': False}\n"
     ]
    }
   ],
   "source": [
    "import joblib\n",
    "import openai\n",
    "import os\n",
    "import dotenv\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "memory = joblib.Memory(\"./cache\", verbose=0)\n",
    "\n",
    "\n",
    "@memory.cache\n",
    "def classify_recipe_35(recipe: str):\n",
    "    completion = openai.ChatCompletion.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"Your goal is to classify a recipe along several dimensions.Pay attention to the instructions.\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": recipe,\n",
    "            },\n",
    "        ],\n",
    "        functions=[\n",
    "            {\n",
    "                \"name\": \"classify\",\n",
    "                \"parameters\": {\n",
    "                    \"type\": \"object\",\n",
    "                    \"properties\": {\n",
    "                        \"has_non_fish_meat\": {\n",
    "                            \"type\": \"boolean\",\n",
    "                            \"description\": \"True if the recipe contains any meat or meat products (eg. chicken broth) besides fish\",\n",
    "                        },\n",
    "                        \"requires_oven\": {\n",
    "                            \"type\": \"boolean\",\n",
    "                            \"description\": \"True if the recipe requires an oven\",\n",
    "                        },\n",
    "                        \"requires_stove\": {\n",
    "                            \"type\": \"boolean\",\n",
    "                            \"description\": \"True if the recipe requires a stove\",\n",
    "                        },\n",
    "                        \"cook_time_over_30_mins\": {\n",
    "                            \"type\": \"boolean\",\n",
    "                            \"description\": \"True if the recipe takes over 30 minutes to prepare and cook, including waiting time\",\n",
    "                        },\n",
    "                        \"main_dish\": {\n",
    "                            \"type\": \"boolean\",\n",
    "                            \"description\": \"True if the recipe can be served as a main dish\",\n",
    "                        },\n",
    "                    },\n",
    "                    \"required\": [\n",
    "                        \"has_non_fish_meat\",\n",
    "                        \"requires_oven\",\n",
    "                        \"requires_stove\",\n",
    "                        \"cook_time_over_30_mins\",\n",
    "                        \"main_dish\",\n",
    "                    ],\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "        function_call={\n",
    "            \"name\": \"classify\",\n",
    "        },\n",
    "    )\n",
    "    return json.loads(completion.choices[0].message.function_call.arguments)\n",
    "\n",
    "\n",
    "print(\"Classifying first recipe:\\n------------------\")\n",
    "print(classify_recipe_35(recipes[0]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "\u001b[A\n",
      "100%|██████████| 500/500 [00:31<00:00, 15.77it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "test_data[\"gpt_3.5\"] = [classify_recipe_35(r) for r in tqdm(recipes)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT-3.5 accuracy: 0.91\n"
     ]
    }
   ],
   "source": [
    "test_data[\"gpt_3.5_accuracy\"] = test_data.apply(\n",
    "    calculate_accuracy, axis=1, labels_col=\"gpt_3.5\"\n",
    ")\n",
    "\n",
    "print(f\"GPT-3.5 accuracy: {test_data['gpt_3.5_accuracy'].mean():.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for completeness, let's try a fine-tuned GPT-3.5 model. You can find the fine-tuning code in [finetune-gpt-3.5.ipynb](./finetune-gpt-3.5.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'has_non_fish_meat': True,\n",
       " 'requires_oven': False,\n",
       " 'requires_stove': True,\n",
       " 'cook_time_over_30_mins': False,\n",
       " 'main_dish': False}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@memory.cache\n",
    "def classify_recipe_35_ft(recipe: str):\n",
    "    completion = openai.ChatCompletion.create(\n",
    "        model=\"ft:gpt-3.5-turbo-0613:openpipe::7rZpPqYn\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"Your goal is to classify a recipe along several \"\n",
    "                \"dimensions.Pay attention to the instructions.\",\n",
    "            },\n",
    "            {\"role\": \"user\", \"content\": recipe},\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    return json.loads(completion.choices[0].message.content)\n",
    "\n",
    "\n",
    "classify_recipe_35_ft(recipes[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 500/500 [07:31<00:00,  1.11it/s]\n"
     ]
    }
   ],
   "source": [
    "test_data[\"gpt_3.5_ft\"] = [classify_recipe_35_ft(r) for r in tqdm(recipes)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT-3.5 FT accuracy: 0.94\n"
     ]
    }
   ],
   "source": [
    "test_data[\"gpt_3.5_ft_accuracy\"] = test_data.apply(\n",
    "    calculate_accuracy, axis=1, labels_col=\"gpt_3.5_ft\"\n",
    ")\n",
    "\n",
    "print(f\"GPT-3.5 FT accuracy: {test_data['gpt_3.5_ft_accuracy'].mean():.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad! However, there are still a few rows where the model outputs don't match. Let's take a closer look.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alligator Sauce Piquant\n",
      "\n",
      "Ingredients:\n",
      "- 2 lb. alligator, boneless and cubed *\n",
      "- 4 onions, diced\n",
      "- 1 c. parsley, chopped\n",
      "- 4 stalks celery, chopped\n",
      "- 1 bell pepper, diced\n",
      "- 1 c. catsup\n",
      "- 2 Tbsp. Heinz steak sauce\n",
      "- 2 Tbsp. soy sauce\n",
      "- 2 Tbsp. Louisiana hot sauce\n",
      "- 2 Tbsp. cornstarch\n",
      "- 1 tsp. salt\n",
      "- 2 tsp. red pepper (ground)\n",
      "- 1/4 c. cooking oil\n",
      "\n",
      "Directions:\n",
      "- *Alligator must be free of all fat; also dark meat is the best (leg and body meat), boneless.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>My model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cook_time_over_30_mins</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>main_dish</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        GPT-4  My model\n",
       "cook_time_over_30_mins   True     False\n",
       "main_dish                True     False"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Veggie Casserole\n",
      "\n",
      "Ingredients:\n",
      "- 1 (8 oz.) bag mixed veggies (corn, peas, carrots, green beans), steamed\n",
      "- 1 c. celery\n",
      "- 1 c. onions\n",
      "- 1 c. Cheddar cheese\n",
      "- 1 c. mayonnaise\n",
      "\n",
      "Directions:\n",
      "- Mix above ingredients.\n",
      "- Bake at 350° for 30 minutes, until bubbly.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>My model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>main_dish</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           GPT-4  My model\n",
       "main_dish  False      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rhonda'S Butter Chess Pie\n",
      "\n",
      "Ingredients:\n",
      "- 5 eggs\n",
      "- 1 stick melted butter\n",
      "- 2 c. sugar\n",
      "- 1 tsp. vanilla\n",
      "- 1 Tbsp. cornstarch\n",
      "- 1/2 c. buttermilk\n",
      "- unbaked 9-inch deep dish pie shell\n",
      "\n",
      "Directions:\n",
      "- Mix eggs with sugar and cornstarch until smooth.\n",
      "- Add melted butter, vanilla and buttermilk.\n",
      "- Bake at 350° for 30 minutes or until done.\n",
      "- Let cool and chill.\n",
      "- Similar to Furr's Butter Chess Pie.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>My model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cook_time_over_30_mins</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        GPT-4  My model\n",
       "cook_time_over_30_mins  False      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Broccoli Gorgonzola Cream Soup\n",
      "\n",
      "Ingredients:\n",
      "- 2 heads Broccoli\n",
      "- 700 milliliters Water\n",
      "- 1 Onion, Peeled And Cut Into Chunks\n",
      "- 1 pinch Salt\n",
      "- 1 teaspoon Oregano\n",
      "- 1 Potato, Peeled And Cut Into Chunks\n",
      "- 200 grams Crumbled Gorgonzola\n",
      "- 1 Tablespoon Finely Grated Parmesan\n",
      "\n",
      "Directions:\n",
      "- Cut off the hard trunks of the broccoli and cut it into small pieces. Prepare a pot with water, add broccoli, onion, salt and oregano and boil for about 30 minutes.\n",
      "- Add the peeled potato and boil for another 20 minutes. When vegetables are cooked, strain and save the stock.\n",
      "- Using a hand blender, puree vegetables, adding as much stock as desired. Bring soup back to heat over low heat, and sir in gorgonzola. Remove from heat and add Parmesan.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>My model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>main_dish</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           GPT-4  My model\n",
       "main_dish  False      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wild Rice With Cucumber And Feta\n",
      "\n",
      "Ingredients:\n",
      "- 1 (8.5-ounce) package precooked wild rice (such as Archer Farms)\n",
      "- 1 cup diced English cucumber\n",
      "- 1 1/2 tablespoons olive oil\n",
      "- 1 tablespoon fresh lemon juice\n",
      "- 2 ounces crumbled feta cheese\n",
      "- 1/2 teaspoon pepper\n",
      "- 1/4 teaspoon salt\n",
      "\n",
      "Directions:\n",
      "- Prepare rice according to the package directions.\n",
      "- Combine cooked rice, cucumber, olive oil, lemon juice, and crumbled feta cheese in a medium bowl; toss to coat. Stir in pepper and salt.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GPT-4</th>\n",
       "      <th>My model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>main_dish</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           GPT-4  My model\n",
       "main_dish   True     False"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "for row in test_data[test_data.accuracy < 1].sample(5).itertuples():\n",
    "    print(json.loads(row.instruction)[1][\"content\"])\n",
    "\n",
    "    gpt4_output = parse_fn_call(row.output)\n",
    "    my_output = parse_fn_call(row.my_outputs)\n",
    "\n",
    "    table = pd.DataFrame(\n",
    "        {\n",
    "            \"GPT-4\": gpt4_output,\n",
    "            \"My model\": my_output,\n",
    "        }\n",
    "    )\n",
    "\n",
    "    table = table[table[\"GPT-4\"] != table[\"My model\"]]\n",
    "    display(table)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the outputs, it's clear that our model still makes some mistakes. But at the same time, there are plenty of examples like \"Rhonda's Butter Chess Pie\" where our model gets it right, even though GPT-4 got it wrong! And there are also cases like the \"Veggie Casserole\", where the \"right\" answer is truly ambiguous and really both answers are defensible.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A realistic point of comparison here might be GPT-3.5. Let's try to classify the same set of recipes using GPT-3.5 and see how it does. We'll use the same prompt that we used with GPT-4 to generate the initial training data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interested in cost/latency benchmarking? You can check out [./4-benchmark.ipynb](./4-benchmark.ipynb) for an overview of my findings!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}