examples/lab/quicksart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# phospho quickstart\n",
    "\n",
    "In this quickstart, we will use the `lab` from the `phospho` package to figure out how many messages in a dataset are questions.\n",
    "\n",
    "1. First, we will detect events on a subset of the dataset using a pipeline powered by OpenAI GPT 3.5\n",
    "\n",
    "2. Then, we will scale analytics with the `lab` optimizer. We will compare the event detection pipeline using MistralAI and a local Ollama model, and pick the best one in term of performance, speed and price.\n",
    "\n",
    "3. Finally, we will use the `lab` to run the best model on the full dataset and visualize the results.\n",
    "\n",
    "This way, we will be able to run semantic analytics at scale on a dataset using LLMs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Skipping /Users/nicolasoulianov/anaconda3/envs/phospho-env/lib/python3.11/site-packages/typing_extensions-4.8.0.dist-info due to invalid metadata entry 'name'\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Skipping /Users/nicolasoulianov/anaconda3/envs/phospho-env/lib/python3.11/site-packages/typing_extensions-4.8.0.dist-info due to invalid metadata entry 'name'\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Skipping /Users/nicolasoulianov/anaconda3/envs/phospho-env/lib/python3.11/site-packages/typing_extensions-4.8.0.dist-info due to invalid metadata entry 'name'\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Skipping /Users/nicolasoulianov/anaconda3/envs/phospho-env/lib/python3.11/site-packages/typing_extensions-4.8.0.dist-info due to invalid metadata entry 'name'\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q python-dotenv \"phospho[lab]\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load and check env variables\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "from phospho import config\n",
    "\n",
    "assert (\n",
    "    config.OPENAI_API_KEY is not None\n",
    "), \"You need to set the OPENAI_API_KEY environment variable\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the Event detection pipeline\n",
    "\n",
    "In phospho, there are two important concepts:\n",
    "\n",
    "- A workload, which is a set of jobs. Those jobs are run asynchronously and in parallel.\n",
    "- A job, which is a python function that returns a JobResult. Jobs are parametrized with a JobConfig.\n",
    "\n",
    "In this example, the Job is to detect an event (\"Event Detection\") using LLM self-reflection (we asked another LLM whether the event occured or not). We will try to detect the event: \"The user asks a question to the assistant\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phospho import lab\n",
    "\n",
    "# Create a workload in our lab\n",
    "workload = lab.Workload()\n",
    "\n",
    "# Add our job to the workload\n",
    "workload.add_job(\n",
    "    lab.Job(\n",
    "        name=\"event_detection\",\n",
    "        id=\"question_answering\",\n",
    "        config=lab.EventConfig(\n",
    "            event_name=\"Question Answering\",\n",
    "            event_description=\"User asks a question to the assistant\",\n",
    "            model=\"openai:gpt-3.5-turbo\",\n",
    "        ),\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that everything is set up, we can run the pipeline on a Message.\n",
    "\n",
    "We want to detect whether the user asks a question. Sometimes, it's easy: there is a question mark. But sometimes, it's not: you understand that it is a question only through context and semantics. That's why you need an LLM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In message 1, the Event question answering was detected: True\n",
      "In message 2, the Event question answering was detected: False\n",
      "In message 3, the Event question answering was detected: True\n"
     ]
    }
   ],
   "source": [
    "await workload.async_run(\n",
    "    messages=[\n",
    "        # This message is a question, very simple to detect.\n",
    "        lab.Message(\n",
    "            id=\"message_1\",\n",
    "            content=\"What is the capital of France?\",\n",
    "        ),\n",
    "        # This message is not a question, so it should not be detected.\n",
    "        lab.Message(\n",
    "            id=\"message_2\",\n",
    "            content=\"I don't like croissants.\",\n",
    "        ),\n",
    "        # This message is also a question, but it lacks a question mark. You need semantics to detect it.\n",
    "        lab.Message(\n",
    "            id=\"message_3\",\n",
    "            content=\"I wonder what's the capital of France...\",\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "\n",
    "for i in range(1, 4):\n",
    "    print(\n",
    "        f\"In message {i}, the Event question answering was detected: {workload.results['message_'+str(i)]['question_answering'].value}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset analytics\n",
    "\n",
    "Now, let's assume we want to find user questions in a large dataset. How would we do it?\n",
    "\n",
    "Let's load a dataset of messages from huggingface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "app 0.1.0 requires pandas==2.1.4, but you have pandas 2.2.1 which is incompatible.\n",
      "app 0.1.0 requires pydantic==2.5.3, but you have pydantic 2.6.3 which is incompatible.\n",
      "app 0.1.0 requires pytest==7.4.3, but you have pytest 8.0.2 which is incompatible.\n",
      "app 0.1.0 requires python-dotenv==1.0.0, but you have python-dotenv 1.0.1 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/nicolasoulianov/anaconda3/envs/phospho-env-min/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Downloading data: 100%|██████████| 3.61M/3.61M [00:01<00:00, 2.62MB/s]\n",
      "Downloading data: 100%|██████████| 334k/334k [00:00<00:00, 1.46MB/s]\n",
      "Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 1.42MB/s]\n",
      "Generating train split: 100%|██████████| 11118/11118 [00:00<00:00, 405279.52 examples/s]\n",
      "Generating validation split: 100%|██████████| 1000/1000 [00:00<00:00, 310367.32 examples/s]\n",
      "Generating test split: 100%|██████████| 1000/1000 [00:00<00:00, 364817.26 examples/s]\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset(\"daily_dialog\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset has more than 10 000 samples. That's a lot and running analytics on it can quickly become pricy. So let's just select a subsample of it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['dialog', 'act', 'emotion'],\n",
       "    num_rows: 11118\n",
       "})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[\"train\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Say , Jim , how about going for a few beers after dinner ? \n"
     ]
    }
   ],
   "source": [
    "# Generate a sub dataset with 30 messages\n",
    "sub_dataset = dataset[\"train\"].select(range(30))\n",
    "\n",
    "# Let's print one of the messages\n",
    "print(sub_dataset[0][\"dialog\"][0])\n",
    "\n",
    "# Build the message list for our lab\n",
    "messages = []\n",
    "for row in sub_dataset:\n",
    "    text = row[\"dialog\"][0]\n",
    "    messages.append(lab.Message(content=text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's run the analytics pipeline on the subset.\n",
    "\n",
    "The workload run is **asynchronous** and **parallelized**, which means this will go much faster than just writing a for loop.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the lab on it\n",
    "# The job will be run with the default model (openai:gpt-3.5-turbo)\n",
    "workload_results = await workload.async_run(messages=messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Message 81c3f8b29ae34453a88d7a2aceb8404c was a question: False\n",
      "Message 21fc8278403e4889a17ce1cb6897a52c was a question: True\n",
      "Message 8fa427ef1ac24d61a3d3955ec7f3ae1c was a question: True\n",
      "Message a116a00a1b7c4967a66e505726e4188b was a question: False\n",
      "Message d53a2c59d5894dae8efaa6e6217e2fb5 was a question: False\n",
      "Message 3272059df3a94ae9a6cb4dabec30255e was a question: False\n",
      "Message 12a929618ab548eba86fd1d7e7ef7211 was a question: False\n",
      "Message 7dd653af06084bb8a3f9f29f56090206 was a question: False\n",
      "Message 5f5de3812249455c96ec66f61d98a952 was a question: False\n",
      "Message 9d0eb102593a4fea9ea0ca716db856eb was a question: True\n",
      "Message c2bd9d2ed53947deb15bd81269ccca78 was a question: True\n",
      "Message 6b2e908718504ef5ab73f01d398d2fc8 was a question: False\n",
      "Message 0174cad2f84b49fdbf9a91f770b344df was a question: False\n",
      "Message 888e3d7662b949349b049e79217c7587 was a question: True\n",
      "Message d6d1b0c4d8da44358191353d2abf0fd1 was a question: False\n",
      "Message 41bfb2dbe24c4d918e8d6783749718c8 was a question: False\n",
      "Message 8b29e0307f10420aaeb5062a29578af7 was a question: True\n",
      "Message ffc15e866fc54446af80b6598d571650 was a question: True\n",
      "Message e115e4655a8740e4a85f39a196aa34d6 was a question: True\n",
      "Message 8a8f32ded02343bd8459fc729b1e9576 was a question: True\n",
      "Message 535a1df4e7394ac3baedf17125b1effd was a question: False\n",
      "Message 6abf2c4a00144537bccd0ac27cb6e3c8 was a question: True\n",
      "Message 38a7e8de41e14d61b2c05944d7f2b9cd was a question: True\n",
      "Message 7c2c69dad2d34a1c8522b4fb3273682a was a question: False\n",
      "Message 2a8b20baf149426fb622e18444bb11d6 was a question: True\n",
      "Message 1706728a84374e67a07284a7bdff2a69 was a question: False\n",
      "Message d5f2282b80fb408288605bcdd9b75e73 was a question: True\n",
      "Message 90191eb7be804daaac9b7ec5e7caf5bb was a question: False\n",
      "Message 9114ea5ef2a0437aa1242fead8051ce0 was a question: False\n",
      "Message a438e92b56cb403b91d24e585fa49beb was a question: True\n"
     ]
    }
   ],
   "source": [
    "# Print the results\n",
    "for message_id, jobs in workload_results.items():\n",
    "    print(f\"Message {message_id} was a question: {jobs['question_answering'].value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize the pipeline\n",
    "\n",
    "Running semantic analytics with an LLM is great. But it's expensive and slow.\n",
    "\n",
    "You likely want to try other model providers, such as Mistral, or even small local models. But how do they compare?\n",
    "\n",
    "Let's run the pipeline on these models, and then figure out which one matches the reference, GPT-4.\n",
    "\n",
    "For the purpose of this demo, we consider a considertion good enough if it matches gpt-4 on at least 80% of the dataset. Good old Paretto.\n",
    "\n",
    "### Installation and setup\n",
    "\n",
    "You will need:\n",
    "\n",
    "- a Mistral AI API key (find yours [here](https://console.mistral.ai/api-keys/))\n",
    "- Ollama running on your local machine, with the Mistral 7B model installed. You can find the installation instructions for Ollama [here](https://ollama.com)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phospho import config\n",
    "\n",
    "# Check the environment variable\n",
    "assert (\n",
    "    config.MISTRAL_API_KEY is not None\n",
    "), \"You need to set the MISTRAL_API_KEY environment variable\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It's subjective, but a common agreement is that Brie is one of the best French cheeses.\n"
     ]
    }
   ],
   "source": [
    "from phospho.lab.language_models import get_sync_client\n",
    "\n",
    "# Create a client\n",
    "ollama = get_sync_client(\"ollama\")\n",
    "\n",
    "try:\n",
    "    # Let's check we can reach your local Ollama API\n",
    "    response = ollama.chat.completions.create(\n",
    "        model=\"mistral\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": \"What is the best French cheese? Keep your answer short.\",\n",
    "            },\n",
    "        ],\n",
    "    )\n",
    "    print(response.choices[0].message.content)\n",
    "except Exception as e:\n",
    "    print(f\"Error: {e}\")\n",
    "    print(\n",
    "        \"You need to have a local Ollama server running to continue and the mistral model downloaded. \\nRemove references to Ollama otherwise.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the results with the alternative configurations\n",
    "\n",
    "To run the jobs on multiple models at the same time, we will simply set up our job with a different configuration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Literal\n",
    "\n",
    "workload = lab.Workload()\n",
    "\n",
    "\n",
    "# Setup the configs for our job\n",
    "class EventConfig(lab.JobConfig):\n",
    "    event_name: str = \"Question Answering\"\n",
    "    event_description: str = \"User asks a question to the assistant\"\n",
    "    # Model are ordered from the least desired to the most desired\n",
    "    # The default model is set to be the \"reference\"\n",
    "    model: Literal[\n",
    "        \"openai:gpt-4\",\n",
    "        \"mistral:mistral-large-latest\",\n",
    "        \"mistral:mistral-small-latest\",\n",
    "        \"ollama:mistral\",\n",
    "    ] = \"openai:gpt-4\"\n",
    "\n",
    "\n",
    "# Add our job to the workload\n",
    "workload.add_job(\n",
    "    lab.Job(\n",
    "        name=\"event_detection\",\n",
    "        id=\"question_answering\",\n",
    "        config=EventConfig(),\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the workload in the same way.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "workload_results = await workload.async_run(messages=messages, executor_type=\"parallel\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'openai:gpt-4'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note the default model is currently set to \"openai:gpt-4\"\n",
    "workload.jobs[0].config.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's also run the pipeline on alternative models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute alternative results with the Mistral API and Ollama\n",
    "await workload.async_run_on_alternative_configurations(\n",
    "    messages=messages, executor_type=\"parallel\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We ask the workload to figure out which model is better.\n",
    "\n",
    "Note that this can actually work with any set of parameters, not just models. It's a flexible way to perform a grid search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracies: [0.8666666666666667, 0.7333333333333333, 0.6]\n"
     ]
    }
   ],
   "source": [
    "workload.optimize_jobs(accuracy_threshold=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check what model was picked.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mistral:mistral-large-latest'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the new model_id (if it has changed)\n",
    "workload.jobs[0].config.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting! We can run our pipeline with roughly the same accuracy using a smaller model. That's a lot of time, compute and money saved.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run our workload on the full dataset, with optimized parameters\n",
    "\n",
    "Now that we have benchmarked different models for our Event detection pipeline, let's run the optimal configuration on a larger chunk of the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_dataset = dataset[\"train\"].select(\n",
    "    range(200)\n",
    ")  # Here you can just leave it as dataset[\"train\"] if you want to use the whole dataset\n",
    "\n",
    "# Build the message list for our lab\n",
    "messages = []\n",
    "for row in sub_dataset:\n",
    "    text = row[\"dialog\"][0]\n",
    "    messages.append(lab.Message(content=text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to run the evals on a lot of messages in parallel. However, if we just bombarded the model provider API, we'd have a **rate limit error.**\n",
    "\n",
    "Indeed, each model provider has their own limitation. For Mistral, it's 5 requests per second max per default.\n",
    "\n",
    "In order to avoid that, **we set a throttle of maximum 5 requests per second** using the parameter `max_parallelism`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The job will be runned with the best model (mistral:mistral-small-latest in our case)\n",
    "workload_results = await workload.async_run(\n",
    "    messages=messages,\n",
    "    executor_type=\"parallel\",\n",
    "    max_parallelism=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze the results\n",
    "\n",
    "Now, we were trying to see which share of the dataset is actually a question. Let's get the results as a dataframe and visualize them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_answering</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d7a64156f5ca421b91ebc6d1891a80c7</th>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92eddb110aae4295979b58a650b59334</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>021f67dc28df420b9c7035d551893467</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62ad8c82a8474403a44fec7ee9e3a29e</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a0712c8ca02b4ec5accd9e706636ec64</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96460d781a1943b8b23f12fb6b999bac</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>069356ca77fd4747a62ad0ed98fcd9e7</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>643ac36ed5c4423f8de61b263421e281</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86a624b4d1514ee19d97d4073a0ebbc3</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>df36ed76f1fc4aa1be4859f5d18319e4</th>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  question_answering\n",
       "d7a64156f5ca421b91ebc6d1891a80c7               False\n",
       "92eddb110aae4295979b58a650b59334                True\n",
       "021f67dc28df420b9c7035d551893467                True\n",
       "62ad8c82a8474403a44fec7ee9e3a29e                True\n",
       "a0712c8ca02b4ec5accd9e706636ec64                True\n",
       "...                                              ...\n",
       "96460d781a1943b8b23f12fb6b999bac                True\n",
       "069356ca77fd4747a62ad0ed98fcd9e7                True\n",
       "643ac36ed5c4423f8de61b263421e281                True\n",
       "86a624b4d1514ee19d97d4073a0ebbc3                True\n",
       "df36ed76f1fc4aa1be4859f5d18319e4                True\n",
       "\n",
       "[200 rows x 1 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using .results_df() we can get a pandas dataframe with the results\n",
    "workload.results_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are many questions in this dataset...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "question_answering\n",
       "False     99\n",
       "True     101\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workload.results_df().groupby(\"question_answering\").size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a nice plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='question_answering'>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAHFCAYAAADYPwJEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmJUlEQVR4nO3df1RUdf7H8dcgCCjMKKSAGxSl5o80EtRFTU05S67r0SNbetJNzY3WVTd1M2VL81dhpS1ruVq2q9mxttpNym3TCoP8QYSgbD+M/EHKpoOlAYLxI7nfPzzdb5NWWoPzQZ6Pc+acnXvv3HnPtKNP79yZcViWZQkAAMAgfr4eAAAA4NsIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYx9/XA/wYDQ0NOnLkiEJDQ+VwOHw9DgAAOA+WZenkyZPq0KGD/Py+/xhJkwyUI0eOKDo62tdjAACAH6G0tFSXX375927TJAMlNDRU0pkH6HQ6fTwNAAA4H5WVlYqOjrb/Hv8+TTJQvn5bx+l0EigAADQx53N6BifJAgAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwjv+F3uDtt9/WI488ooKCAh09elQbN27UqFGj7PWWZen+++/XmjVrVF5erv79+2vVqlXq1KmTvc2JEyc0ffp0bdq0SX5+fkpJSdFf/vIXhYSEeOVBAUBTdOXcV309Ai6iT5YO9/UIRrvgIyjV1dW67rrrtHLlynOuf/jhh7VixQqtXr1aeXl5at26tZKTk1VTU2NvM27cOH3wwQd644039O9//1tvv/22UlNTf/yjAAAAl5QLPoIybNgwDRs27JzrLMtSRkaG7rvvPo0cOVKStH79ekVERCgzM1Njx47V3r17tXnzZuXn5yshIUGS9Nhjj+mXv/ylli1bpg4dOpy139raWtXW1trXKysrL3RsAADQhHj1HJSSkhK53W4lJSXZy1wul/r27avc3FxJUm5urtq0aWPHiSQlJSXJz89PeXl559xvenq6XC6XfYmOjvbm2AAAwDBeDRS32y1JioiI8FgeERFhr3O73Wrfvr3Hen9/f4WFhdnbfFtaWpoqKirsS2lpqTfHBgAAhrngt3h8ITAwUIGBgb4eAwAAXCRePYISGRkpSSorK/NYXlZWZq+LjIzUsWPHPNZ/9dVXOnHihL0NAABo3rx6BCU2NlaRkZHKyspSXFycpDMntObl5WnKlCmSpMTERJWXl6ugoEDx8fGSpK1bt6qhoUF9+/b15jiXJD6G2LzwMUQAzdUFB0pVVZX2799vXy8pKdGePXsUFhammJgYzZgxQ0uWLFGnTp0UGxurefPmqUOHDvZ3pXTt2lU33XST7rjjDq1evVr19fWaNm2axo4de85P8AAAgObnggNl165duvHGG+3rs2bNkiRNmDBB69at0z333KPq6mqlpqaqvLxcAwYM0ObNmxUUFGTfZsOGDZo2bZqGDh1qf1HbihUrvPBwAADApcBhWZbl6yEuVGVlpVwulyoqKuR0On09zkXFWzzNC2/xNC+8vpuX5vj6vpC/v/ktHgAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAQAAxiFQAACAcQgUAABgHAIFAAAYh0ABAADGIVAAAIBxCBQAAGAcAgUAABjH64Fy+vRpzZs3T7GxsQoODtbVV1+txYsXy7IsexvLsjR//nxFRUUpODhYSUlJ2rdvn7dHAQAATZTXA+Whhx7SqlWr9Pjjj2vv3r166KGH9PDDD+uxxx6zt3n44Ye1YsUKrV69Wnl5eWrdurWSk5NVU1Pj7XEAAEAT5O/tHe7cuVMjR47U8OHDJUlXXnmlnnvuOb377ruSzhw9ycjI0H333aeRI0dKktavX6+IiAhlZmZq7Nix3h4JAAA0MV4/gtKvXz9lZWXp448/liQVFRVp+/btGjZsmCSppKREbrdbSUlJ9m1cLpf69u2r3Nzcc+6ztrZWlZWVHhcAAHDp8voRlLlz56qyslJdunRRixYtdPr0aT3wwAMaN26cJMntdkuSIiIiPG4XERFhr/u29PR0LVy40NujAgAAQ3n9CMoLL7ygDRs26Nlnn1VhYaGefvppLVu2TE8//fSP3mdaWpoqKirsS2lpqRcnBgAApvH6EZTZs2dr7ty59rkkPXr00KFDh5Senq4JEyYoMjJSklRWVqaoqCj7dmVlZYqLizvnPgMDAxUYGOjtUQEAgKG8fgTl1KlT8vPz3G2LFi3U0NAgSYqNjVVkZKSysrLs9ZWVlcrLy1NiYqK3xwEAAE2Q14+gjBgxQg888IBiYmLUvXt37d69W48++qhuv/12SZLD4dCMGTO0ZMkSderUSbGxsZo3b546dOigUaNGeXscAADQBHk9UB577DHNmzdPv//973Xs2DF16NBBd955p+bPn29vc88996i6ulqpqakqLy/XgAEDtHnzZgUFBXl7HAAA0AQ5rG9+xWsTUVlZKZfLpYqKCjmdTl+Pc1FdOfdVX4+Ai+iTpcN9PQIuIl7fzUtzfH1fyN/f/BYPAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4zRKoHz66acaP368wsPDFRwcrB49emjXrl32esuyNH/+fEVFRSk4OFhJSUnat29fY4wCAACaIK8HyhdffKH+/fsrICBAr732mj788EMtX75cbdu2tbd5+OGHtWLFCq1evVp5eXlq3bq1kpOTVVNT4+1xAABAE+Tv7R0+9NBDio6O1tq1a+1lsbGx9v+2LEsZGRm67777NHLkSEnS+vXrFRERoczMTI0dO9bbIwEAgCbG60dQXnnlFSUkJOjmm29W+/btdf3112vNmjX2+pKSErndbiUlJdnLXC6X+vbtq9zc3HPus7a2VpWVlR4XAABw6fJ6oBw8eFCrVq1Sp06dtGXLFk2ZMkV/+MMf9PTTT0uS3G63JCkiIsLjdhEREfa6b0tPT5fL5bIv0dHR3h4bAAAYxOuB0tDQoF69eunBBx/U9ddfr9TUVN1xxx1avXr1j95nWlqaKioq7EtpaakXJwYAAKbxeqBERUWpW7duHsu6du2qw4cPS5IiIyMlSWVlZR7blJWV2eu+LTAwUE6n0+MCAAAuXV4PlP79+6u4uNhj2ccff6wrrrhC0pkTZiMjI5WVlWWvr6ysVF5enhITE709DgAAaIK8/imemTNnql+/fnrwwQd1yy236N1339WTTz6pJ598UpLkcDg0Y8YMLVmyRJ06dVJsbKzmzZunDh06aNSoUd4eBwAANEFeD5TevXtr48aNSktL06JFixQbG6uMjAyNGzfO3uaee+5RdXW1UlNTVV5ergEDBmjz5s0KCgry9jgAAKAJ8nqgSNKvfvUr/epXv/rO9Q6HQ4sWLdKiRYsa4+4BAEATx2/xAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAME6jB8rSpUvlcDg0Y8YMe1lNTY2mTp2q8PBwhYSEKCUlRWVlZY09CgAAaCIaNVDy8/P1xBNPqGfPnh7LZ86cqU2bNunFF19UTk6Ojhw5otGjRzfmKAAAoAlptECpqqrSuHHjtGbNGrVt29ZeXlFRob/97W969NFHNWTIEMXHx2vt2rXauXOn3nnnnXPuq7a2VpWVlR4XAABw6Wq0QJk6daqGDx+upKQkj+UFBQWqr6/3WN6lSxfFxMQoNzf3nPtKT0+Xy+WyL9HR0Y01NgAAMECjBMo//vEPFRYWKj09/ax1brdbLVu2VJs2bTyWR0REyO12n3N/aWlpqqiosC+lpaWNMTYAADCEv7d3WFpaqrvuuktvvPGGgoKCvLLPwMBABQYGemVfAADAfF4/glJQUKBjx46pV69e8vf3l7+/v3JycrRixQr5+/srIiJCdXV1Ki8v97hdWVmZIiMjvT0OAABogrx+BGXo0KF67733PJZNmjRJXbp00Zw5cxQdHa2AgABlZWUpJSVFklRcXKzDhw8rMTHR2+MAAIAmyOuBEhoaqmuvvdZjWevWrRUeHm4vnzx5smbNmqWwsDA5nU5Nnz5diYmJ+vnPf+7tcQAAQBPk9UA5H3/+85/l5+enlJQU1dbWKjk5WX/96199MQoAADDQRQmU7Oxsj+tBQUFauXKlVq5ceTHuHgAANDH8Fg8AADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACM4/VASU9PV+/evRUaGqr27dtr1KhRKi4u9timpqZGU6dOVXh4uEJCQpSSkqKysjJvjwIAAJoorwdKTk6Opk6dqnfeeUdvvPGG6uvr9Ytf/ELV1dX2NjNnztSmTZv04osvKicnR0eOHNHo0aO9PQoAAGii/L29w82bN3tcX7dundq3b6+CggINHDhQFRUV+tvf/qZnn31WQ4YMkSStXbtWXbt21TvvvKOf//zn3h4JAAA0MY1+DkpFRYUkKSwsTJJUUFCg+vp6JSUl2dt06dJFMTExys3NPec+amtrVVlZ6XEBAACXrkYNlIaGBs2YMUP9+/fXtddeK0lyu91q2bKl2rRp47FtRESE3G73OfeTnp4ul8tlX6KjoxtzbAAA4GONGihTp07V+++/r3/84x8/aT9paWmqqKiwL6WlpV6aEAAAmMjr56B8bdq0afr3v/+tt99+W5dffrm9PDIyUnV1dSovL/c4ilJWVqbIyMhz7iswMFCBgYGNNSoAADCM14+gWJaladOmaePGjdq6datiY2M91sfHxysgIEBZWVn2suLiYh0+fFiJiYneHgcAADRBXj+CMnXqVD377LN6+eWXFRoaap9X4nK5FBwcLJfLpcmTJ2vWrFkKCwuT0+nU9OnTlZiYyCd4AACApEYIlFWrVkmSBg8e7LF87dq1mjhxoiTpz3/+s/z8/JSSkqLa2lolJyfrr3/9q7dHAQAATZTXA8WyrB/cJigoSCtXrtTKlSu9ffcAAOASwG/xAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMA6BAgAAjEOgAAAA4xAoAADAOAQKAAAwDoECAACMQ6AAAADjECgAAMA4BAoAADAOgQIAAIxDoAAAAOMQKAAAwDgECgAAMI5PA2XlypW68sorFRQUpL59++rdd9/15TgAAMAQPguU559/XrNmzdL999+vwsJCXXfddUpOTtaxY8d8NRIAADCEzwLl0Ucf1R133KFJkyapW7duWr16tVq1aqW///3vvhoJAAAYwt8Xd1pXV6eCggKlpaXZy/z8/JSUlKTc3Nyztq+trVVtba19vaKiQpJUWVnZ+MMapqH2lK9HwEXUHP8/3pzx+m5emuPr++vHbFnWD27rk0D5/PPPdfr0aUVERHgsj4iI0EcffXTW9unp6Vq4cOFZy6OjoxttRsAErgxfTwCgsTTn1/fJkyflcrm+dxufBMqFSktL06xZs+zrDQ0NOnHihMLDw+VwOHw4GS6GyspKRUdHq7S0VE6n09fjAPAiXt/Ni2VZOnnypDp06PCD2/okUC677DK1aNFCZWVlHsvLysoUGRl51vaBgYEKDAz0WNamTZvGHBEGcjqd/AEGXKJ4fTcfP3Tk5Gs+OUm2ZcuWio+PV1ZWlr2soaFBWVlZSkxM9MVIAADAID57i2fWrFmaMGGCEhIS1KdPH2VkZKi6ulqTJk3y1UgAAMAQPguUMWPG6LPPPtP8+fPldrsVFxenzZs3n3XiLBAYGKj777//rLf5ADR9vL7xXRzW+XzWBwAA4CLit3gAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFADARbVt2zaNHz9eiYmJ+vTTTyVJzzzzjLZv3+7jyWASAgVGq6urU3Fxsb766itfjwLAC/71r38pOTlZwcHB2r17t/1L9RUVFXrwwQd9PB1MQqDASKdOndLkyZPVqlUrde/eXYcPH5YkTZ8+XUuXLvXxdAB+rCVLlmj16tVas2aNAgIC7OX9+/dXYWGhDyeDaQgUGCktLU1FRUXKzs5WUFCQvTwpKUnPP/+8DycD8FMUFxdr4MCBZy13uVwqLy+/+APBWAQKjJSZmanHH39cAwYMkMPhsJd3795dBw4c8OFkAH6KyMhI7d+//6zl27dv11VXXeWDiWAqAgVG+uyzz9S+ffuzlldXV3sEC4Cm5Y477tBdd92lvLw8ORwOHTlyRBs2bNDdd9+tKVOm+Ho8GMRnPxYIfJ+EhAS9+uqrmj59uiTZUfLUU08pMTHRl6MB+Anmzp2rhoYGDR06VKdOndLAgQMVGBiou+++2369AxI/FghDbd++XcOGDdP48eO1bt063Xnnnfrwww+1c+dO5eTkKD4+3tcjAvgJ6urqtH//flVVValbt24KCQnx9UgwDIECYx04cEBLly5VUVGRqqqq1KtXL82ZM0c9evTw9WgAgEZGoAAALpobb7zxe88j27p160WcBibjHBQYqbCwUAEBAfbRkpdffllr165Vt27dtGDBArVs2dLHEwL4MeLi4jyu19fXa8+ePXr//fc1YcIE3wwFI3EEBUbq3bu35s6dq5SUFB08eFDdunXT6NGjlZ+fr+HDhysjI8PXIwLwogULFqiqqkrLli3z9SgwBIECI7lcLhUWFurqq6/WQw89pK1bt2rLli3asWOHxo4dq9LSUl+PCMCL9u/frz59+ujEiRO+HgWG4HtQYCTLstTQ0CBJevPNN/XLX/5SkhQdHa3PP//cl6MBaAS5ubke3xoNcA4KjJSQkKAlS5YoKSlJOTk5WrVqlSSppKREERERPp4OwI81evRoj+uWZeno0aPatWuX5s2b56OpYCICBUbKyMjQuHHjlJmZqXvvvVcdO3aUJP3zn/9Uv379fDwdgB/L5XJ5XPfz89M111yjRYsW6Re/+IWPpoKJOAcFTUpNTY1atGjh8SuoAJqG06dPa8eOHerRo4fatm3r63FgOAIFAHDRBAUFae/evYqNjfX1KDAcb/HAGG3btj3vHwLkTH+gabr22mt18OBBAgU/iECBMfhuE+DSt2TJEt19991avHix4uPj1bp1a4/1TqfTR5PBNLzFAwBodIsWLdIf//hHhYaG2su+ecTUsiw5HA6dPn3aF+PBQAQKjFdTU6O6ujqPZfwrC2haWrRooaNHj2rv3r3fu92gQYMu0kQwHYECI1VXV2vOnDl64YUXdPz48bPW868soGnx8/OT2+1W+/btfT0Kmgi+SRZGuueee7R161atWrVKgYGBeuqpp7Rw4UJ16NBB69ev9/V4AH6E8z0JHpA4ggJDxcTEaP369Ro8eLCcTqcKCwvVsWNHPfPMM3ruuef0n//8x9cjArgAfn5+crlcPxgpfEIPX+NTPDDSiRMndNVVV0k6c77J139oDRgwQFOmTPHlaAB+pIULF571TbLAdyFQYKSrrrpKJSUliomJUZcuXfTCCy+oT58+2rRpk9q0aePr8QD8CGPHjuUcFJw3zkGBUQ4ePKiGhgZNmjRJRUVFkqS5c+dq5cqVCgoK0syZMzV79mwfTwngQnH+CS4U56DAKF9/FPHrf2WNGTNGK1asUE1NjQoKCtSxY0f17NnTx1MCuFB8igcXikCBUb79h1hoaKiKiors81EAAM0Db/EAAADjECgwisPhOOu9at67BoDmh0/xwCiWZWnixIkKDAyUdOZr7n/3u9+d9YNiL730ki/GAwBcJAQKjDJhwgSP6+PHj/fRJAAAX+IkWQAAYBzOQQEAAMYhUAAAgHEIFAAAYBwCBQAAGIdAAfC9HA6HMjMzfT1GkzN48GDNmDHD12MATRaf4gEgSVqwYIEyMzO1Z88ej+Vut1tt27a1v5sG5+fEiRMKCAhQaGior0cBmiS+BwXA94qMjPT1CE1KXV2dWrZsqbCwMF+PAjRpvMUDGKK6ulq33XabQkJCFBUVpeXLl3u8TXCut1ratGmjdevW2ddLS0t1yy23qE2bNgoLC9PIkSP1ySef2Ouzs7PVp08ftW7dWm3atFH//v116NAhrVu3TgsXLlRRUZH9cwNf7/fb9/vee+9pyJAhCg4OVnh4uFJTU1VVVWWvnzhxokaNGqVly5YpKipK4eHhmjp1qurr68/reXjmmWeUkJCg0NBQRUZG6tZbb9WxY8c8HoPD4VBWVpYSEhLUqlUr9evXT8XFxfY2RUVFuvHGGxUaGiqn06n4+Hjt2rVLlmWpXbt2+uc//2lvGxcXp6ioKPv69u3bFRgYqFOnTkmSysvL9dvf/lbt2rWT0+nUkCFDVFRUZG+/YMECxcXF6amnnlJsbKyCgoIknf0Wz5VXXqkHH3xQt99+u0JDQxUTE6Mnn3zS47Hv3LlTcXFxCgoKUkJCgjIzM+VwOM46qgU0BwQKYIjZs2crJydHL7/8sl5//XVlZ2ersLDwvG9fX1+v5ORkhYaGatu2bdqxY4dCQkJ00003qa6uTl999ZVGjRqlQYMG6b///a9yc3OVmpoqh8OhMWPG6I9//KO6d++uo0eP6ujRoxozZsxZ91FdXa3k5GS1bdtW+fn5evHFF/Xmm29q2rRpHtu99dZbOnDggN566y09/fTTWrdunUdI/dDjWLx4sYqKipSZmalPPvlEEydOPGu7e++9V8uXL9euXbvk7++v22+/3V43btw4XX755crPz1dBQYHmzp2rgIAAORwODRw4UNnZ2ZKkL774Qnv37tWXX36pjz76SJKUk5Oj3r17q1WrVpKkm2++WceOHdNrr72mgoIC9erVS0OHDtWJEyfs+9u/f7/+9a9/6aWXXvremFi+fLkSEhK0e/du/f73v9eUKVPssKqsrNSIESPUo0cPFRYWavHixZozZ855PWfAJckC4HMnT560WrZsab3wwgv2suPHj1vBwcHWXXfdZVmWZUmyNm7c6HE7l8tlrV271rIsy3rmmWesa665xmpoaLDX19bWWsHBwdaWLVus48ePW5Ks7Ozsc85w//33W9ddd91Zy795v08++aTVtm1bq6qqyl7/6quvWn5+fpbb7bYsy7ImTJhgXXHFFdZXX31lb3PzzTdbY8aMOd+nw0N+fr4lyTp58qRlWZb11ltvWZKsN99802MGSdaXX35pWZZlhYaGWuvWrTvn/lasWGF1797dsizLyszMtPr27WuNHDnSWrVqlWVZlpWUlGT96U9/sizLsrZt22Y5nU6rpqbGYx9XX3219cQTT1iWdeZ5CwgIsI4dO+axzaBBg+z/dpZlWVdccYU1fvx4+3pDQ4PVvn17+35XrVplhYeH24/BsixrzZo1liRr9+7d5/dkAZcQjqAABjhw4IDq6urUt29fe1lYWJiuueaa895HUVGR9u/fr9DQUIWEhCgkJERhYWGqqanRgQMHFBYWpokTJyo5OVkjRozQX/7yFx09evSC5ty7d6+uu+46jx9v7N+/vxoaGjzeYunevbtatGhhX4+KivJ4m+b7FBQUaMSIEYqJiVFoaKgGDRokSTp8+LDHdj179vTYvyT7PmbNmqXf/va3SkpK0tKlS3XgwAF720GDBunDDz/UZ599ppycHA0ePFiDBw9Wdna26uvrtXPnTg0ePFjSmee0qqpK4eHh9nMaEhKikpISj31eccUVateu3Q8+tm/O7HA4FBkZac9cXFysnj172m8RSVKfPn3O6zkDLkUECtBEOBwOWd/60N03z+uoqqpSfHy89uzZ43H5+OOPdeutt0qS1q5dq9zcXPXr10/PP/+8OnfurHfeecfrswYEBJw1e0NDww/e7uu3kJxOpzZs2KD8/Hxt3LhR0pmTT7/rPhwOhyTZ97FgwQJ98MEHGj58uLZu3apu3brZ++nRo4fCwsKUk5PjESg5OTnKz89XfX29+vXrJ+nMcxoVFXXWc1pcXKzZs2fb9//tX9v29vMCNEcECmCAq6++WgEBAcrLy7OXffHFF/r444/t6+3atfM44rFv3z77RE5J6tWrl/bt26f27durY8eOHheXy2Vvd/311ystLU07d+7Utddeq2effVaS1LJlS50+ffp75+zatauKiopUXV1tL9uxY4f8/Pwu6GjPd/noo490/PhxLV26VDfccIO6dOly3kdevq1z586aOXOmXn/9dY0ePVpr166VdCYKbrjhBr388sv64IMPNGDAAPXs2VO1tbV64oknlJCQYAdHr1695Ha75e/vf9Zzetlll/3kx/tN11xzjd577z3V1tbay/Lz8716H0BTQqAABggJCdHkyZM1e/Zsbd26Ve+//74mTpwoP7//f4kOGTJEjz/+uHbv3q1du3bpd7/7nce/yMeNG6fLLrtMI0eO1LZt21RSUqLs7Gz94Q9/0P/+9z+VlJQoLS1Nubm5OnTokF5//XXt27dPXbt2lXTmUyYlJSXas2ePPv/8c4+/KL95H0FBQZowYYLef/99vfXWW5o+fbp+85vfKCIi4ic/DzExMWrZsqUee+wxHTx4UK+88ooWL158Qfv48ssvNW3aNGVnZ+vQoUPasWOH8vPz7ccpnfmEzXPPPae4uDiFhITIz89PAwcO1IYNG+y3lCQpKSlJiYmJGjVqlF5//XV98skn2rlzp+69917t2rXrJz/eb7r11lvV0NCg1NRU7d27V1u2bNGyZcsk/f8RIqA5IVAAQzzyyCO64YYbNGLECCUlJWnAgAGKj4+31y9fvlzR0dG64YYbdOutt+ruu++2P2kiSa1atdLbb7+tmJgYjR49Wl27dtXkyZNVU1Mjp9OpVq1a6aOPPlJKSoo6d+6s1NRUTZ06VXfeeackKSUlRTfddJNuvPFGtWvXTs8999xZM7Zq1UpbtmzRiRMn1Lt3b/3617/W0KFD9fjjj3vlOWjXrp3WrVunF198Ud26ddPSpUvtv6TPV4sWLXT8+HHddttt6ty5s2655RYNGzZMCxcutLcZNGiQTp8+bZ9rIp2Jlm8vczgc+s9//qOBAwdq0qRJ6ty5s8aOHatDhw55Jci+yel0atOmTdqzZ4/i4uJ07733av78+ZLkcV4K0FzwTbKAwQYPHqy4uDhlZGT4ehT4wIYNGzRp0iRVVFQoODjY1+MAFxXfJAsAhli/fr2uuuoq/exnP1NRUZHmzJmjW265hThBs0SgALhotm3bpmHDhn3n+m9+I21z5Ha7NX/+fLndbkVFRenmm2/WAw884OuxAJ/gLR4AF82XX36pTz/99DvXd+zY8SJOA8BkBAoAADAOn+IBAADGIVAAAIBxCBQAAGAcAgUAABiHQAEAAMYhUAAAgHEIFAAAYJz/A7cK6QF/6C7uAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "workload.results_df().groupby(\"question_answering\").size().plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Going further\n",
    "\n",
    "You can use the `lab` to run other tasks, such as:\n",
    "\n",
    "- Named Entity Recognition\n",
    "- Sentiment Analysis\n",
    "- Evaluations\n",
    "- And more!\n",
    "\n",
    "You can also play around with different models, different hyperparameters, and different datasets.\n",
    "\n",
    "You want to have such analysis on your own LLM app, in real time? Check out the cloud hosted version of phospho, available on [phospho.ai](https://phospho.ai)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}