2-annotation-options.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7ccd7811-5e67-4ccf-9f0b-307596acfdad",
   "metadata": {
    "tags": [
     "remove"
    ]
   },
   "outputs": [],
   "source": [
    "import hashlib\n",
    "import numpy as np\n",
    "import pybryt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0d09f6-0794-422a-87c0-112427284c91",
   "metadata": {},
   "source": [
    "There are a few options that are common to all annotations; in the last module, you learned about the `success_message` and `failure_message` options. In this module, we'll discuss three more options that can be applied to annotations and how they can be used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d52be876-48d1-45d6-a6ce-f69d8b5927f5",
   "metadata": {},
   "source": [
    "## `name`\n",
    "\n",
    "The `name` option is used to group different instances of annotation classes that represent the same annotation together. This is mainly used to prevent messages from being displayed multiple times when they don't need to be. Let's consider an example: the `maximum` function below just calls Python's `max` function to check that the student correctly identified the maximum, but one success message is printed for each input the function is tested on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "10de6cc9-9392-44af-a44d-6c150c985503",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: maximum\n",
      "SATISFIED: True\n",
      "MESSAGES:\n",
      "  - Found the max!\n",
      "  - Found the max!\n",
      "  - Found the max!\n",
      "  - Found the max!\n"
     ]
    }
   ],
   "source": [
    "max_ref = []\n",
    "def maximum(l, track=False):\n",
    "    m = max(l)\n",
    "    if track:\n",
    "        max_ref.append(pybryt.Value(\n",
    "            m,\n",
    "            success_message=\"Found the max!\", \n",
    "            failure_message=\"Did not find the max\",\n",
    "        ))\n",
    "    return m\n",
    "\n",
    "test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]\n",
    "for test_list in test_lists:\n",
    "    maximum(test_list, track=True)\n",
    "\n",
    "max_ref = pybryt.ReferenceImplementation(\"maximum\", max_ref)\n",
    "\n",
    "with pybryt.check(max_ref):\n",
    "    for test_list in test_lists:\n",
    "        maximum(test_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d8f80f-6469-4e14-b073-33f6d3a7b254",
   "metadata": {},
   "source": [
    "The problem with this is simple: the annotation being created in each test is fundamentally checking the same thing, whether the student returned the correct value. Having the same message printed multiple times makes it seem like the annotations are testing different conditions and clutters the report generated by PyBryt. We can collapse all of these messages together by naming the annotation created in the `maximum` function: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f840d8da-5f9d-407f-93e0-d6d08df2be81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: maximum\n",
      "SATISFIED: True\n",
      "MESSAGES:\n",
      "  - Found the max!\n"
     ]
    }
   ],
   "source": [
    "max_ref = []\n",
    "def maximum(l, track=False):\n",
    "    m = max(l)\n",
    "    if track:\n",
    "        max_ref.append(pybryt.Value(\n",
    "            m,\n",
    "            name=\"list-maximum\",\n",
    "            success_message=\"Found the max!\", \n",
    "            failure_message=\"Did not find the max\",\n",
    "        ))\n",
    "    return m\n",
    "\n",
    "test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]\n",
    "for test_list in test_lists:\n",
    "    maximum(test_list, track=True)\n",
    "\n",
    "max_ref = pybryt.ReferenceImplementation(\"maximum\", max_ref)\n",
    "\n",
    "with pybryt.check(max_ref):\n",
    "    for test_list in test_lists:\n",
    "        maximum(test_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d951f20-84b9-434a-9fcc-717cdfae34fc",
   "metadata": {},
   "source": [
    "Now, we can see that the message is only printed once.\n",
    "\n",
    "When PyBryt collapses the annotations into a single message, it will only display the success message if all of the annotations in the name group are satisfied; if any fails, it will display the failure message instead. Let's introduce a bug into `maximum` to demonstrate this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "16c1e1d8-83d9-40ae-b70d-d616c5e3a25e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: maximum\n",
      "SATISFIED: False\n",
      "MESSAGES:\n",
      "  - Did not find the max\n"
     ]
    }
   ],
   "source": [
    "def maximum(l):\n",
    "    if len(l) % 2 == 0:\n",
    "        m = min(l)\n",
    "    else:\n",
    "        m = max(l)\n",
    "    return m\n",
    "\n",
    "with pybryt.check(max_ref):\n",
    "    for test_list in test_lists:\n",
    "        maximum(test_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "899e76d7-ae0d-4c10-8128-ee28d715bd33",
   "metadata": {},
   "source": [
    "## `limit`\n",
    "\n",
    "The `limit` option allows you to control how many copies of named annotations are included in the reference implementation. This helps for cases in which the functions constructing the annotations are reused many times throughout an assignment but a few initial tests are sufficient for checking the validity of the implementation by reducing the size of the reference implementation itself.\n",
    "\n",
    "Let's illustrate this using our maximum function. We'll use a similar implementation to the reference above but set `limit` to five annotations and test it on several input lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1ff66f0-916d-4855-86a2-dfb2115dfe08",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Anntoations created: 1000\n",
      "Anntoations in reference: 5\n"
     ]
    }
   ],
   "source": [
    "max_ref = []\n",
    "def maximum(l, track=False):\n",
    "    m = max(l)\n",
    "    if track:\n",
    "        max_ref.append(pybryt.Value(\n",
    "            m,\n",
    "            name=\"list-maximum\",\n",
    "            limit=5,\n",
    "            success_message=\"Found the max!\", \n",
    "            failure_message=\"Did not find the max\",\n",
    "        ))\n",
    "    return m\n",
    "\n",
    "for _ in range(1000):\n",
    "    test_list = np.random.normal(size=100)\n",
    "    maximum(test_list, track=True)\n",
    "\n",
    "print(f\"Anntoations created: {len(max_ref)}\")\n",
    "max_ref = pybryt.ReferenceImplementation(\"maximum\", max_ref)\n",
    "print(f\"Anntoations in reference: {len(max_ref.annotations)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d8db096-5807-4705-a847-99534185e877",
   "metadata": {},
   "source": [
    "As you can see, the length of `max_ref.annotations` is 5 even though 1,000 annotations were included in the list passed to the constructor."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "435dbf0e-5627-4b1f-b751-fc14d4a77a45",
   "metadata": {
    "tags": []
   },
   "source": [
    "## `group`\n",
    "\n",
    "The `group` option is similar to the `name` option in that it's used to group annotations together, but these annotations don't necessarily represent the \"same annotation\"; instead, they're grouped into meaningful chunks so that specific portions of references can be checked one at a time instead of all at once. This can be useful in constructing assignments with multiple questions in PyBryt.\n",
    "\n",
    "For example, consider a simple assignment that asks students to implement a `mean` and `median` function. You may divide it up into two questions like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "619bb248-3d8b-4035-bc89-fa6361e9f4e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 1\n",
    "mean_ref = []\n",
    "\n",
    "def mean(l, track=False):\n",
    "    size = len(l)\n",
    "    if track:\n",
    "        mean_ref.append(pybryt.Value(\n",
    "            size,\n",
    "            name=\"len\",\n",
    "            group=\"mean\",\n",
    "            success_message=\"Determined the length of the list\",\n",
    "        ))\n",
    "\n",
    "    m = sum(l) / size\n",
    "    if track:\n",
    "        mean_ref.append(pybryt.Value(\n",
    "            m,\n",
    "            name=\"mean\",\n",
    "            group=\"mean\",\n",
    "            success_message=\"Calculated the correct mean of the list\",\n",
    "            failure_message=\"Did not find the correct mean of the list\",\n",
    "        ))\n",
    "\n",
    "    return m\n",
    "\n",
    "# Question 2\n",
    "median_ref = []\n",
    "\n",
    "def median(l, track=True):\n",
    "    sorted_l = sorted(l)\n",
    "    if track:\n",
    "        median_ref.append(pybryt.Value(\n",
    "            sorted_l,\n",
    "            name=\"sorted\",\n",
    "            group=\"median\",\n",
    "            success_message=\"Sorted the list\",\n",
    "        ))\n",
    "    \n",
    "    size = len(l)\n",
    "    if track:\n",
    "        mean_ref.append(pybryt.Value(\n",
    "            size,\n",
    "            name=\"len\",\n",
    "            group=\"median\",\n",
    "            success_message=\"Determined the length of the list\",\n",
    "        ))\n",
    "\n",
    "    middle = size // 2\n",
    "    is_set_size_even = size % 2 == 0\n",
    "\n",
    "    if is_set_size_even:\n",
    "        m = (sorted_l[middle - 1] + sorted_l[middle]) / 2\n",
    "    else:\n",
    "        m = sorted_l[middle]\n",
    "\n",
    "    if track:\n",
    "        mean_ref.append(pybryt.Value(\n",
    "            m,\n",
    "            name=\"mean\",\n",
    "            group=\"mean\",\n",
    "            success_message=\"Calculated the correct mean of the list\",\n",
    "            failure_message=\"Did not find the correct mean of the list\",\n",
    "        ))\n",
    "\n",
    "    return m\n",
    "\n",
    "test_lists = [[1, 2, 3], [-1, 0, 1], [10, -4, 2, 0], [1]]\n",
    "for test_list in test_lists:\n",
    "    mean(test_list, track=True)\n",
    "    median(test_list, track=True)\n",
    "\n",
    "assignment_ref = pybryt.ReferenceImplementation(\"mean-median\", [*mean_ref, *median_ref])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0370deab-601c-4c5e-9fba-f53ad7b52cea",
   "metadata": {},
   "source": [
    "With a reference constructed as above, we can give students a chance to check their work on each individual question before moving on to the next by telling PyBryt which group of annotations to consider:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d69e21c-159b-4fbb-aa65-4a6e6cb9de1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: mean-median\n",
      "SATISFIED: True\n",
      "MESSAGES:\n",
      "  - Determined the length of the list\n",
      "  - Calculated the correct mean of the list\n"
     ]
    }
   ],
   "source": [
    "with pybryt.check(assignment_ref, group=\"mean\"):\n",
    "    for test_list in test_lists:\n",
    "        mean(test_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ab8f2144-4fb8-47ed-8de6-047441f1f96c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: mean-median\n",
      "SATISFIED: True\n",
      "MESSAGES:\n",
      "  - Determined the length of the list\n",
      "  - Sorted the list\n"
     ]
    }
   ],
   "source": [
    "with pybryt.check(assignment_ref, group=\"median\"):\n",
    "    for test_list in test_lists:\n",
    "        median(test_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "55c9211e-3cd4-45a5-9a24-e934555a768d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REFERENCE: mean-median\n",
      "SATISFIED: True\n",
      "MESSAGES:\n",
      "  - Determined the length of the list\n",
      "  - Calculated the correct mean of the list\n",
      "  - Sorted the list\n"
     ]
    }
   ],
   "source": [
    "with pybryt.check(assignment_ref):\n",
    "    for test_list in test_lists:\n",
    "        mean(test_list)\n",
    "        median(test_list)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}