Diagnosis/Week1/C1_W1_Lab_2_counting_labels_and_weighted_loss_function.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting Labels & Weighted Loss Function\n",
    "\n",
    "As you saw in the lecture videos, one way to avoid having class imbalance impact the loss function is to weight the losses differently.  To choose the weights, you first need to calculate the class frequencies.\n",
    "\n",
    "For this exercise, you'll just get the count of each label.  Later on, you'll use the concepts practiced here to calculate frequencies in the assignment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the necessary packages\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read csv file containing training datadata\n",
    "train_df = pd.read_csv(\"data/nih/train-small.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count up the number of instances of each class (drop non-class columns from the counts)\n",
    "class_counts = train_df.sum().drop(['Image','PatientId'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for column in class_counts.keys():\n",
    "    print(f\"The class {column} has {train_df[column].sum()} samples\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot up the distribution of counts\n",
    "sns.barplot(class_counts.values, class_counts.index, color='b')\n",
    "plt.title('Distribution of Classes for Training Dataset', fontsize=15)\n",
    "plt.xlabel('Number of Patients', fontsize=15)\n",
    "plt.ylabel('Diseases', fontsize=15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Weighted Loss Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is an example of calculating weighted loss.  In the assignment, you will calculate a weighted loss function.  This sample code will give you some intuition for what the weighted loss function is doing, and also help you practice some syntax you will use in the graded assignment.\n",
    "\n",
    "For this example, you'll first define a hypothetical set of true labels and then a set of predictions.\n",
    "\n",
    "Run the next cell to create the 'ground truth' labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate an array of 4 binary label values, 3 positive and 1 negative\n",
    "y_true = np.array(\n",
    "        [[1],\n",
    "         [1],\n",
    "         [1],\n",
    "         [0]])\n",
    "print(f\"y_true: \\n{y_true}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Understanding Weighted Loss Function\n",
    "\n",
    "#### 1.1.1 Two Models\n",
    "\n",
    "To better understand the loss function, you will pretend that you have two models.\n",
    "- Model 1 always outputs a 0.9 for any example that it's given.  \n",
    "- Model 2 always outputs a 0.1 for any example that it's given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make model predictions that are always 0.9 for all examples\n",
    "y_pred_1 = 0.9 * np.ones(y_true.shape)\n",
    "print(f\"y_pred_1: \\n{y_pred_1}\")\n",
    "print()\n",
    "y_pred_2 = 0.1 * np.ones(y_true.shape)\n",
    "print(f\"y_pred_2: \\n{y_pred_2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Problems with the Regular Loss Function\n",
    "The learning goal here is to notice that with a regular loss function (not a weighted loss), the model that always outputs 0.9 has a smaller loss (performs better) than model 2.\n",
    "- This is because there is a class imbalance, where 3 out of the 4 labels are 1.\n",
    "- If the data were perfectly balanced, (two labels were 1, and two labels were 0), model 1 and model 2 would have the same loss.  Each would get two examples correct and two examples incorrect.\n",
    "- However, since the data is not balanced, the regular loss function implies that model 1 is better than model 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1 Shortcomings of a Regular Non-weighted Loss\n",
    "\n",
    "See what loss you get from these two models (model 1 always predicts 0.9, and model 2 always predicts 0.1), see what the regular (unweighted) loss function is for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_reg_1 = -1 * np.sum(y_true * np.log(y_pred_1)) + \\\n",
    "                -1 * np.sum((1 - y_true) * np.log(1 - y_pred_1))\n",
    "print(f\"loss_reg_1: {loss_reg_1:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_reg_2 = -1 * np.sum(y_true * np.log(y_pred_2)) + \\\n",
    "                -1 * np.sum((1 - y_true) * np.log(1 - y_pred_2))\n",
    "print(f\"loss_reg_2: {loss_reg_2:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"When the model 1 always predicts 0.9, the regular loss is {loss_reg_1:.4f}\")\n",
    "print(f\"When the model 2 always predicts 0.1, the regular loss is {loss_reg_2:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the loss function gives a greater loss when the predictions are always 0.1, because the data is imbalanced, and has three labels of `1` but only one label for `0`.\n",
    "\n",
    "Given a class imbalance with more positive labels, the regular loss function implies that the model with the higher prediction of 0.9 performs better than the model with the lower prediction of 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Treatment by a Weighted Loss on the Two Models\n",
    "With a weighted loss function, you will get the same weighted loss when the predictions are all 0.9 versus when the predictions are all 0.1.  \n",
    "- Notice how a prediction of 0.9 is 0.1 away from the positive label of 1.\n",
    "- Also notice how a prediction of 0.1 is 0.1 away from the negative label of 0\n",
    "- So model 1 and 2 are \"symmetric\" along the midpoint of 0.5, if you plot them on a number line between 0 and 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1 Weighted Loss Equation\n",
    "Calculate the loss for the zero-th label (column at index 0)\n",
    "\n",
    "- The loss is made up of two terms.  To make it easier to read the code, you will calculate each of these terms separately.  We are giving each of these two terms a name for explanatory purposes, but these are not officially called $loss_{pos}$ or $loss_{neg}$\n",
    "\n",
    "    - $loss_{pos}$: we'll use this to refer to the loss where the actual label is positive (the positive examples).\n",
    "    - $loss_{neg}$: we'll use this to refer to the loss where the actual label is negative (the negative examples).  \n",
    "\n",
    "$$ loss^{(i)} = loss_{pos}^{(i)} + los_{neg}^{(i)} $$\n",
    "\n",
    "$$loss_{pos}^{(i)} = -1 \\times weight_{pos}^{(i)} \\times y^{(i)} \\times log(\\hat{y}^{(i)})$$\n",
    "\n",
    "$$loss_{neg}^{(i)} = -1 \\times weight_{neg}^{(i)} \\times (1- y^{(i)}) \\times log(1 - \\hat{y}^{(i)})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this sample dataset is small enough, you can calculate the positive weight to be used in the weighted loss function.  To get the positive weight, count how many NEGATIVE labels are present, divided by the total number of examples.\n",
    "\n",
    "In this case, there is one negative label, and four total examples.\n",
    "\n",
    "Similarly, the negative weight is the fraction of positive labels.\n",
    "\n",
    "Run the next cell to define positive and negative weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the positive weight as the fraction of negative labels\n",
    "w_p = 1/4\n",
    "\n",
    "# calculate the negative weight as the fraction of positive labels\n",
    "w_n = 3/4\n",
    "\n",
    "print(f\"positive weight w_p: {w_p}\")\n",
    "print(f\"negative weight w_n {w_n}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2 Weighted Loss: Model 1\n",
    "\n",
    "Run the next two cells to calculate the two loss terms separately.\n",
    "\n",
    "Here, `loss_1_pos` and `loss_1_neg` are calculated using the `y_pred_1` predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate and print out the first term in the loss function, which we are calling 'loss_pos'\n",
    "loss_1_pos = -1 * np.sum(w_p * y_true * np.log(y_pred_1 ))\n",
    "print(f\"loss_1_pos: {loss_1_pos:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate and print out the second term in the loss function, which we're calling 'loss_neg'\n",
    "loss_1_neg = -1 * np.sum(w_n * (1 - y_true) * np.log(1 - y_pred_1 ))\n",
    "print(f\"loss_1_neg: {loss_1_neg:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sum positive and negative losses to calculate total loss\n",
    "loss_1 = loss_1_pos + loss_1_neg\n",
    "print(f\"loss_1: {loss_1:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.3 Weighted Loss: Model 2\n",
    "\n",
    "Now do the same calculations for when the predictions are from `y_pred_2'.  Calculate the two terms of the weighted loss function and add them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate and print out the first term in the loss function, which we are calling 'loss_pos'\n",
    "loss_2_pos = -1 * np.sum(w_p * y_true * np.log(y_pred_2))\n",
    "print(f\"loss_2_pos: {loss_2_pos:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate and print out the second term in the loss function, which we're calling 'loss_neg'\n",
    "loss_2_neg = -1 * np.sum(w_n * (1 - y_true) * np.log(1 - y_pred_2))\n",
    "print(f\"loss_2_neg: {loss_2_neg:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sum positive and negative losses to calculate total loss when the prediction is y_pred_2\n",
    "loss_2 = loss_2_pos + loss_2_neg\n",
    "print(f\"loss_2: {loss_2:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.4 Comparing Model 1 and Model 2 Weighted Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"When the model always predicts 0.9, the total loss is {loss_1:.4f}\")\n",
    "print(f\"When the model always predicts 0.1, the total loss is {loss_2:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What do you notice?\n",
    "Since you used a weighted loss, the calculated loss is the same whether the model always predicts 0.9 or always predicts 0.1.  \n",
    "\n",
    "You may have also noticed that when you calculate each term of the weighted loss separately, there is a bit of symmetry when comparing between the two sets of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"loss_1_pos: {loss_1_pos:.4f} \\t loss_1_neg: {loss_1_neg:.4f}\")\n",
    "print()\n",
    "print(f\"loss_2_pos: {loss_2_pos:.4f} \\t loss_2_neg: {loss_2_neg:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though there is a class imbalance, where there are 3 positive labels but only one negative label, the weighted loss accounts for this by giving more weight to the negative label than to the positive label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Weighted Loss for More than One Class\n",
    "\n",
    "In this week's assignment, you will calculate the multi-class weighted loss (when there is more than one disease class that your model is learning to predict).  Here, you can practice working with 2D numpy arrays, which will help you implement the multi-class weighted loss in the graded assignment.\n",
    "\n",
    "You will work with a dataset that has two disease classes (two columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the labels (true values) that you will practice with\n",
    "y_true = np.array(\n",
    "        [[1,0],\n",
    "         [1,0],\n",
    "         [1,0],\n",
    "         [1,0],\n",
    "         [0,1]\n",
    "        ])\n",
    "y_true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Choosing axis=0 or axis=1\n",
    "\n",
    "You will use `numpy.sum` to count the number of times column `0` has the value 0.  \n",
    "First, notice the difference when you set axis=0 versus axis=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See what happens when you set axis=0\n",
    "print(f\"using axis = 0 {np.sum(y_true,axis=0)}\")\n",
    "\n",
    "# Compare this to what happens when you set axis=1\n",
    "print(f\"using axis = 1 {np.sum(y_true,axis=1)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that if you choose `axis=0`, the sum is taken for each of the two columns.  This is what you want to do in this case. If you set `axis=1`, the sum is taken for each row."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Calculate the Weights\n",
    "\n",
    "Previously, you visually inspected the data to calculate the fraction of negative and positive labels.  Here, you can do this programmatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the positive weights as the fraction of negative labels (0) for each class (each column)\n",
    "w_p = np.sum(y_true == 0,axis=0) / y_true.shape[0]\n",
    "w_p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the negative weights as the fraction of positive labels (1) for each class\n",
    "w_n = np.sum(y_true == 1, axis=0) / y_true.shape[0]\n",
    "w_n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the assignment, you will train a model to try and make useful predictions.  In order to make this example easier to follow, you will pretend that your model always predicts the same value for every example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set model predictions where all predictions are the same\n",
    "y_pred = np.ones(y_true.shape)\n",
    "y_pred[:,0] = 0.3 * y_pred[:,0]\n",
    "y_pred[:,1] = 0.7 * y_pred[:,1]\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, calculate the two terms that make up the loss function.  Notice that you are working with more than one class (represented by columns).  In this case, there are two classes.\n",
    "\n",
    "Start by calculating the loss for class `0`.\n",
    "\n",
    "$$ loss^{(i)} = loss_{pos}^{(i)} + los_{neg}^{(i)} $$\n",
    "\n",
    "$$loss_{pos}^{(i)} = -1 \\times weight_{pos}^{(i)} \\times y^{(i)} \\times log(\\hat{y}^{(i)})$$\n",
    "\n",
    "$$loss_{neg}^{(i)} = -1 \\times weight_{neg}^{(i)} \\times (1- y^{(i)}) \\times log(1 - \\hat{y}^{(i)})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the zero column for the weights, true values, and predictions that you will use to calculate the loss from the positive predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print and view column zero of the weight\n",
    "print(f\"w_p[0]: {w_p[0]}\")\n",
    "print(f\"y_true[:,0]: {y_true[:,0]}\")\n",
    "print(f\"y_pred[:,0]: {y_pred[:,0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the loss from the positive predictions, for class 0\n",
    "loss_0_pos = -1 * np.sum(w_p[0] * \n",
    "                y_true[:, 0] * \n",
    "                np.log(y_pred[:, 0])\n",
    "              )\n",
    "print(f\"loss_0_pos: {loss_0_pos:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the zero column for the weights, true values, and predictions that you will use to calculate the loss from the negative predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print and view column zero of the weight\n",
    "print(f\"w_n[0]: {w_n[0]}\")\n",
    "print(f\"y_true[:,0]: {y_true[:,0]}\")\n",
    "print(f\"y_pred[:,0]: {y_pred[:,0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the loss from the negative predictions, for class 0\n",
    "loss_0_neg = -1 * np.sum( \n",
    "                w_n[0] * \n",
    "                (1 - y_true[:, 0]) * \n",
    "                np.log(1 - y_pred[:, 0])\n",
    "              )\n",
    "print(f\"loss_0_neg: {loss_0_neg:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the two loss terms to get the total loss for class 0\n",
    "loss_0 = loss_0_neg + loss_0_pos\n",
    "print(f\"loss_0: {loss_0:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you are familiar with the array slicing that you would use when there are multiple disease classes stored in a two-dimensional array.\n",
    "\n",
    "#### Now it's your turn!\n",
    "* Can you calculate the loss for class (column) `1`?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the loss from the positive predictions, for class 1\n",
    "loss_1_pos = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output\n",
    "```CPP\n",
    "loss_1_pos: 0.2853\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the loss from the negative predictions, for class 1\n",
    "loss_1_neg = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```CPP\n",
    "loss_1_neg: 0.9632\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the two loss terms to get the total loss for class 1\n",
    "loss_1 = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected output\n",
    "```CPP\n",
    "loss_1: 1.2485\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note\n",
    "The data for the two classes (two columns) as well as the predictions were chosen so that you end up getting the same weighted loss for both categories.  \n",
    " - In general, you will expect to calculate different weighted loss values for each disease category, as the model predictions and data will differ from one category to another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want some help, please click on the green \"Solution\" cell below to reveal the solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Solution</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<code>\n",
    "-- # calculate the loss from the positive predictions, for class 1\n",
    "loss_1_pos = -1 * np.sum(w_p[1] * \n",
    "                y_true[:, 1] * \n",
    "                np.log(y_pred[:, 1])\n",
    "              )\n",
    "print(f\"loss_1_pos: {loss_1_pos:.4f}\")\n",
    "    \n",
    "-- # Calculate the loss from the negative predictions, for class 1\n",
    "loss_1_neg = -1 * np.sum( \n",
    "                w_n[1] * \n",
    "                (1 - y_true[:, 1]) * \n",
    "                np.log(1 - y_pred[:, 1])\n",
    "              )\n",
    "print(f\"loss_1_neg: {loss_1_neg:.4f}\")\n",
    "\n",
    "-- # add the two loss terms to get the total loss for class 1\n",
    "loss_1 = loss_1_neg + loss_1_pos\n",
    "print(f\"loss_1: {loss_1:.4f}\")\n",
    "    </code>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How this practice relates to and differs from the upcoming graded assignment\n",
    "- In the assignment, you will generalize this to calculating the loss for any number of classes.\n",
    "- Also in the assignment, you will learn how to avoid taking the log of zero by adding a small number (more details will be explained in the assignment).\n",
    "- Note that in the lecture videos and in this lecture notebook, you are taking the **sum** of losses for all examples.  In the assignment, you will take the **average (the mean)** for all examples.\n",
    "- Finally, in the assignment, you will work with \"tensors\" in TensorFlow, so you will use the TensorFlow equivalents of the numpy operations (keras.mean instead of numpy.mean)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### That's all for this lab. You now have a couple more tools you'll need for this week's assignment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}