labs/lab03/lab.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"lab.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3 – Merging and Pivoting\n",
    "\n",
    "## DSC 80, Winter 2024\n",
    "\n",
    "### Due Date: Monday, January 29th at 11:59PM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "Welcome to the third DSC 80 lab this quarter!\n",
    "\n",
    "Much like in DSC 10, this Jupyter Notebook contains the statements of the problems and provides code and Markdown cells to display your answers to the problems. Unlike DSC 10, the notebook is *only* for displaying a readable version of your final answers. The coding will be done in an accompanying `lab.py` file that is imported into the current notebook, and **you will only submit that `lab.py` file**, not this notebook!\n",
    "\n",
    "Some additional guidelines:\n",
    "- **Unlike in DSC 10, labs will have both public tests and hidden tests.** The bulk of your grade will come from your scores on hidden tests, which you will only see on Gradescope after the assignment deadline.\n",
    "- **Do not change the function names in the `lab.py` file!** The functions in the `lab.py` file are how your assignment is graded, and they are graded by their name. If you changed something you weren't supposed to, you can find the original code in the [course GitHub repository](https://github.com/dsc-courses/dsc80-2024-wi).\n",
    "- Notebooks are nice for testing and experimenting with different implementations before designing your function in your `lab.py` file. You can write code here, but make sure that all of your real work is in the `lab.py` file, since that's all you're submitting.\n",
    "- You are encouraged to write your own additional helper functions to solve the lab, as long as they also end up in `lab.py`.\n",
    "\n",
    "**To ensure that all of the work you want to submit is in `lab.py`, we've included a script named `lab-validation.py` in the lab folder. You shouldn't edit it, but instead, you should call it from the command line (e.g. the Terminal) to test your work.** More details on its usage are given at the bottom of this notebook.\n",
    "\n",
    "**Importing code from `lab.py`**:\n",
    "\n",
    "* Below, we import the `.py` file that's contained in the same directory as this notebook.\n",
    "* We use the `autoreload` notebook extension to make changes to our `lab.py` file immediately available in our notebook. Without this extension, we would need to restart the notebook kernel to see any changes to `lab.py` in the notebook.\n",
    "    - `autoreload` is necessary because, upon import, `lab.py` is compiled to bytecode (in the directory `__pycache__`). Subsequent imports of `lab` merely import the existing compiled python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lab import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\" markdown=\"1\">\n",
    "\n",
    "**There are only two functions in this lab in which you may use a `for`-loop:**\n",
    "- **`read_linkedin_survey` in Question 1.**\n",
    "- **`read_student_surveys` in Question 2.**\n",
    "    \n",
    "**You may lose points if you use a `for`-loop or `while`-loop in any other question!**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Combining Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 – Making Connections 🤝\n",
    "\n",
    "A group of students decided to send out a survey to their connections on LinkedIn. Each student asks 1000 of their connections for their first and last name, the company they currently work at, their job title, their email, and the university they attended.\n",
    "\n",
    "**Your job is to combine all the data contained in the files `survey*.csv` (stored within the `data/responses` folder) into a single DataFrame. The number of files and the number of rows in each file may vary, so don't hardcode your answers!** To do so, implement the following two functions.\n",
    "\n",
    "#### `read_linkedin_survey`\n",
    "\n",
    "Complete the implementation of the function `read_linkedin_survey`, which takes in a string for the directory where `survey*.csv` files are located and outputs a single DataFrame with six columns titled `'first name'`, `'last name'`, `'current company'`, `'job title'`, `'email'`, and `'university'` (in that order) containing the all of survey information from all the `.csv` files. Make sure to reset the index of the combined DataFrame before returning it so that the index is unique. \n",
    "\n",
    "***Hints***:\n",
    "\n",
    "- Take a look at a few of the files in the `responses` folder. You may have to do some data cleaning to combine the DataFrames!\n",
    "\n",
    "- You can list the files in a directory using `.iterdir()`.\n",
    "\n",
    "***Note***: Remember to use `Path()` to build file paths.\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `com_stats`\n",
    "\n",
    "Complete the implementation of the function `com_stats`, which takes in a DataFrame returned by `read_linkedin_survey` and returns a list containing, in the following order: \n",
    "- The proportion of people who went to a university with the string `'Ohio'` in its name that have the string `'Programmer'` somewhere in their job title.\n",
    "- The number of job titles that **end** with the exact string `'Engineer'`. Note that we're asking for the number of job titles, **not** the number of people!\n",
    "- The job title that has the longest name (there are no ties).\n",
    "- The number of people who have the word `'manager'` in their job title, uppercase or lowercase (`'Manager'`, `'manager'`, and `'mANAgeR'` should all count)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do not edit this cell -- it is needed for the tests\n",
    "dirname = Path('data') / 'responses'\n",
    "q1_out = read_linkedin_survey(dirname)\n",
    "stats_out = com_stats(q1_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 – Survey Says... 👨‍👩‍👧‍👦\n",
    "\n",
    "Professor Aritra often sends out extra credit surveys asking students for their favorite animals, movies, and other favorite things. These surveys are stored in the `data/extra-credit-surveys` folder. Each file in that folder corresponds to a different survey question (except for `favorite1.csv`, which contains students' names and IDs).\n",
    "\n",
    "Here's how extra credit works:\n",
    "- Each student who has completed at least 50% of the survey questions receives 5 points of extra credit.\n",
    "- If there is at least one survey question that at least 90% of the class answered (e.g. favorite animal), **everyone** in the class receives 1 point of extra credit. This overall class extra credit only applies twice, so if for example 95% of students answer the favorite color survey question and 91% answer the favorite animal survey question, and and 97% answer the favorite movie question, the entire class still receives 2 extra point as a class, not 3.\n",
    "- Note that this means that the most extra credit any student can earn is 7 points.\n",
    "\n",
    "#### `read_student_surveys`\n",
    "\n",
    "Complete the implementation of the function `read_student_surveys` which takes in a string describing the path to a folder containing `favorite*.csv` files and outputs a DataFrame containing all of the survey data combined, indexed by student ID (a value 1-1000).\n",
    "\n",
    "***Note***: Remember to use `Path()` to build file paths.\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `check_credit`\n",
    "\n",
    "Complete the implementation of the function `check_credit` which takes in a DataFrame returned by `read_student_surveys` and outputs a DataFrame indexed by student ID (a value 1-1000) with two columns:\n",
    "- `'name'`, containing the name of each student, and\n",
    "- `'ec'`, containing the number of extra credit points each student earned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do not edit this cell -- it is needed for the tests\n",
    "dirname = Path('data') / 'extra-credit-surveys'\n",
    "q2_out = read_student_surveys(dirname)\n",
    "check_credit_out = check_credit(q2_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 – Paw Patrol 🐾\n",
    "\n",
    "You are analyzing data from a veterinarian clinic. The datasets contain several types of information from the clinic, including its customers (pet owners), pets, available procedures, and procedure history. The column names are self-explanatory. These DataFrames are provided to you:\n",
    "-  `owners` stores the customer information, where every `'OwnerID'` is unique (verify this yourself).\n",
    "-  `pets` stores the pet information. Each pet belongs to a customer in `owners`.\n",
    "-  `procedure_detail` contains a catalog of procedures that are offered by the clinic.\n",
    "-  `procedure_history` has procedure records. Most procedures were given to a pet in `pets`.\n",
    "\n",
    "Complete the implementation of the following three functions, which each ask you to answer a specific question.\n",
    "\n",
    "#### `most_popular_procedure`\n",
    "\n",
    "What is the most popular `'ProcedureType'` amongst all pets in the `pets` DataFrame? Complete the implementation of the function `most_popular_procedure`, which takes in two DataFrames, `pets` and `procedure_history`, and returns the name of the most popular `'ProcedureType'` as a string.\n",
    "\n",
    "Note that some pets are registered but haven't had any procedures performed. Also, some pets that have had procedures done are not registered in `pets`.\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `pet_name_by_owner`\n",
    "\n",
    "What is the name of each customer's pet(s)? Complete the implementation of the function `pet_name_by_owner`, which takes in two DataFrames, `owners` and `pets`, and returns a Series whose index contains owner first names, and whose values are pet names as **strings**. If an owner has multiple pets, the value corresponding to that owner should instead be a **list of pet names as strings**.\n",
    "\n",
    "Note that owner first names are not necessarily unique, and so the Series you return will not necessarily have a unique index.\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `total_cost_per_city`\n",
    "\n",
    "Note that the `owners` DataFrame has a `'City'` column, describing the city in which each pet owner and their pets live. How much did each city spend in total on procedures? Complete the implementation of the function `total_cost_per_city`, which takes in four DataFrames, `owners`, `pets`, `procedure_history`, and `procedure_detail`, and returns a Series indexed by `'City'` that describes the total amount that each city has spent on pets' procedures.\n",
    "\n",
    "***Hint***: At some point, you may have to merge on multiple columns.\n",
    "\n",
    "***Note***: Some owners may have never visited the veterinarian clinic in their city. This means some cities may have zero operational costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do not edit this cell -- it is needed for the tests\n",
    "pets_fp = Path('data') / 'pets' / 'Pets.csv'\n",
    "procedure_history_fp =  Path('data') / 'pets' / 'ProceduresHistory.csv'\n",
    "owners_fp = Path('data') / 'pets' / 'Owners.csv'\n",
    "procedure_detail_fp = Path('data') / 'pets' / 'ProceduresDetails.csv'\n",
    "pets = pd.read_csv(pets_fp)\n",
    "procedure_history = pd.read_csv(procedure_history_fp)\n",
    "owners = pd.read_csv(owners_fp)\n",
    "procedure_detail = pd.read_csv(procedure_detail_fp)\n",
    "\n",
    "out_01 = most_popular_procedure(pets, procedure_history)\n",
    "out_02 = pet_name_by_owner(owners, pets)\n",
    "out_03 = total_cost_per_city(owners, pets, procedure_history, procedure_detail)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Pivot Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 – Summarizing Sales 💰\n",
    "\n",
    "Recall from [Lecture 3](https://dsc80.com/resources/lectures/lec03/lec03-filled.html#Pivot-tables-using-the-pivot_table-method), a pivot table allows you to aggregate the entries in a DataFrame based on two categorical columns. In this question, you are given a simple dataset, `data/sales.csv`, and are asked to solve a few problems using the `pivot_table` method. \n",
    "\n",
    "**We have provided outlines for the DataFrames you need to create, but yours may have a different number of rows and columns and different values.**\n",
    "\n",
    "***Note***: If it helps, [here](https://jakevdp.github.io/PythonDataScienceHandbook/03.09-pivot-tables.html) is another great resource that provides an overview of `pivot_table` with many examples from the Titanic dataset.\n",
    "\n",
    "\n",
    "#### `average_seller`\n",
    "\n",
    "Complete the implementation of the function `average_seller`, which takes in the `sales` DataFrame and returns a DataFrame that contains the average sales for each seller, indexed by `'Name'` and containing the column `'Average Sales'`. There should not be any `NaN`s.\n",
    "\n",
    "***Note***: You may be able to implement `average_seller` without using `pivot_table`.\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th></th>\n",
    "      <th>Average Sales</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Name</th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>Jones</th>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Smith</th>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Trump</th>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `product_name`\n",
    "\n",
    "Complete the implementation of the function `product_name` that takes in the `sales` DataFrame and returns a DataFrame that contains the total sales for each product, indexed by `'Name'`. Do not fill in `NaN`s.\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th>Product</th>\n",
    "      <th>boat</th>\n",
    "      <th>book</th>\n",
    "      <th>hotel</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Name</th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>Jones</th>\n",
    "      <td>NaN</td>\n",
    "      <td>0.0</td>\n",
    "      <td>NaN</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Smith</th>\n",
    "      <td>NaN</td>\n",
    "      <td>0.0</td>\n",
    "      <td>NaN</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Trump</th>\n",
    "      <td>0.0</td>\n",
    "      <td>0.0</td>\n",
    "      <td>0.0</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `count_product`\n",
    "\n",
    "Complete the implementation of the function `count_product` that takes in the `sales` DataFrame and returns a DataFrame that contains the total number of items sold product-wise and name-wise per date. Replace `NaN`s with 0s. Don't reset the index after pivoting.\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th></th>\n",
    "      <th>Date</th>\n",
    "      <th>01.01.2012</th>\n",
    "      <th>02.20.2013</th>\n",
    "      <th>02.25.2015</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Product</th>\n",
    "      <th>Name</th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>boat</th>\n",
    "      <th>Trump</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th rowspan=\"3\" valign=\"top\">book</th>\n",
    "      <th>Jones</th>\n",
    "      <td>0</td>\n",
    "      <td>1</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Smith</th>\n",
    "      <td>1</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Trump</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>hotel</th>\n",
    "      <th>Trump</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `total_by_month`\n",
    "\n",
    "Complete the implementation of the function `total_by_month` that takes in the `sales` DataFrame and returns a pivot table that contains the total sales name-wise, product-wise per month. Replace `NaN`s with 0s. Don't reset the index after pivoting.\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th></th>\n",
    "      <th>Month</th>\n",
    "      <th>February</th>\n",
    "      <th>January</th>\n",
    "      <th>July</th>\n",
    "      <th>March</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Name</th>\n",
    "      <th>Product</th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th rowspan=\"3\" valign=\"top\">Jones</th>\n",
    "      <th>book</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>pen</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>ruler</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th rowspan=\"3\" valign=\"top\">Smith</th>\n",
    "      <th>book</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>pen</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>ruler</th>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell -- it is needed for the tests to work\n",
    "fp = Path('data') / 'sales.csv'\n",
    "sales = pd.read_csv(fp)\n",
    "q4_average_seller_out = average_seller(sales)\n",
    "q4_product_name_out = product_name(sales)\n",
    "q4_product_count_out = count_product(sales)\n",
    "q4_total_by_month_out = total_by_month(sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You're done Lab 3! 🏁\n",
    "\n",
    "As a reminder, all of the work you want to submit needs to be in `lab.py`.\n",
    "\n",
    "To ensure that all of the work you want to submit is in `lab.py`, we've included a script named `lab-validation.py` in the lab folder. You shouldn't edit it, but instead, you should call it from the command line (e.g. the Terminal) to test your work.\n",
    "\n",
    "Once you've finished the lab, you should open the command line and run, in the directory for this lab:\n",
    "\n",
    "```\n",
    "python lab-validation.py\n",
    "```\n",
    "\n",
    "**This will run all of the `grader.check` cells that you see in this notebook, but only using the code in `lab.py` – that is, it doesn't look at any of the code in this notebook. If all of your `grader.check` cells pass in this notebook but not all of them pass in your command line with the above command, then you likely have code in your notebook that isn't in your `lab.py`!**\n",
    "\n",
    "You can also use `lab-validation.py` to test individual questions. For instance,\n",
    "\n",
    "```\n",
    "python lab-validation.py q1 q2 q4\n",
    "```\n",
    "\n",
    "will run the `grader.check` cells for Questions 1, 2, and 4 – again, only using the code in `lab.py`. [This video](https://www.loom.com/share/0ea254b85b2745e59322b5e5a8692e91?sid=5acc92e6-0dfe-4555-9b6a-8115b6a52f99) how to use the script as well.\n",
    "\n",
    "Once `python lab-validation.py` shows that you're passing all test cases, you're ready to submit your `lab.py` (and only your `lab.py`) to Gradescope. Once submitting to Gradescope, make sure to stick around until all test cases pass.\n",
    "\n",
    "There is also a call to `grader.check_all()` below in _this_ notebook, but make sure to also follow the steps above.\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\" markdown=\"1\">\n",
    "\n",
    "**There are only two functions in this lab in which you may use a `for`-loop:**\n",
    "- **`read_linkedin_survey` in Question 1.**\n",
    "- **`read_student_surveys` in Question 2.**\n",
    "    \n",
    "**You may lose points if you use a `for`-loop or `while`-loop in any other question!**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check_all()"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "nteract": {
   "version": "0.15.0"
  },
  "otter": {
   "tests": {
    "q1": {
     "name": "q1",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> isinstance(q1_out, pd.DataFrame) == True\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> len(q1_out) == 5000\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> try:\n...     read_linkedin_survey('nonexistentfile')\n... except FileNotFoundError:\n...     print(True)\nTrue\n",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> len(stats_out) == 4\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> isinstance(stats_out[0], float)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> isinstance(stats_out[1], int)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> isinstance(stats_out[2], str)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q2": {
     "name": "q2",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> isinstance(q2_out, pd.DataFrame)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> q2_out.shape == (1000, 6)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> try:\n...     read_student_surveys('nonexistentfile')\n... except FileNotFoundError:\n...     print(True)\nTrue\n",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> check_credit_out.shape == (1000, 2)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> check_credit_out['ec'].max() == 6\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q3": {
     "name": "q3",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> isinstance(out_01, str)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> len(out_02) == len(owners)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> 'Sarah' in out_02.index\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> 'Cookie' in out_02.values\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> set(out_03.index) <= set(owners['City'])\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q4": {
     "name": "q4",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> q4_average_seller_out.shape[0] == 3\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q4_average_seller_out[\"Average Sales\"].sum() < 5000\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q4_product_name_out.size == 15\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q4_product_name_out[\"pen\"].isnull().sum() == 0\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q4_product_count_out.size == 70\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q4_total_by_month_out.shape[1] == 5\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> int(q4_product_name_out['hotel'].isnull().sum()) == 2\nTrue",
         "failure_message": "null values: hotel row",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> q4_product_count_out.shape == (10, 7)\nTrue",
         "failure_message": "correct shape",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> (q4_total_by_month_out > 2000).sum().sum() == 3\nTrue",
         "failure_message": "number of entries strictly over 2000",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> (q4_total_by_month_out == 0).sum().sum() == 39\nTrue",
         "failure_message": "number of zero entries",
         "hidden": false,
         "locked": false,
         "points": 0.5
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}