01 - Data Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Data with Python\n",
    "\n",
    "A significant part of a data scientist's role is to explore, analyze, and visualize data. There's a wide range of tools and programming languages that they can use to do this, and of the most popular approaches is to use Jupyter notebooks (like this one) and Python.\n",
    "\n",
    "Python is a flexible programming language that is used in a wide range of scenarios; from web applications to device programming. It's extremely popular in the data science and machine learning community because of the many packages it supports for data analysis and visualization.\n",
    "\n",
    "In this notebook, we'll explore some of these packages, and apply basic techniques to analyze data. This is not intended to be a comprehensive Python programming exercise; or even a deep dive into data analysis. Rather, it's intended as a crash course in some of the common ways in which data scientists can use Python to work with data.\n",
    "\n",
    "> **Note**: If you've never used the Jupyter Notebooks environment before, there are a few things you should be aware of:\n",
    "> \n",
    "> - Notebooks are made up of *cells*. Some cells (like this one) contain *markdown* text, while others (like the one beneath this one) contain code.\n",
    "> - The notebook is connected to a Python *kernel* (you can see which one at the top right of the page - if you're running this notebook in an Azure Machine Learning compute instance it should be connected to the **Python 3.6 - AzureML** kernel). If you stop the kernel or disconnect from the server (for example, by closing and reopening the notebook, or ending and resuming your session), the output from cells that have been run will still be displayed; but any variables or functions defined in those cells will have been lost - you must rerun the cells before running any subsequent cells that depend on them.\n",
    "> - You can run each code cell by using the **&#9658; Run** button. The **&#9711;** symbol next to the kernel name at the top right will briefly turn to **&#9899;** while the cell runs before turning back to **&#9711;**.\n",
    "> - The output from each code cell will be displayed immediately below the cell.\n",
    "> - Even though the code cells can be run individually, some variables used in the code are global to the notebook. That means that you should run all of the code cells <u>**in order**</u>. There may be dependencies between code cells, so if you skip a cell, subsequent cells might not run correctly.\n",
    "\n",
    "\n",
    "## Exploring data arrays with NumPy\n",
    "\n",
    "Let's start by looking at some simple data.\n",
    "\n",
    "Suppose a college takes a sample of student grades for a data science class.\n",
    "\n",
    "Run the code in the cell below by clicking the **&#9658; Run** button to see the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = [50,50,47,97,49,3,53,42,26,74,82,62,37,15,70,27,36,35,48,52,63,64]\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data has been loaded into a Python **list** structure, which is a good data type for general data manipulation, but not optimized for numeric analysis. For that, we're going to use the **NumPy** package, which includes specific data types and functions for working with *Num*bers in *Py*thon.\n",
    "\n",
    "Run the cell below to load the data into a NumPy **array**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "grades = np.array(data)\n",
    "print(grades)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just in case you're wondering about the differences between a **list** and a NumPy **array**, let's compare how these data types behave when we use them in an expression that multiplies them by 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print (type(data),'x 2:', data * 2)\n",
    "print('---')\n",
    "print (type(grades),'x 2:', grades * 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that multiplying a list by 2 creates a new list of twice the length with the original sequence of list elements repeated. Multiplying a NumPy array on the other hand performs an element-wise calculation in which the array behaves like a *vector*, so we end up with an array of the same size in which each element has been multiplied by 2.\n",
    "\n",
    "The key takeaway from this is that NumPy arrays are specifically designed to support mathematical operations on numeric data - which makes them more useful for data analysis than a generic list.\n",
    "\n",
    "You might have spotted that the class type for the numpy array above is a **numpy.ndarray**. The **nd** indicates that this is a structure that can consists of multiple *dimensions* (it can have *n* dimensions). Our specific instance has a single dimension of student grades.\n",
    "\n",
    "Run the cell below to view the **shape** of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grades.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape confirms that this array has only one dimension, which contains 22 elements (there are 22 grades in the original list). You can access the individual elements in the array by their zero-based ordinal position. Let's get the first element (the one in position 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grades[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, now you know your way around a NumPy array, it's time to perform some analysis of the grades data.\n",
    "\n",
    "You can apply aggregations across the elements in the array, so let's find the simple average grade (in other words, the *mean* grade value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grades.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the mean grade is just around 50 - more or less in the middle of the possible range from 0 to 100.\n",
    "\n",
    "Let's add a second set of data for the same students, this time recording the typical number of hours per week they devoted to studying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define an array of study hours\n",
    "study_hours = [10.0,11.5,9.0,16.0,9.25,1.0,11.5,9.0,8.5,14.5,15.5,\n",
    "               13.75,9.0,8.0,15.5,8.0,9.0,6.0,10.0,12.0,12.5,12.0]\n",
    "\n",
    "# Create a 2D array (an array of arrays)\n",
    "student_data = np.array([study_hours, grades])\n",
    "\n",
    "# display the array\n",
    "student_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the data consists of a 2-dimensional array - an array of arrays. Let's look at its shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show shape of 2D array\n",
    "student_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **student_data** array contains two elements, each of which is an array containing 22 elements.\n",
    "\n",
    "To navigate this structure, you need to specify the position of each element in the hierarchy. So to find the first value in the first array (which contains the study hours data), you can use the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the first element of the first element\n",
    "student_data[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a multidimensional array containing both the student's study time and grade information, which you can use to compare data. For example, how does the mean study time compare to the mean grade?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the mean value of each sub-array\n",
    "avg_study = student_data[0].mean()\n",
    "avg_grade = student_data[1].mean()\n",
    "\n",
    "print('Average study hours: {:.2f}\\nAverage grade: {:.2f}'.format(avg_study, avg_grade))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring tabular data with Pandas\n",
    "\n",
    "While NumPy provides a lot of the functionality you need to work with numbers, and specifically arrays of numeric values; when you start to deal with two-dimensional tables of data, the **Pandas** package offers a more convenient structure to work with - the **DataFrame**.\n",
    "\n",
    "Run the following cell to import the Pandas library and create a DataFrame with three columns. The first column is a list of student names, and the second and third columns are the NumPy arrays containing the study time and grade data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df_students = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', \n",
    "                                     'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny',\n",
    "                                     'Jakeem','Helena','Ismat','Anila','Skye','Daniel','Aisha'],\n",
    "                            'StudyHours':student_data[0],\n",
    "                            'Grade':student_data[1]})\n",
    "\n",
    "df_students "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in addition to the columns you specified, the DataFrame includes an *index* to unique identify each row. We could have specified the index explicitly, and assigned any kind of appropriate value (for example, an email address); but because we didn't specify an index, one has been created with a unique integer value for each row.\n",
    "\n",
    "### Finding and filtering data in a DataFrame\n",
    "\n",
    "You can use the DataFrame's **loc** method to retrieve data for a specific index value, like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the data for index value 5\n",
    "df_students.loc[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get the data at a range of index values, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the rows with index values from 0 to 5\n",
    "df_students.loc[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to being able to use the **loc** method to find rows based on the index, you can use the **iloc** method to find rows based on their ordinal position in the DataFrame (regardless of the index):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get data in the first five rows\n",
    "df_students.iloc[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look carefully at the `iloc[0:5]` results, and compare them to the `loc[0:5]` results you obtained previously. Can you spot the difference?\n",
    "\n",
    "The **loc** method returned rows with index *label* in the list of values from *0* to *5* - which includes *0*, *1*, *2*, *3*, *4*, and *5* (six rows). However, the **iloc** method returns the rows in the *positions* included in the range 0 to 5, and since integer ranges don't include the upper-bound value, this includes positions *0*, *1*, *2*, *3*, and *4* (five rows).\n",
    "\n",
    "**iloc** identifies data values in a DataFrame by *position*, which extends beyond rows to columns. So for example, you can use it to find the values for the columns in positions 1 and 2 in row 0, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.iloc[0,[1,2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's return to the **loc** method, and see how it works with columns. Remember that **loc** is used to locate data items based on index values rather than positions. In the absence of an explicit index column, the rows in our dataframe are indexed as integer values, but the columns are identified by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.loc[0,'Grade']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's another useful trick. You can use the **loc** method to find indexed rows based on a filtering expression that references named columns other than the index, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.loc[df_students['Name']=='Aisha']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, you don't need to explicitly use the **loc** method to do this - you can simply apply a DataFrame filtering expression, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students[df_students['Name']=='Aisha']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for good measure, you can achieve the same results by using the DataFrame's **query** method, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.query('Name==\"Aisha\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three previous examples underline an occassionally confusing truth about working with Pandas. Often, there are multiple ways to achieve the same results. Another example of this is the way you refer to a DataFrame column name. You can specify the column name as a named index value (as in the `df_students['Name']` examples we've seen so far), or you can use the column as a property of the DataFrame, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students[df_students.Name == 'Aisha']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading a DataFrame from a  file\n",
    "\n",
    "We constructed the DataFrame from some existing arrays. However, in many real-world scenarios, data is loaded from sources such as files. Let's replace the student grades DataFrame with the contents of a text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students = pd.read_csv('data/grades.csv',delimiter=',',header='infer')\n",
    "df_students.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DataFrame's **read_csv** method is used to load data from text files. As you can see in the example code, you can specify options such as the column delimiter and which row (if any) contains column headers (in this case, the delimiter is a comma and the first row contains the column names - these are the default settings, so the parameters could have been omitted).\n",
    "\n",
    "\n",
    "### Handling missing values\n",
    "\n",
    "One of the most common issues data scientists need to deal with is incomplete or missing data. So how would we know that the DataFrame contains missing values? You can use the **isnull** method to identify which individual values are null, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, with a larger DataFrame, it would be inefficient to review all of the rows and columns individually; so we can get the sum of missing values for each column, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we know that there's one missing **StudyHours** value, and two missing **Grade** values.\n",
    "\n",
    "To see them in context, we can filter the dataframe to include only rows where any of the columns (axis 1 of the DataFrame) are null."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students[df_students.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the DataFrame is retrieved, the missing numeric values show up as **NaN** (*not a number*).\n",
    "\n",
    "So now that we've found the null values, what can we do about them?\n",
    "\n",
    "One common approach is to *impute* replacement values. For example, if the number of study hours is missing, we could just assume that the student studied for an average amount of time and replace the missing value with the mean study hours. To do this, we can use the **fillna** method, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.StudyHours = df_students.StudyHours.fillna(df_students.StudyHours.mean())\n",
    "df_students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, it might be important to ensure that you only use data you know to be absolutely correct; so you can drop rows or columns that contains null values by using the **dropna** method. In this case, we'll remove rows (axis 0 of the DataFrame) where any of the columns contain null values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students = df_students.dropna(axis=0, how='any')\n",
    "df_students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore data in the DataFrame\n",
    "\n",
    "Now that we've cleaned up the missing values, we're ready to explore the data in the DataFrame. Let's start by comparing the mean study hours and grades."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the mean study hours using the column name as an index\n",
    "mean_study = df_students['StudyHours'].mean()\n",
    "\n",
    "# Get the mean grade using the column name as a property (just to make the point!)\n",
    "mean_grade = df_students.Grade.mean()\n",
    "\n",
    "# Print the mean study hours and mean grade\n",
    "print('Average weekly study hours: {:.2f}\\nAverage grade: {:.2f}'.format(mean_study, mean_grade))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, let's filter the DataFrame to find only the students who studied for more than the average amount of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get students who studied for the mean or more hours\n",
    "df_students[df_students.StudyHours > mean_study]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the filtered result is itself a DataFrame, so you can work with its columns just like any other DataFrame.\n",
    "\n",
    "For example, let's find the average grade for students who undertook more than the average amount of study time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What was their mean grade?\n",
    "df_students[df_students.StudyHours > mean_study].Grade.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assume that the passing grade for the course is 60.\n",
    "\n",
    "We can use that information to add a new column to the DataFrame, indicating whether or not each student passed.\n",
    "\n",
    "First, we'll create a Pandas **Series** containing the pass/fail indicator (True or False), and then we'll concatenate that series as a new column (axis 1) in the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passes  = pd.Series(df_students['Grade'] >= 60)\n",
    "df_students = pd.concat([df_students, passes.rename(\"Pass\")], axis=1)\n",
    "\n",
    "df_students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrames are designed for tabular data, and you can use them to perform many of the kinds of data analytics operation you can do in a relational database; such as grouping and aggregating tables of data.\n",
    "\n",
    "For example, you can use the **groupby** method to group the student data into groups based on the **Pass** column you added previously, and count the number of names in each group - in other words, you can determine how many students passed and failed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df_students.groupby(df_students.Pass).Name.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can aggregate multiple fields in a group using any available aggregation function. For example, you can find the mean study time and grade for the groups of students who passed and failed the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df_students.groupby(df_students.Pass)['StudyHours', 'Grade'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrames are amazingly versatile, and make it easy to manipulate data. Many DataFrame operations return a new copy of the DataFrame; so if you want to modify a DataFrame but keep the existing variable, you need to assign the result of the operation to the existing variable. For example, the following code sorts the student data into descending order of Grade, and assigns the resulting sorted DataFrame to the original **df_students** variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame with the data sorted by Grade (descending)\n",
    "df_students = df_students.sort_values('Grade', ascending=False)\n",
    "\n",
    "# Show the DataFrame\n",
    "df_students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing data with Matplotlib\n",
    "\n",
    "DataFrames provide a great way to explore and analyze tabular data, but sometimes a picture is worth a thousand rows and columns. The **Matplotlib** library provides the foundation for plotting data visualizations that can greatly enhance your ability to analyze the data.\n",
    "\n",
    "Let's start with a simple bar chart that shows the grade of each student."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure plots are displayed inline in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Create a bar plot of name vs grade\n",
    "plt.bar(x=df_students.Name, height=df_students.Grade)\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that worked; but the chart could use some improvements to make it clearer what we're looking at.\n",
    "\n",
    "Note that you used the **pyplot** class from Matplotlib to plot the chart. This class provides a whole bunch of ways to improve the visual elements of the plot. For example, the following code:\n",
    "\n",
    "- Specifies the color of the bar chart.\n",
    "- Adds a title to the chart (so we know what it represents)\n",
    "- Adds labels to the X and Y (so we know which axis shows which data)\n",
    "- Adds a grid (to make it easier to determine the values for the bars)\n",
    "- Rotates the X markers (so we can read them)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a bar plot of name vs grade\n",
    "plt.bar(x=df_students.Name, height=df_students.Grade, color='orange')\n",
    "\n",
    "# Customize the chart\n",
    "plt.title('Student Grades')\n",
    "plt.xlabel('Student')\n",
    "plt.ylabel('Grade')\n",
    "plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A plot is technically contained with a **Figure**. In the previous examples, the figure was created implicitly for you; but you can create it explicitly. For example, the following code creates a figure with a specific size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Figure\n",
    "fig = plt.figure(figsize=(8,3))\n",
    "\n",
    "# Create a bar plot of name vs grade\n",
    "plt.bar(x=df_students.Name, height=df_students.Grade, color='orange')\n",
    "\n",
    "# Customize the chart\n",
    "plt.title('Student Grades')\n",
    "plt.xlabel('Student')\n",
    "plt.ylabel('Grade')\n",
    "plt.grid(color='#95a5a6', linestyle='--', linewidth=2, axis='y', alpha=0.7)\n",
    "plt.xticks(rotation=90)\n",
    "\n",
    "# Show the figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A figure can contain multiple subplots, each on its own *axis*.\n",
    "\n",
    "For example, the following code creates a figure with two subplots - one is a bar chart showing student grades, and the other is a pie chart comparing the number of passing grades to non-passing grades."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a figure for 2 subplots (1 row, 2 columns)\n",
    "fig, ax = plt.subplots(1, 2, figsize = (10,4))\n",
    "\n",
    "# Create a bar plot of name vs grade on the first axis\n",
    "ax[0].bar(x=df_students.Name, height=df_students.Grade, color='orange')\n",
    "ax[0].set_title('Grades')\n",
    "ax[0].set_xticklabels(df_students.Name, rotation=90)\n",
    "\n",
    "# Create a pie chart of pass counts on the second axis\n",
    "pass_counts = df_students['Pass'].value_counts()\n",
    "ax[1].pie(pass_counts, labels=pass_counts)\n",
    "ax[1].set_title('Passing Grades')\n",
    "ax[1].legend(pass_counts.keys().tolist())\n",
    "\n",
    "# Add a title to the Figure\n",
    "fig.suptitle('Student Data')\n",
    "\n",
    "# Show the figure\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, you've used methods of the Matplotlib.pyplot object to plot charts. However, Matplotlib is so foundational to graphics in Python that many packages, including Pandas, provide methods that abstract the underlying Matplotlib functions and simplify plotting. For example, the DataFrame provides its own methods for plotting data, as shown in the following example to plot a bar chart of study hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.plot.bar(x='Name', y='StudyHours', color='teal', figsize=(6,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started with statistical analysis\n",
    "\n",
    "Now that you know how to use Python to manipulate and visualize data, you can start analyzing it.\n",
    "\n",
    "A lot of data science is rooted in *statistics*, so we'll explore some basic statistical techniques.\n",
    "\n",
    "> **Note**: This is not intended to teach you statistics - that's much too big a topic for this notebook. It will however introduce you to some statistical concepts and techniques that data scientists use as they explore data in preparation for machine learning modeling.\n",
    "\n",
    "### Descriptive statistics and data distribution\n",
    "\n",
    "When examining a *variable* (for example a sample of student grades), data scientists are particularly interested in its *distribution* (in other words, how are all the different grade values spread across the sample). The starting point for this exploration is often to visualize the data as a histogram, and see how frequently each value for the variable occurs.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the variable to examine\n",
    "var_data = df_students['Grade']\n",
    "\n",
    "# Create a Figure\n",
    "fig = plt.figure(figsize=(10,4))\n",
    "\n",
    "# Plot a histogram\n",
    "plt.hist(var_data)\n",
    "\n",
    "# Add titles and labels\n",
    "plt.title('Data Distribution')\n",
    "plt.xlabel('Value')\n",
    "plt.ylabel('Frequency')\n",
    "\n",
    "# Show the figure\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The histogram for grades is a symmetric shape, where the most frequently occurring grades tend to be in the middle of the range (around 50), with fewer grades at the extreme ends of the scale.\n",
    "\n",
    "#### Measures of central tendency\n",
    "\n",
    "To understand the distribution better, we can examine so-called *measures of central tendency*; which is a fancy way of describing statistics that represent the \"middle\" of the data. The goal of this is to try to find a \"typical\" value. Common ways to define the middle of the data include:\n",
    "\n",
    "- The *mean*: A simple average based on adding together all of the values in the sample set, and then dividing the total by the number of samples.\n",
    "- The *median*: The value in the middle of the range of all of the sample values.\n",
    "- The *mode*: The most commonly occuring value in the sample set<sup>\\*</sup>.\n",
    "\n",
    "Let's calculate these values, along with the minimum and maximum values for comparison, and show them on the histogram.\n",
    "\n",
    "> <sup>\\*</sup>Of course, in some sample sets , there may be a tie for the most common value - in which case the dataset is described as *bimodal* or even *multimodal*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the variable to examine\n",
    "var = df_students['Grade']\n",
    "\n",
    "# Get statistics\n",
    "min_val = var.min()\n",
    "max_val = var.max()\n",
    "mean_val = var.mean()\n",
    "med_val = var.median()\n",
    "mod_val = var.mode()[0]\n",
    "\n",
    "print('Minimum:{:.2f}\\nMean:{:.2f}\\nMedian:{:.2f}\\nMode:{:.2f}\\nMaximum:{:.2f}\\n'.format(min_val,\n",
    "                                                                                        mean_val,\n",
    "                                                                                        med_val,\n",
    "                                                                                        mod_val,\n",
    "                                                                                        max_val))\n",
    "\n",
    "# Create a Figure\n",
    "fig = plt.figure(figsize=(10,4))\n",
    "\n",
    "# Plot a histogram\n",
    "plt.hist(var)\n",
    "\n",
    "# Add lines for the statistics\n",
    "plt.axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)\n",
    "plt.axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)\n",
    "plt.axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)\n",
    "plt.axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)\n",
    "plt.axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)\n",
    "\n",
    "# Add titles and labels\n",
    "plt.title('Data Distribution')\n",
    "plt.xlabel('Value')\n",
    "plt.ylabel('Frequency')\n",
    "\n",
    "# Show the figure\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the grade data, the mean, median, and mode all seem to be more or less in the middle of the minimum and maximum, at around 50.\n",
    "\n",
    "Another way to visualize the distribution of a variable is to use a *box* plot (sometimes called a *box-and-whiskers* plot). Let's create one for the grade data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the variable to examine\n",
    "var = df_students['Grade']\n",
    "\n",
    "# Create a Figure\n",
    "fig = plt.figure(figsize=(10,4))\n",
    "\n",
    "# Plot a histogram\n",
    "plt.boxplot(var)\n",
    "\n",
    "# Add titles and labels\n",
    "plt.title('Data Distribution')\n",
    "\n",
    "# Show the figure\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The box plot shows the distribution of the grade values in a different format to the histogram. The *box* part of the plot shows where the inner two *quartiles* of the data reside - so in this case, half of the grades are between approximately 36 and 63. The *whiskers* extending from the box show the outer two quartiles; so the other half of the grades in this case are between 0 and 36 or 63 and 100. The line in the box indicates the *median* value.\n",
    "\n",
    "It's often useful to combine histograms and box plots, with the box plot's orientation changed to align it with the histogram (in some ways, it can be helpful to think of the histogram as a \"front elevation\" view of the distribution, and the box plot as a \"plan\" view of the distribution from above.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a function that we can re-use\n",
    "def show_distribution(var_data):\n",
    "    from matplotlib import pyplot as plt\n",
    "\n",
    "    # Get statistics\n",
    "    min_val = var_data.min()\n",
    "    max_val = var_data.max()\n",
    "    mean_val = var_data.mean()\n",
    "    med_val = var_data.median()\n",
    "    mod_val = var_data.mode()[0]\n",
    "\n",
    "    print('Minimum:{:.2f}\\nMean:{:.2f}\\nMedian:{:.2f}\\nMode:{:.2f}\\nMaximum:{:.2f}\\n'.format(min_val,\n",
    "                                                                                            mean_val,\n",
    "                                                                                            med_val,\n",
    "                                                                                            mod_val,\n",
    "                                                                                            max_val))\n",
    "\n",
    "    # Create a figure for 2 subplots (2 rows, 1 column)\n",
    "    fig, ax = plt.subplots(2, 1, figsize = (10,4))\n",
    "\n",
    "    # Plot the histogram   \n",
    "    ax[0].hist(var_data)\n",
    "    ax[0].set_ylabel('Frequency')\n",
    "\n",
    "    # Add lines for the mean, median, and mode\n",
    "    ax[0].axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)\n",
    "    ax[0].axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)\n",
    "    ax[0].axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)\n",
    "    ax[0].axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)\n",
    "    ax[0].axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)\n",
    "\n",
    "    # Plot the boxplot   \n",
    "    ax[1].boxplot(var_data, vert=False)\n",
    "    ax[1].set_xlabel('Value')\n",
    "\n",
    "    # Add a title to the Figure\n",
    "    fig.suptitle('Data Distribution')\n",
    "\n",
    "    # Show the figure\n",
    "    fig.show()\n",
    "\n",
    "# Get the variable to examine\n",
    "col = df_students['Grade']\n",
    "# Call the function\n",
    "show_distribution(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the measurements of central tendency are right in the middle of the data distribution, which is symmetric with values becoming progressively lower in both directions from the middle.\n",
    "\n",
    "To explore this distribution in more detail, you need to understand that statistics is fundamentally about taking *samples* of data and using probability functions to extrapolate information about the full *population* of data. For example, the student data consists of 22 samples, and for each sample there is a grade value. You can think of each sample grade as a variable that's been randomly selected from the set of all grades awarded for this course. With enough of these random variables, you can calculate something called a *probability density function*, which estimates the distribution of grades for the full population.\n",
    "\n",
    "The Pandas DataFrame class provides a helpful plot function to show this density."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_density(var_data):\n",
    "    from matplotlib import pyplot as plt\n",
    "\n",
    "    fig = plt.figure(figsize=(10,4))\n",
    "\n",
    "    # Plot density\n",
    "    var_data.plot.density()\n",
    "\n",
    "    # Add titles and labels\n",
    "    plt.title('Data Density')\n",
    "\n",
    "    # Show the mean, median, and mode\n",
    "    plt.axvline(x=var_data.mean(), color = 'cyan', linestyle='dashed', linewidth = 2)\n",
    "    plt.axvline(x=var_data.median(), color = 'red', linestyle='dashed', linewidth = 2)\n",
    "    plt.axvline(x=var_data.mode()[0], color = 'yellow', linestyle='dashed', linewidth = 2)\n",
    "\n",
    "    # Show the figure\n",
    "    plt.show()\n",
    "\n",
    "# Get the density of Grade\n",
    "col = df_students['Grade']\n",
    "show_density(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected from the histogram of the sample, the density shows the characteristic 'bell curve\" of what statisticians call a *normal* distribution with the mean and mode at the center and symmetric tails.\n",
    "\n",
    "Now let's take a look at the distribution of the study hours data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the variable to examine\n",
    "col = df_students['StudyHours']\n",
    "# Call the function\n",
    "show_distribution(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the study time data is significantly different from that of the grades.\n",
    "\n",
    "Note that the whiskers of the box plot only extend to around 6.0, indicating that the vast majority of the first quarter of the data is above this value. The minimum is marked with an **o**, indicating that it is statistically an *outlier* - a value that lies significantly outside the range of the rest of the distribution.\n",
    "\n",
    "Outliers can occur for many reasons. Maybe a student meant to record \"10\" hours of study time, but entered \"1\" and missed the \"0\". Or maybe the student was abnormally lazy when it comes to studying! Either way, it's a statistical anomaly that doesn't represent a typical student. Let's see what the distribution looks like without it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get the variable to examine\n",
    "col = df_students[df_students.StudyHours>1]['StudyHours']\n",
    "# Call the function\n",
    "show_distribution(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the dataset is small enough to clearly see that the value **1** is an outlier for the **StudyHours** column, so you can exclude it explicitly. In most real-world cases, it's easier to consider outliers as being values that fall below or above percentiles within which most of the data lie. For example, the following code uses the Pandas **quantile** function to exclude observations below the 0.01th percentile (the value above which 99% of the data reside)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "q01 = df_students.StudyHours.quantile(0.01)\n",
    "# Get the variable to examine\n",
    "col = df_students[df_students.StudyHours>q01]['StudyHours']\n",
    "# Call the function\n",
    "show_distribution(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Tip**: You can also eliminate outliers at the upper end of the distribution by defining a threshold at a high percentile value - for example, you could use the **quantile** function to find the 0.99 percentile below which 99% of the data reside.\n",
    "\n",
    "With the outliers removed, the box plot shows all data within the four quartiles. Note that the distribution is not symmetric like it is for the grade data though - there are some students with very high study times of around 16 hours, but the bulk of the data is between 7 and 13 hours; The few extremely high values pull the mean towards the higher end of the scale.\n",
    "\n",
    "Let's look at the density for this distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the density of StudyHours\n",
    "show_density(col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This kind of distribution is called *right skewed*. The mass of the data is on the left side of the distribution, creating a long tail to the right because of the values at the extreme high end; which pull the mean to the right.\n",
    "\n",
    "#### Measures of variance\n",
    "\n",
    "So now we have a good idea where the middle of the grade and study hours data distributions are. However, there's another aspect of the distributions we should examine: how much variability is there in the data?\n",
    "\n",
    "Typical statistics that measure variability in the data include:\n",
    "\n",
    "- **Range**: The difference between the maximum and minimum. There's no built-in function for this, but it's easy to calculate using the **min** and **max** functions.\n",
    "- **Variance**: The average of the squared difference from the mean. You can use the built-in **var** function to find this.\n",
    "- **Standard Deviation**: The square root of the variance. You can use the built-in **std** function to find this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for col_name in ['Grade','StudyHours']:\n",
    "    col = df_students[col_name]\n",
    "    rng = col.max() - col.min()\n",
    "    var = col.var()\n",
    "    std = col.std()\n",
    "    print('\\n{}:\\n - Range: {:.2f}\\n - Variance: {:.2f}\\n - Std.Dev: {:.2f}'.format(col_name, rng, var, std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of these statistics, the standard deviation is generally the most useful. It provides a measure of variance in the data on the same scale as the data itself (so grade points for the Grade distribution and hours for the StudyHours distribution). The higher the standard deviation, the more variance there is when comparing values in the distribution to the distribution mean - in other words, the data is more spread out.\n",
    "\n",
    "When working with a *normal* distribution, the standard deviation works with the particular characteristics of a normal distribution to provide even greater insight. Run the cell below to see the relationship between standard deviations and the data in the normal distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "\n",
    "# Get the Grade column\n",
    "col = df_students['Grade']\n",
    "\n",
    "# get the density\n",
    "density = stats.gaussian_kde(col)\n",
    "\n",
    "# Plot the density\n",
    "col.plot.density()\n",
    "\n",
    "# Get the mean and standard deviation\n",
    "s = col.std()\n",
    "m = col.mean()\n",
    "\n",
    "# Annotate 1 stdev\n",
    "x1 = [m-s, m+s]\n",
    "y1 = density(x1)\n",
    "plt.plot(x1,y1, color='magenta')\n",
    "plt.annotate('1 std (68.26%)', (x1[1],y1[1]))\n",
    "\n",
    "# Annotate 2 stdevs\n",
    "x2 = [m-(s*2), m+(s*2)]\n",
    "y2 = density(x2)\n",
    "plt.plot(x2,y2, color='green')\n",
    "plt.annotate('2 std (95.45%)', (x2[1],y2[1]))\n",
    "\n",
    "# Annotate 3 stdevs\n",
    "x3 = [m-(s*3), m+(s*3)]\n",
    "y3 = density(x3)\n",
    "plt.plot(x3,y3, color='orange')\n",
    "plt.annotate('3 std (99.73%)', (x3[1],y3[1]))\n",
    "\n",
    "# Show the location of the mean\n",
    "plt.axvline(col.mean(), color='cyan', linestyle='dashed', linewidth=1)\n",
    "\n",
    "plt.axis('off')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The horizontal lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).\n",
    "\n",
    "In any normal distribution:\n",
    "- Approximately 68.26% of values fall within one standard deviation from the mean.\n",
    "- Approximately 95.45% of values fall within two standard deviations from the mean.\n",
    "- Approximately 99.73% of values fall within three standard deviations from the mean.\n",
    "\n",
    "So, since we know that the mean grade is 49.18, the standard deviation is 21.74, and distribution of grades is approximately normal; we can calculate that 68.26% of students should achieve a grade between 27.44 and 70.92.\n",
    "\n",
    "The descriptive statistics we've used to understand the distribution of the student data variables are the basis of statistical analysis; and because they're such an important part of exploring your data, there's a built-in **Describe** method of the DataFrame object that returns the main descriptive statistics for all numeric columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_students.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing data\n",
    "\n",
    "Now that you know something about the statistical distribution of the data in your dataset, you're ready to examine your data to identify any apparent relationships between variables.\n",
    "\n",
    "First of all, let's get rid of any rows that contain outliers so that we have a sample that is representative of a typical class of students. We identified that the StudyHours column contains some outliers with extremely low values, so we'll remove those rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample = df_students[df_students['StudyHours']>1]\n",
    "df_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing numeric and categorical variables\n",
    "\n",
    "The data includes two *numeric* variables (**StudyHours** and **Grade**) and two *categorical* variables (**Name** and **Pass**). Let's start by comparing the numeric **StudyHours** column to the categorical **Pass** column to see if there's an apparent relationship between the number of hours studied and a passing grade.\n",
    "\n",
    "To make this comparison, let's create box plots showing the distribution of StudyHours for each possible Pass value (true and false)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample.boxplot(column='StudyHours', by='Pass', figsize=(8,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the StudyHours distributions, it's immediately apparent (if not particularly surprising) that students who passed the course tended to study for more hours than students who didn't. So if you wanted to predict whether or not a student is likely to pass the course, the amount of time they spend studying may be a good predictive feature.\n",
    "\n",
    "### Comparing numeric variables\n",
    "\n",
    "Now let's compare two numeric variables. We'll start by creating a bar chart that shows both grade and study hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a bar plot of name vs grade and study hours\n",
    "df_sample.plot(x='Name', y=['Grade','StudyHours'], kind='bar', figsize=(8,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chart shows bars for both grade and study hours for each student; but it's not easy to compare because the values are on different scales. Grades are measured in grade points, and range from 3 to 97; while study time is measured in hours and ranges from 1 to 16.\n",
    "\n",
    "A common technique when dealing with numeric data in different scales is to *normalize* the data so that the values retain their proportional distribution, but are measured on the same scale. To accomplish this, we'll use a technique called *MinMax* scaling that distributes the values proportionally on a scale of 0 to 1. You could write the code to apply this transformation; but the **Scikit-Learn** library provides a scaler to do it for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Get a scaler object\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Create a new dataframe for the scaled values\n",
    "df_normalized = df_sample[['Name', 'Grade', 'StudyHours']].copy()\n",
    "\n",
    "# Normalize the numeric columns\n",
    "df_normalized[['Grade','StudyHours']] = scaler.fit_transform(df_normalized[['Grade','StudyHours']])\n",
    "\n",
    "# Plot the normalized values\n",
    "df_normalized.plot(x='Name', y=['Grade','StudyHours'], kind='bar', figsize=(8,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the data normalized, it's easier to see an apparent relationship between grade and study time. It's not an exact match, but it definitely seems like students with higher grades tend to have studied more.\n",
    "\n",
    "So there seems to be a correlation between study time and grade; and in fact, there's a statistical *correlation* measurement we can use to quantify the relationship between these columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_normalized.Grade.corr(df_normalized.StudyHours)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The correlation statistic is a value between -1 and 1 that indicates the strength of a relationship. Values above 0 indicate a *positive* correlation (high values of one variable tend to coincide with high values of the other), while values below 0 indicate a *negative* correlation (high values of one variable tend to coincide with low values of the other). In this case, the correlation value is close to 1; showing a strongly positive correlation between study time and grade.\n",
    "\n",
    "> **Note**: Data scientists often quote the maxim \"*correlation* is not *causation*\". In other words, as tempting as it might be, you shouldn't interpret the statistical correlation as explaining *why* one of the values is high. In the case of the student data, the statistics demonstrates that students with high grades tend to also have high amounts of study time; but this is not the same as proving that they achieved high grades *because* they studied a lot. The statistic could equally be used as evidence to support the nonsensical conclusion that the students studied a lot *because* their grades were going to be high.\n",
    "\n",
    "Another way to visualise the apparent correlation between two numeric columns is to use a *scatter* plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a scatter plot\n",
    "df_sample.plot.scatter(title='Study Time vs Grade', x='StudyHours', y='Grade')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, it looks like there's a discernible pattern in which the students who studied the most hours are also the students who got the highest grades.\n",
    "\n",
    "We can see this more clearly by adding a *regression* line (or a *line of best fit*) to the plot that shows the general trend in the data. To do this, we'll use a statistical technique called *least squares regression*.\n",
    "\n",
    "> **Warning - Math Ahead!**\n",
    ">\n",
    "> Cast your mind back to when you were learning how to solve linear equations in school, and recall that the *slope-intercept* form of a linear equation looks like this:\n",
    ">\n",
    "> \\begin{equation}y = mx + b\\end{equation}\n",
    ">\n",
    "> In this equation, *y* and *x* are the coordinate variables, *m* is the slope of the line, and *b* is the y-intercept (where the line goes through the Y-axis).\n",
    ">\n",
    "> In the case of our scatter plot for our student data, we already have our values for *x* (*StudyHours*) and *y* (*Grade*), so we just need to calculate the intercept and slope of the straight line that lies closest to those points. Then we can form a linear equation that calculates a new *y* value on that line for each of our *x* (*StudyHours*) values - to avoid confusion, we'll call this new *y* value *f(x)* (because it's the output from a linear equation ***f***unction based on *x*). The difference between the original *y* (*Grade*) value and the *f(x)* value is the *error* between our regression line and the actual *Grade* achieved by the student. Our goal is to calculate the slope and intercept for a line with the lowest overall error.\n",
    ">\n",
    "> Specifically, we define the overall error by taking the error for each point, squaring it, and adding all the squared errors together. The line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.\n",
    "\n",
    "Fortunately, you don't need to code the regression calculation yourself - the **SciPy** package includes a **stats** class that provides a **linregress** method to do the hard work for you. This returns (among other things) the coefficients you need for the slope equation - slope (*m*) and intercept (*b*) based on a given pair of variable samples you want to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "\n",
    "#\n",
    "df_regression = df_sample[['Grade', 'StudyHours']].copy()\n",
    "\n",
    "# Get the regression slope and intercept\n",
    "m, b, r, p, se = stats.linregress(df_regression['StudyHours'], df_regression['Grade'])\n",
    "print('slope: {:.4f}\\ny-intercept: {:.4f}'.format(m,b))\n",
    "print('so...\\n f(x) = {:.4f}x + {:.4f}'.format(m,b))\n",
    "\n",
    "# Use the function (mx + b) to calculate f(x) for each x (StudyHours) value\n",
    "df_regression['fx'] = (m * df_regression['StudyHours']) + b\n",
    "\n",
    "# Calculate the error between f(x) and the actual y (Grade) value\n",
    "df_regression['error'] = df_regression['fx'] - df_regression['Grade']\n",
    "\n",
    "# Create a scatter plot of Grade vs StudyHours\n",
    "df_regression.plot.scatter(x='StudyHours', y='Grade')\n",
    "\n",
    "# Plot the regression line\n",
    "plt.plot(df_regression['StudyHours'],df_regression['fx'], color='cyan')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this time, the code plotted two distinct things - the scatter plot of the sample study hours and grades is plotted as before, and then a line of best fit based on the least squares regression coefficients is plotted.\n",
    "\n",
    "The slope and intercept coefficients calculated for the regression line are shown above the plot.\n",
    "\n",
    "The line is based on the ***f*(x)** values calculated for each **StudyHours** value. Run the following cell to see a table that includes the following values:\n",
    "\n",
    "- The **StudyHours** for each student.\n",
    "- The **Grade** achieved by each student.\n",
    "- The ***f(x)*** value calculated using the regression line coefficients.\n",
    "- The *error* between the calculated ***f(x)*** value and the actual **Grade** value.\n",
    "\n",
    "Some of the errors, particularly at the extreme ends, are quite large (up to over 17.5 grade points); but in general, the line is pretty close to the actual grades."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the original x,y values, the f(x) value, and the error\n",
    "df_regression[['StudyHours', 'Grade', 'fx', 'error']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the regression coefficients for prediction\n",
    "\n",
    "Now that you have the regression coefficients for the study time and grade relationship, you can use them in a function to estimate the expected grade for a given amount of study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a function based on our regression coefficients\n",
    "def f(x):\n",
    "    m = 6.3134\n",
    "    b = -17.9164\n",
    "    return m*x + b\n",
    "\n",
    "study_time = 14\n",
    "\n",
    "# Get f(x) for study time\n",
    "prediction = f(study_time)\n",
    "\n",
    "# Grade can't be less than 0 or more than 100\n",
    "expected_grade = max(0,min(100,prediction))\n",
    "\n",
    "#Print the estimated grade\n",
    "print ('Studying for {} hours per week may result in a grade of {:.0f}'.format(study_time, expected_grade))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So by applying statistics to sample data, you've determined a relationship between study time and grade; and encapsulated that relationship in a general function that can be used to predict a grade for a given amount of study time.\n",
    "\n",
    "This technique is in fact the basic premise of machine learning. You can take a set of sample data that includes one or more *features* (in this case, the number of hours studied) and a known *label* value (in this case, the grade achieved) and use the sample data to derive a function that calculates predicted label values for any given set of features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading\n",
    "\n",
    "To learn more about the Python packages you explored in this notebook, see the following documentation:\n",
    "\n",
    "- [NumPy](https://numpy.org/doc/stable/)\n",
    "- [Pandas](https://pandas.pydata.org/pandas-docs/stable/)\n",
    "- [Matplotlib](https://matplotlib.org/contents.html)\n",
    "\n",
    "## Challenge: Analyze Flight Data\n",
    "\n",
    "If this notebook has inspired you to try exploring data for yourself, why not take on the challenge of a real-world dataset containing flight records from the US Department of Transportation? You'll find the challenge in the [/challenges/01 - Flights Challenge.ipynb](./challenges/01%20-%20Flights%20Challenge.ipynb) notebook!\n",
    "\n",
    "> **Note**: The time to complete this optional challenge is not included in the estimated time for this exercise - you can spend as little or as much time on it as you like!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}