python_advance/pandas_intro.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOWBCqhYSb4Zw09ulmjOPMr",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jngadiub/ML_course_Pavia_23/blob/main/python_advance/pandas_intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction to Pandas\n",
        "\n",
        "Based on [Python Pandas Tutorial A Complete Introduction for Beginners](https://github.com/LearnDataSci/articles/blob/master/Python%20Pandas%20Tutorial%20A%20Complete%20Introduction%20for%20Beginners/notebook.ipynb).\n",
        "\n",
        "[**Pandas**](https://github.com/pandas-dev/pandas) is a Python library used for working with data sets. It has functions for analyzing, cleaning, exploring, and manipulating data. The name \"Pandas\" has a reference to both \"Panel Data\", and \"Python Data Analysis\" and was created by Wes McKinney in 2008."
      ],
      "metadata": {
        "id": "2qJD6xcWV2H_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "print(pd.__version__) "
      ],
      "metadata": {
        "id": "N5uozK5uY2EZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Core components: Series and DataFrames\n",
        "\n",
        "The primary two components of pandas are the `Series` and `DataFrame`.\n",
        "\n",
        "A `Series` is essentially a column, and a `DataFrame` is a multi-dimensional table made up of a collection of `Series`. They are quite similar in that many operations that you can do with one you can do with the other, such as filling in null values and calculating the mean. In the following we focus on `DataFrame` objects."
      ],
      "metadata": {
        "id": "hvQGMcG41AUn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating DataFrame from scratch\n",
        "\n",
        "There are many ways to create a `DataFrame` from scratch, but a great option is to just use a simple python `dict` and then pass it to the `DataFrame` constructor:"
      ],
      "metadata": {
        "id": "EMzwzGAqZQeR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = {\n",
        "  \"calories\": [420, 380, 390],\n",
        "  \"duration\": [50, 40, 45]\n",
        "}\n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df"
      ],
      "metadata": {
        "id": "dTGEGve3ZSl6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Each *(key, value)* item in `data dict` corresponds to a column in the resulting `DataFrame`. The **index** of this `DataFrame` was given to us on creation as the numbers `0-3`, but we could also create our own when we initialize the `DataFrame`. Let's have customer names as our `index`:"
      ],
      "metadata": {
        "id": "iOqZKPmp137W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.DataFrame(data, index = [\"day1\", \"day2\", \"day3\"])\n",
        "df"
      ],
      "metadata": {
        "id": "X89UMDc32Zgd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `DataFrame` object uses the `loc` attribute to return one or more specified row(s) using the row index name (or number):"
      ],
      "metadata": {
        "id": "K_jcPw7NajZ-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#return row \"day2\"\n",
        "df.loc[\"day2\"]"
      ],
      "metadata": {
        "id": "1Fq-5Fy0eVu6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#return row \"day1\" and \"day3\"\n",
        "df.loc[[\"day1\",\"day3\"]]"
      ],
      "metadata": {
        "id": "a_sTzSYq2_Pq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Read data\n",
        "\n",
        "It’s quite simple to load data from various file formats into a DataFrame. \n",
        "\n",
        "### From CSV and JSON Files\n",
        "\n",
        "A simple way to store big data sets is to use CSV files (comma separated files).\n",
        "CSV files contains plain text and is a well know format that can be read by everyone including Pandas. In our examples we will be using a CSV file called [`data.csv`](https://github.com/jngadiub/ML_course_Pavia_23/blob/main/python_advance/data.csv)."
      ],
      "metadata": {
        "id": "zSUjgeBJebHR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#fetch file from git\n",
        "!curl https://raw.githubusercontent.com/jngadiub/ML_course_Pavia_23/main/python_advance/data.csv -o data.csv\n",
        "!head data.csv"
      ],
      "metadata": {
        "id": "_5ww1GnpedjI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#load in DataFrame\n",
        "df = pd.read_csv('data.csv')\n",
        "df"
      ],
      "metadata": {
        "id": "Z30l9HnAk1Sx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you have a large `DataFrame` with many rows, Pandas will only return the first 5 rows, and the last 5 rows as above. To print the entire DF you can use the method `to_string()`:"
      ],
      "metadata": {
        "id": "mKDqTxE1mNLv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(df.to_string())"
      ],
      "metadata": {
        "id": "LAwZo4JimWz6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Big data sets are often stored, or extracted as `JSON`. It is plain text, but has the format of an object, and is well known in the world of programming, including Pandas. In our examples we will be using a `JSON` file called `data.json`."
      ],
      "metadata": {
        "id": "l3T1MOJLnkUe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#fetch file from git\n",
        "!curl https://raw.githubusercontent.com/jngadiub/ML_course_Pavia_23/main/python_advance/data.json -o data.json\n",
        "!head data.json"
      ],
      "metadata": {
        "id": "X4-JaKCidOO5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#load in DF\n",
        "df = pd.read_json('data.json')\n",
        "print(df.to_string()) "
      ],
      "metadata": {
        "id": "qkdy1pONdN5i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If your `JSON` object is not in a file, but in a Python Dictionary, you can load it into a `DataFrame` directly:"
      ],
      "metadata": {
        "id": "u-QlXAXJoMyj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = {\n",
        "  \"Duration\":{\n",
        "    \"0\":60,\n",
        "    \"1\":60,\n",
        "    \"2\":60,\n",
        "    \"3\":45,\n",
        "    \"4\":45,\n",
        "    \"5\":60\n",
        "  },\n",
        "  \"Pulse\":{\n",
        "    \"0\":110,\n",
        "    \"1\":117,\n",
        "    \"2\":103,\n",
        "    \"3\":109,\n",
        "    \"4\":117,\n",
        "    \"5\":102\n",
        "  },\n",
        "  \"Maxpulse\":{\n",
        "    \"0\":130,\n",
        "    \"1\":145,\n",
        "    \"2\":135,\n",
        "    \"3\":175,\n",
        "    \"4\":148,\n",
        "    \"5\":127\n",
        "  },\n",
        "  \"Calories\":{\n",
        "    \"0\":409,\n",
        "    \"1\":479,\n",
        "    \"2\":340,\n",
        "    \"3\":282,\n",
        "    \"4\":406,\n",
        "    \"5\":300\n",
        "  }\n",
        "}\n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df"
      ],
      "metadata": {
        "id": "BbLzq_Z8Yp4x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Convert back to CSV and JSON file\n",
        "\n",
        "So after extensive work on cleaning your data, you’re now ready to save it as a file of your choice. Similar to the ways we read in data, Pandas provides intuitive commands to save it:"
      ],
      "metadata": {
        "id": "1AI2QoWQ4IDO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv('new_data.csv')\n",
        "df.to_json('new_data.json')\n",
        "!ls"
      ],
      "metadata": {
        "id": "VyI3NmZS4RdW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Important DataFrame operations\n",
        "\n",
        "DataFrames possess hundreds of methods and other operations that are crucial to any analysis. As a beginner, you should know the operations that perform simple transformations of your data and those that provide fundamental statistical analysis.\n",
        "\n",
        "Let's load in the IMDB movies dataset and designate the movie titles to be our index:"
      ],
      "metadata": {
        "id": "S0DxZtMG5NF2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#fetch csv movie file from git\n",
        "!curl https://raw.githubusercontent.com/jngadiub/ML_course_Pavia_23/main/python_advance/IMDB-Movie-Data.csv -o IMDB-Movie-Data.csv\n",
        "!head IMDB-Movie-Data.csv"
      ],
      "metadata": {
        "id": "g6Wqs1EN5ThO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#load into DF\n",
        "movies_df = pd.read_csv(\"IMDB-Movie-Data.csv\", index_col=\"Title\")"
      ],
      "metadata": {
        "id": "ePFdAvj16Hwv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Viewing the data\n",
        "\n",
        "The first thing to do when opening a new dataset is print out a few rows to keep as a visual reference. We accomplish this with `head()`:\n"
      ],
      "metadata": {
        "id": "vxMUaGGw58ug"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.head()"
      ],
      "metadata": {
        "id": "X2iRGGXb6DiK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `head()` method outputs the first five rows of your `DataFrame` by default, but we could also pass a number as well: `movies_df.head(10)` would output the top ten rows, for example.\n",
        "\n",
        "To see the last five rows use `tail()`. It also accepts a number, and in this case we printing the bottom two rows:"
      ],
      "metadata": {
        "id": "FWMNaa856T2x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.tail(2)"
      ],
      "metadata": {
        "id": "Bzy0__TU6sCb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Typically when we load in a dataset, we like to view the first five or so rows to see what's under the hood. Here we can see the names of each column, the index, and examples of values in each row.\n",
        "\n",
        "You'll notice that the index in our `DataFrame` is the *Title* column, which you can tell by how the word *Title* is slightly lower than the rest of the columns headers.\n"
      ],
      "metadata": {
        "id": "W9zkudIP63KZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Getting info about the data\n",
        "\n",
        "The `info()` should be one of the very first commands you run after loading your data:"
      ],
      "metadata": {
        "id": "xy4dLxAp7Ghy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.info()"
      ],
      "metadata": {
        "id": "eFssdbbb7ESC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `info()` method provides the essential details about your dataset, such as the number of rows and columns, the number of non-null values, what type of data is in each column, and how much memory your `DataFrame` is using.\n",
        "\n",
        "Notice in our movies dataset we have some obvious missing values in the *Revenue* and *Metascore* columns. We'll look at how to handle those below.\n",
        "\n",
        "Seeing the datatype quickly is actually quite useful. Imagine you just imported some JSON and the integers were recorded as strings. You go to do some arithmetic and find an *unsupported operand exception* because you can't do math with strings. Calling `info()` will quickly point out that your column you thought was all integers are actually string objects.\n",
        "\n",
        "Another fast and useful attribute is `shape`, which outputs just a tuple of (rows, columns):\n"
      ],
      "metadata": {
        "id": "GvOBsX1a7UPr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.shape"
      ],
      "metadata": {
        "id": "wfAJD-gq7tX_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Handling duplicates\n",
        "\n",
        "This dataset does not have duplicate rows, but it is always important to verify you aren't aggregating duplicate rows.\n",
        "\n",
        "To demonstrate, let's simply just double up our movies DataFrame by appending it to itself:"
      ],
      "metadata": {
        "id": "vM0Sj0jz7zEh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = movies_df.append(movies_df)\n",
        "temp_df.shape"
      ],
      "metadata": {
        "id": "qWxdIde474YU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using `append()` will return a copy without affecting the original `DataFrame`. We are capturing this copy in temp so we aren't working with the real data. Notice call `shape` quickly proves our `DataFrame` rows have doubled.\n",
        "\n",
        "Now we can try dropping duplicates:"
      ],
      "metadata": {
        "id": "ERU4Z8GW7_WM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = temp_df.drop_duplicates()\n",
        "temp_df.shape"
      ],
      "metadata": {
        "id": "T0p02aI98FtB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Just like `append()`, the `drop_duplicates()` method will also return a copy of your `DataFrame`, but this time with duplicates removed. Calling `shape` confirms we're back to the `1000` rows of our original dataset.\n",
        "\n",
        "It's a little verbose to keep assigning `DataFrames` to the same variable like in this example. For this reason, Pandas has the inplace keyword argument on many of its methods. Using `inplace=True` will modify the `DataFrame` object in place:\n"
      ],
      "metadata": {
        "id": "gpn8vUsE8MPa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = movies_df.append(movies_df) # make a new copy\n",
        "temp_df.drop_duplicates(inplace=True)\n",
        "temp_df.shape"
      ],
      "metadata": {
        "id": "aRpyZnJt8Wec"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now our `temp_df` will have the transformed data automatically.\n",
        "\n",
        "Another important argument for `drop_duplicates()` is `keep`, which has three possible options:\n",
        "\n",
        "* `first`: (default) Drop duplicates except for the first occurrence.\n",
        "* `last`: Drop duplicates except for the last occurrence.\n",
        "* `False`: Drop all duplicates.\n",
        "\n",
        "Since we didn't define the `keep` arugment in the previous example it was defaulted to `first`. This means that if two rows are the same pandas will drop the second row and keep the first row. Using `last` has the opposite effect: the first row is dropped. `False`, on the other hand, will drop all duplicates. If two rows are the same then both will be dropped. Example:\n"
      ],
      "metadata": {
        "id": "1TPd6wa38ruA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = movies_df.append(movies_df)  # make a new copy\n",
        "temp_df.drop_duplicates(inplace=True, keep=False)\n",
        "temp_df.shape"
      ],
      "metadata": {
        "id": "lwhEKk-48_rE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since all rows were duplicates, `keep=False` dropped them all resulting in zero rows being left over. If you're wondering why you would want to do this, one reason is that it allows you to locate all duplicates in your dataset. When conditional selections are shown below you'll see how to do that."
      ],
      "metadata": {
        "id": "yzwbN-EG9K40"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Column cleanup\n",
        "\n",
        "Many times datasets will have verbose column names with symbols, upper and lowercase words, spaces, and typos. To make selecting data by column name easier we can spend a little time cleaning up their names.\n",
        "\n",
        "Here's how to print the column names of our dataset:"
      ],
      "metadata": {
        "id": "dxnC8jZx9Px4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.columns"
      ],
      "metadata": {
        "id": "q4cz_ro5-Vw2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Not only does the attribute `columns` come in handy if you want to rename columns by allowing for simple copy and paste, it's also useful if you need to understand why you are receiving a `Key Error` when selecting data by column.\n",
        "\n",
        "We can use the `rename()` method to rename certain or all columns via a `dict`. We don't want parentheses, so let's rename those:\n"
      ],
      "metadata": {
        "id": "eUhogj5w-dyE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.rename(columns={\n",
        "        'Runtime (Minutes)': 'Runtime', \n",
        "        'Revenue (Millions)': 'Revenue_millions'\n",
        "    }, inplace=True)\n",
        "movies_df.columns"
      ],
      "metadata": {
        "id": "8zgcVgFw-lcb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "What if we want to lowercase all names? Instead of using `rename()` we could also set a list of names to the columns:"
      ],
      "metadata": {
        "id": "b0AFEUIm-sbC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.columns = ['rank', 'genre', 'description', 'director', 'actors', 'year', 'runtime', \n",
        "                     'rating', 'votes', 'revenue_millions', 'metascore']\n",
        "movies_df.columns"
      ],
      "metadata": {
        "id": "_bt0kYrS-wv5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Faster approach: instead of just renaming each column manually we can do a list comprehension:\n"
      ],
      "metadata": {
        "id": "gL-7VweA-3F0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Reload original\n",
        "movies_df = pd.read_csv(\"IMDB-Movie-Data.csv\", index_col=\"Title\")\n",
        "\n",
        "#remove parentheses\n",
        "movies_df.rename(columns={\n",
        "        'Runtime (Minutes)': 'Runtime', \n",
        "        'Revenue (Millions)': 'Revenue_millions'\n",
        "    }, inplace=True)\n",
        "\n",
        "#change to lower case in column headers\n",
        "movies_df.columns = [col.lower() for col in movies_df]\n",
        "\n",
        "#check result\n",
        "movies_df.columns"
      ],
      "metadata": {
        "id": "D43pYJOC_A0m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Missing values\n",
        "\n",
        "When exploring data, you’ll most likely encounter missing or null values, which are essentially placeholders for non-existent values. Most commonly you'll see Python's `None` or NumPy's `np.nan`, each of which are handled differently in some situations.\n",
        "\n",
        "There are two options in dealing with nulls:\n",
        "\n",
        "* Get rid of rows or columns with nulls\n",
        "* Replace nulls with non-null values, a technique known as **imputation**\n",
        "\n",
        "Let's calculate to total number of nulls in each column of our dataset. The first step is to check which cells in our `DataFrame` are null:"
      ],
      "metadata": {
        "id": "kbHd5KuL_UPX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.isnull()"
      ],
      "metadata": {
        "id": "TVrl1cl9_jkM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice `isnull()` returns a `DataFrame` where each cell is either `True` or `False` depending on that cell's null status.\n",
        "\n",
        "To count the number of nulls in each column we use an aggregate function for summing:\n"
      ],
      "metadata": {
        "id": "hSFkAFy1_pPA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.isnull().sum()"
      ],
      "metadata": {
        "id": "PVgnz1_M_tju"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `isnull()` method just by iteself isn't very useful, and is usually used in conjunction with other methods, like `sum()`.\n",
        "\n",
        "We can see now that our data has `128` missing values for `revenue_millions` and `64` missing values for metascore.\n"
      ],
      "metadata": {
        "id": "IxduGzZV_yrS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Remove null values\n",
        "\n",
        "Data Scientists and Analysts regularly face the dilemma of dropping or imputing null values, and is a decision that requires intimate knowledge of your data and its context. Overall, removing null data is only suggested if you have a small amount of missing data.\n",
        "\n",
        "Remove nulls is pretty simple:\n"
      ],
      "metadata": {
        "id": "HZqeMgv5AQu2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = movies_df.dropna()\n",
        "temp_df.isnull().sum()"
      ],
      "metadata": {
        "id": "TdJNYhjAAgES"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This operation will delete any row with at least a single null value, but it will return a new `DataFrame` without altering the original one. You could specify `inplace=True` in this method as well.\n",
        "\n",
        "So in the case of our dataset, this operation would remove `128` rows where `revenue_millions` is null and `64` rows where `metascore` is null. This obviously seems like a waste since there's perfectly good data in the other columns of those dropped rows. That's why we'll look at *imputation* next.\n",
        "\n",
        "Other than just dropping rows, you can also drop columns with null values by setting `axis=1`:\n"
      ],
      "metadata": {
        "id": "9tlx7jl-A6bB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "temp_df = movies_df.dropna(axis=1)\n",
        "temp_df.shape"
      ],
      "metadata": {
        "id": "Da8PCG3cBIgD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Imputation\n",
        "\n",
        "*Imputation* is a conventional feature engineering technique used to keep valuable data that have null values. There may be instances where dropping every row with a null value removes too big a chunk from your dataset, so instead we can impute that null with another value, usually the mean or the median of that column.\n",
        "\n",
        "Let's look at imputing the missing values in the `revenue_millions` column. First we'll extract that column into its own variable:\n"
      ],
      "metadata": {
        "id": "tBwtu3D1BgEJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "revenue = movies_df['revenue_millions']\n",
        "revenue.head()"
      ],
      "metadata": {
        "id": "kicLokd0Brp2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll impute the missing values of revenue using the mean. Here's the mean value:"
      ],
      "metadata": {
        "id": "F--elTspB7w_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "revenue_mean = revenue.mean()\n",
        "revenue_mean"
      ],
      "metadata": {
        "id": "gWQmE66aB8uu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's fill the nulls with the mean using `fillna()`:"
      ],
      "metadata": {
        "id": "nPz42HumCCUf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "revenue.fillna(revenue_mean, inplace=True)"
      ],
      "metadata": {
        "id": "zJkkxpvdCFyW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have now replaced all nulls in `revenue` with the mean of the column. Notice that by using `inplace=True` we have actually affected the original `movies_df`:\n"
      ],
      "metadata": {
        "id": "_Fw9_LQmCL5A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.isnull().sum()"
      ],
      "metadata": {
        "id": "Q5IAVLIlCQo7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Imputing an entire column with the same value like this is a basic example. It would be a better idea to try a more granular imputation by `Genre` or `Director`.\n",
        "\n",
        "For example, you would find the mean of the revenue generated in each genre individually and impute the nulls in each genre with that genre's mean.\n",
        "\n",
        "Let's now look at more ways to examine and understand the dataset.\n"
      ],
      "metadata": {
        "id": "W4EyRCMPCaCd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrieve data statistics\n",
        "\n",
        "Using `describe()` on an entire `DataFrame` we can get a summary of the distribution of continuous variables:"
      ],
      "metadata": {
        "id": "LTHQJ3xICj0i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.describe()"
      ],
      "metadata": {
        "id": "XXKIrJRCEUiV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Understanding which numbers are continuous also comes in handy when thinking about the type of plot to use to represent your data visually.\n",
        "\n",
        "The `describe()` method can also be used on a categorical variable to get the count of rows, unique count of categories, top category, and freq of top category:\n"
      ],
      "metadata": {
        "id": "8UD_-Z0lEw-T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df['genre'].describe()"
      ],
      "metadata": {
        "id": "0ep_KfE0E1T0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This tells us that the genre column has `207` unique values, the top value is Action/Adventure/Sci-Fi, which shows up `50` times (freq).\n",
        "\n",
        "The `value_counts()` method can tell us the frequency of all values in a column:\n"
      ],
      "metadata": {
        "id": "Lv_LtH4JE6bo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df['genre'].value_counts().head(10)"
      ],
      "metadata": {
        "id": "YjlthBJPE-KP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Relationships between continuous variables\n",
        "\n",
        "By using the correlation method `corr()` we can generate the relationship between each continuous variable:\n"
      ],
      "metadata": {
        "id": "rfDpj4VzFLO1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.corr()"
      ],
      "metadata": {
        "id": "KUe7BNgmFRvY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Correlation tables are a numerical representation of the bivariate relationships in the dataset.\n",
        "\n",
        "Positive numbers indicate a positive correlation — one goes up the other goes up — and negative numbers represent an inverse correlation — one goes up the other goes down. `1.0` indicates a perfect correlation.\n",
        "\n",
        "So looking in the first row, first column we see `rank` has a perfect correlation with itself, which is obvious. On the other hand, the correlation between `votes` and `revenue_millions` is `0.6`. A little more interesting.\n",
        "\n",
        "Examining bivariate relationships comes in handy when you have an outcome or dependent variable in mind and would like to see the features most correlated to the increase or decrease of the outcome. You can visually represent bivariate relationships with scatterplots (seen below in the plotting section)."
      ],
      "metadata": {
        "id": "XrRA1POSFdnc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### DataFrame slicing, selecting, extracting\n",
        "\n",
        "Up until now we've focused on some basic summaries of our data. We've learned about simple column extraction using single brackets, and we imputed null values in a column using `fillna()`. Below are the other methods of slicing, selecting, and extracting you'll need to use constantly.\n",
        "\n",
        "It's important to note that, although many methods are the same, `DataFrames` and `Series` have different attributes, so you'll need be sure to know which type you are working with or else you will receive attribute errors.\n",
        "\n"
      ],
      "metadata": {
        "id": "Gm_pTzH5Fo-M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### By column\n",
        "\n",
        "You already saw how to extract a column using square brackets like this:"
      ],
      "metadata": {
        "id": "RY3crHnUF2mz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "genre_col = movies_df['genre']\n",
        "type(genre_col)"
      ],
      "metadata": {
        "id": "-Z09hu5nF6vr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This will return a `Series`. To extract a column as a `DataFrame`, you need to pass a list of column names. In our case that's just a single column:\n"
      ],
      "metadata": {
        "id": "Gxs1v6ZGF-8q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "genre_col = movies_df[['genre']]\n",
        "type(genre_col)"
      ],
      "metadata": {
        "id": "a4TSzlprGDKD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since it's just a list, adding another column name is easy:"
      ],
      "metadata": {
        "id": "YkA27cbWGH9J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "subset = movies_df[['genre', 'rating']]\n",
        "subset.head()"
      ],
      "metadata": {
        "id": "RyEjmLxDGIlP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### By row\n",
        "\n",
        "For rows, we have two options:\n",
        "\n",
        "* `loc` - locates by name\n",
        "* `iloc` - locates by numerical index\n",
        "\n",
        "Remember that we are still indexed by movie *Title*, so to use `loc` we give it the *Title* of a movie:\n"
      ],
      "metadata": {
        "id": "Frr50NeyGMiU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prom = movies_df.loc[\"Prometheus\"]\n",
        "prom"
      ],
      "metadata": {
        "id": "kpmsxW-xGXjl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "On the other hand, with `iloc` we give it the numerical index of *Prometheus*:"
      ],
      "metadata": {
        "id": "p-pEimAXGdLk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prom = movies_df.iloc[1]\n",
        "prom"
      ],
      "metadata": {
        "id": "N7i_ob_aGgae"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The methods `loc` and `iloc` can be thought of as similar to Python list slicing. To show this even further, let's select multiple rows.\n",
        "\n",
        "How would you do it with a list? In Python, just slice with brackets like `example_list[1:4]`. It's works the same way in Pandas:\n"
      ],
      "metadata": {
        "id": "mjOP9K_KGqQJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#by name index\n",
        "movie_subset = movies_df.loc['Prometheus':'Sing']\n",
        "movie_subset"
      ],
      "metadata": {
        "id": "kMb1AWbQGu7D"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#by numerical index\n",
        "movie_subset = movies_df.iloc[1:4]\n",
        "movie_subset"
      ],
      "metadata": {
        "id": "Cwgip5LbHBCi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Conditional selections\n",
        "\n",
        "We’ve gone over how to select columns and rows, but what if we want to make a conditional selection?\n",
        "\n",
        "For example, what if we want to filter our movies `DataFrame` to show only films directed by Ridley Scott or films with a rating greater than or equal to 8.0?\n",
        "\n",
        "To do that, we take a column from the `DataFrame` and apply a Boolean condition to it. Here's an example of a Boolean condition:"
      ],
      "metadata": {
        "id": "o0koHKTdHQVV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "condition = (movies_df['director'] == \"Ridley Scott\")\n",
        "condition.head()"
      ],
      "metadata": {
        "id": "HzFQRyZmHZ-Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similar to `isnull()`, this returns a `Series` of `True` and `False` values: `True` for films directed by Ridley Scott and `False` for ones not directed by him.\n",
        "\n",
        "We want to filter out all movies directed by Ridley Scott, in other words, we don’t want the `False` films. To return the rows where that condition is `True` we have to pass this operation into the `DataFrame`:\n"
      ],
      "metadata": {
        "id": "1Y4v_w8lHhqB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[movies_df['director'] == \"Ridley Scott\"].head()"
      ],
      "metadata": {
        "id": "CZzrl2WeH10f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's look at conditional selections using numerical values by filtering the `DataFrame` by ratings:"
      ],
      "metadata": {
        "id": "T7dhSegAH9Mf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[movies_df['rating'] >= 8.6].head(3)"
      ],
      "metadata": {
        "id": "ObiMA42NH-0-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can make some richer conditionals by using logical operators `|` or `&`.\n",
        "\n",
        "Let's filter the `DataFrame` to show only movies by Christopher Nolan OR Ridley Scott:\n"
      ],
      "metadata": {
        "id": "F34teJHXIFPr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[(movies_df['director'] == 'Christopher Nolan') | (movies_df['director'] == 'Ridley Scott')].head()"
      ],
      "metadata": {
        "id": "3SR7bc0gIMOB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the `isin()` method we could make this more concise though:"
      ],
      "metadata": {
        "id": "PJd_G112ITSd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[movies_df['director'].isin(['Christopher Nolan', 'Ridley Scott'])].head()"
      ],
      "metadata": {
        "id": "pvWL-VupIUiG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we want all movies that were released between 2005 and 2010, have a rating above 8.0, but made below the 25th percentile in revenue:\n"
      ],
      "metadata": {
        "id": "8wDG2MamIjBv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[\n",
        "    ((movies_df['year'] >= 2005) & (movies_df['year'] <= 2010))\n",
        "    & (movies_df['rating'] > 8.0)\n",
        "    & (movies_df['revenue_millions'] < movies_df['revenue_millions'].quantile(0.25))\n",
        "]"
      ],
      "metadata": {
        "id": "sEXvLYkvHq-3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The apply function\n",
        "\n",
        "It is possible to iterate over a `DataFrame` or `Series` as you would with a list, but doing so — especially on large datasets — is very slow.\n",
        "\n",
        "An efficient alternative is the `apply()` method. For example, we could use a function to convert movies with an 8.0 or greater to a string value of \"good\" and the rest to \"bad\" and use this transformed values to create a new column.\n",
        "\n",
        "First we would create a function that, when given a rating, determines if it's good or bad:"
      ],
      "metadata": {
        "id": "rjaJ-mdaI6j9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def rating_function(x):\n",
        "    if x >= 8.0:\n",
        "        return \"good\"\n",
        "    else:\n",
        "        return \"bad\""
      ],
      "metadata": {
        "id": "FGucsfcDJdRZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we want to send the entire rating column through this function, which is what `apply()` does:\n"
      ],
      "metadata": {
        "id": "hrA6Uh7wJhj6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[\"rating_category\"] = movies_df[\"rating\"].apply(rating_function)\n",
        "movies_df.head(2)"
      ],
      "metadata": {
        "id": "p6mtBWqNJk7o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `apply()` method passes every value in the rating column through the `rating_function` and then returns a new `Series`. This `Series` is then assigned to a new column called `rating_category`.\n",
        "\n",
        "You can also use anonymous functions as well. This `lambda` function achieves the same result as `rating_function`:\n"
      ],
      "metadata": {
        "id": "ppOMQiDEJqVF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df[\"rating_category\"] = movies_df[\"rating\"].apply(lambda x: 'good' if x >= 8.0 else 'bad')\n",
        "movies_df.head(2)"
      ],
      "metadata": {
        "id": "mKLHYj_7JyFb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Overall, using `apply()` will be much faster than iterating manually over rows because pandas is utilizing vectorization."
      ],
      "metadata": {
        "id": "jEKShlwBJ5vp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Plotting\n",
        "\n",
        "Another great thing about Pandas is that it integrates with Matplotlib, so you get the ability to plot directly off `DataFrames` and `Series`.\n",
        "\n",
        "Let's plot the relationship between ratings and revenue. All we need to do is call `plot()` with some info about how to construct the plot:"
      ],
      "metadata": {
        "id": "YVQSqC_iJ8ko"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.plot(kind='scatter', x='rating', y='revenue_millions', title='Revenue (millions) vs Rating')"
      ],
      "metadata": {
        "id": "gp1UxdtgKYaP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we want to plot a simple histogram based on a single column, we can call `plot()` on a column:"
      ],
      "metadata": {
        "id": "20rBlPCLKfe-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df['rating'].plot(kind='hist', title='Rating')"
      ],
      "metadata": {
        "id": "JXnzIEF8Kia-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Do you remember the `describe()` method at the beginning of this tutorial? Well, there's a graphical representation of the interquartile range, called the `Boxplot`. Let's recall what `describe()` gives us on the ratings column:\n"
      ],
      "metadata": {
        "id": "j_GUsmLsK3dj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df['rating'].describe()"
      ],
      "metadata": {
        "id": "ypLmgWr5K8av"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using a `Boxplot` we can visualize this data:"
      ],
      "metadata": {
        "id": "c8ZvIpH4K_hw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df['rating'].plot(kind=\"box\");"
      ],
      "metadata": {
        "id": "FqnPguidLBkv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "By combining categorical and continuous data, we can create a `Boxplot` of revenue that is grouped by the rating category we created above:"
      ],
      "metadata": {
        "id": "kSoplEwBLOg5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movies_df.boxplot(column='revenue_millions', by='rating_category');"
      ],
      "metadata": {
        "id": "cw3XvdwULHnC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "That's the general idea of plotting with Pandas. There's too many plots to mention, so definitely take a look at the [`plot()` docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html) for more information on what it can do."
      ],
      "metadata": {
        "id": "WxkBmk3MLhbp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Conclusions\n",
        "\n",
        "Exploring, cleaning, transforming, and visualization data with Pandas in Python is an essential skill in data science. Just cleaning wrangling data is 80% of your job as a Data Scientist. After a few projects and some practice, you should be very comfortable with most of the basics.\n",
        "\n",
        "To keep improving, view the [extensive tutorials](https://pandas.pydata.org/pandas-docs/stable/getting_started/tutorials.html) offered by the official Pandas docs."
      ],
      "metadata": {
        "id": "9HhwMahYLqTf"
      }
    }
  ]
}