pandas_cs.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Cheat Sheet\n",
    "\n",
    "Pandas is a software library written for the Python programming language for data manipulation and analysis. Below are a quick list of popular  methods used in my own data analysis. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pavsingh7/Pandas/blob/master/pandas_cs.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## References and Resources\n",
    "\n",
    "- W3Schools for [Numpy](https://www.w3schools.com/python/numpy/default.asp), [Pandas](https://www.w3schools.com/python/pandas/default.asp), [Matplotlib](https://www.w3schools.com/python/matplotlib_intro.asp)\n",
    "\n",
    "## Set Up\n",
    "\n",
    "First things first, load pandas library.\n",
    "\n",
    "```py\n",
    "import pandas as pd\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load modules\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents \n",
    "\n",
    "- [Read and Writing](#toc2_)    \n",
    "  - [Read](#toc2_1_)    \n",
    "  - [Write](#toc2_2_)    \n",
    "  - [SQL](#toc2_3_)    \n",
    "- [Create](#toc3_)    \n",
    "    - [Dictionary](#toc3_1_1_)    \n",
    "    - [Series](#toc3_1_2_)    \n",
    "    - [DataFrame](#toc3_1_3_)    \n",
    "- [DataFrame Information](#toc4_)    \n",
    "    - [Shape](#toc4_1_1_)    \n",
    "    - [Rows](#toc4_1_2_)    \n",
    "    - [Size](#toc4_1_3_)    \n",
    "    - [Columns](#toc4_1_4_)    \n",
    "    - [Descriptive Statistics](#toc4_1_5_)    \n",
    "    - [Index](#toc4_1_6_)    \n",
    "    - [Unique](#toc4_1_7_)    \n",
    "- [Plotting](#toc5_)    \n",
    "    - [Line](#toc5_1_1_)    \n",
    "    - [Scatter](#toc5_1_2_)    \n",
    "    - [Bar](#toc5_1_3_)    \n",
    "    - [Pie](#toc5_1_4_)    \n",
    "    - [Histogram](#toc5_1_5_)    \n",
    "    - [Boxplot](#toc5_1_6_)    \n",
    "    - [Area Plot](#toc5_1_7_)    \n",
    "    - [Scatter Matrix](#toc5_1_8_)    \n",
    "  - [Kernel Density Estimate](#toc5_2_)    \n",
    "- [Change](#toc6_)    \n",
    "    - [Index](#toc6_1_1_)    \n",
    "    - [Values (replace values)](#toc6_1_2_)    \n",
    "    - [Data Type](#toc6_1_3_)    \n",
    "    - [NaN Values (fill in NA values)](#toc6_1_4_)    \n",
    "- [Grouping](#toc7_)    \n",
    "    - [GroupBy](#toc7_1_1_)    \n",
    "    - [Value Counts](#toc7_1_2_)    \n",
    "- [Deleting](#toc8_)    \n",
    "    - [Index](#toc8_1_1_)    \n",
    "    - [Column](#toc8_1_2_)    \n",
    "    - [Duplicates](#toc8_1_3_)    \n",
    "    - [NaN Values](#toc8_1_4_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Read and Writing](#toc0_)\n",
    "\n",
    "Pandas allows for reading and writing data in various formats, including CSV, Excel, JSON, and SQL.\n",
    "\n",
    "## <a id='toc2_1_'></a>[Read](#toc0_)\n",
    "\n",
    "``` py\n",
    "# Read CSV\n",
    "pd.read_csv('file.csv')\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "``` py\n",
    "# Read Excel\n",
    "pd.read_excel('file.xlsx')\n",
    "```\n",
    "\n",
    "or\n",
    "\n",
    "```py\n",
    "# Read JSON\n",
    "df = pd.read_json('file.json')\n",
    "```\n",
    "\n",
    "\n",
    "## <a id='toc2_2_'></a>[Write](#toc0_)\n",
    "\n",
    "```py\n",
    "# Write to CSV\n",
    "df.to_csv('file.csv', index=False)\n",
    "```\n",
    "\n",
    "``` py\n",
    "# Write to Excel\n",
    "df.to_excel('file.xlsx', index=False)\n",
    "```\n",
    "\n",
    "```py\n",
    "# Write to JSON\n",
    "df.to_json('file.json', index=False)\n",
    "```\n",
    "\n",
    "## <a id='toc2_3_'></a>[SQL](#toc0_)\n",
    "\n",
    "To read and write SQL databases you will need to use `sqlalchemy` library and create an engine to connect to the database:\n",
    "\n",
    "```py\n",
    "# Read and Write SQL\n",
    "from sqlalchemy import create_engine\n",
    "engine = create_engine('postgresql://username:password@host:port/database')\n",
    "df.to_sql('table_name', engine)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read CSV\n",
    "pd.read_csv('file.csv')\n",
    "\n",
    "# Read Excel\n",
    "pd.read_excel('file.xlsx')\n",
    "\n",
    "# Write to CSV\n",
    "df.to_csv('file.csv', index=False)\n",
    "\n",
    "# Write to Excel\n",
    "df.to_excel('file.xlsx', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "***\n",
    "# <a id='toc3_'></a>[Create](#toc0_)\n",
    "\n",
    "We look at creating a:\n",
    "- Series\n",
    "- DataFrame\n",
    "- Dictionary\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_1_'></a>[Dictionary](#toc0_)\n",
    "\n",
    "You can create a dictionary using the `pd.DataFrame.to_dict()` method. This method converts a DataFrame to a dictionary, where the keys are the column names and the values are the data in the columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Converting DataFrame to dictionary\n",
    "my_dict = df.to_dict()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can also use the `pd.Series.to_dict()` method to convert a pandas Series to a dictionary. The keys will be the index values and the values will be the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series\n",
    "data = [1, 2, 3, 4, 5]\n",
    "s = pd.Series(data, index=['a', 'b', 'c', 'd', 'e'])\n",
    "\n",
    "# Converting Series to dictionary\n",
    "my_dict = s.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_2_'></a>[Series](#toc0_)\n",
    "\n",
    "A Pandas series is a one-dimensional labeled array that can hold any data type. Below we create a series from a list, array and dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series from a list\n",
    "data = [1, 2, 3, 4, 5]\n",
    "s = pd.Series(data)\n",
    "\n",
    "# Creating a Series from a numpy array\n",
    "data = np.array([1, 2, 3, 4, 5])\n",
    "s = pd.Series(data)\n",
    "\n",
    "# Creating a Series from a dictionary\n",
    "data = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}\n",
    "s = pd.Series(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <a id='toc3_1_3_'></a>[DataFrame](#toc0_)\n",
    "\n",
    "To create a new DataFrame in pandas, you can use the `pd.DataFrame()` constructor. We create a dataframe  from a list, array and dictionary below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame from a list\n",
    "data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Creating a DataFrame from a numpy array\n",
    "data = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Creating a DataFrame from a dictionary\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# <a id='toc4_'></a>[DataFrame Information](#toc0_)\n",
    "\n",
    "We shall look at methods which help us:\n",
    "\n",
    "- look at shape of DataFrame\n",
    "- Number of rows\n",
    "- Column names and attributes\n",
    "- Descriptive statistics\n",
    "- Index\n",
    "- Unique values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <a id='toc4_1_1_'></a>[Shape](#toc0_)\n",
    "\n",
    "Returns the dimensions of a DataFrame or Series. The first element is the number of rows and the second element is the number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Getting the shape of the DataFrame\n",
    "print(df.shape)  # Output: (3, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Creating a Series\n",
    "data = [1, 2, 3, 4, 5]\n",
    "s = pd.Series(data)\n",
    "\n",
    "# Getting the shape of the Series\n",
    "print(s.shape)  # Output: (5,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <a id='toc4_1_2_'></a>[Rows](#toc0_)\n",
    "\n",
    "The `len()` function returns the number of rows in a DataFrame or Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Getting the number of rows in the DataFrame\n",
    "num_rows = len(df)\n",
    "print(num_rows)  # Output: 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc4_1_3_'></a>[Size](#toc0_)\n",
    "\n",
    "The `.size` attribute returns the total number of elements in a DataFrame or a Series. It is equivalent to the number of rows times the number of columns for a DataFrame, and the number of elements for a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Getting the size of the DataFrame\n",
    "print(df.size)  # Output: 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series\n",
    "data = [1, 2, 3, 4, 5]\n",
    "s = pd.Series(data)\n",
    "\n",
    "# Getting the size of the Series\n",
    "print(s.size)  # Output: 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc4_1_4_'></a>[Columns](#toc0_)\n",
    "\n",
    "To retrieve column/attribute names, use `.columns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Retrieving the column names of the DataFrame\n",
    "column_names = df.columns\n",
    "print(column_names)  # Output: Index(['a', 'b', 'c'], dtype='object')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrieve the column names\n",
    "df.keys()\n",
    "\n",
    "# columns of DataFrame as list\n",
    "list(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc4_1_5_'></a>[Descriptive Statistics](#toc0_)\n",
    "\n",
    "The `describe()` method is used to generate descriptive statistics of a DataFrame or a Series. These statistics include the count, mean, standard deviation, minimum, and maximum of the data, as well as the 25th, 50th, and 75th percentiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Generating descriptive statistics of the DataFrame\n",
    "df.describe().transpose()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# descriptive statistics for Series\n",
    "s = pd.Series([1, 2, 3, 4, 5])\n",
    "s.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc4_1_6_'></a>[Index](#toc0_)\n",
    "\n",
    "Can use the `index` attribute to retrieve the index of a DataFrame or a Series. The `index` attribute returns an Index object containing the index labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Retrieving the index of the DataFrame\n",
    "index = df.index\n",
    "print(index)  # Output: RangeIndex(start=0, stop=3, step=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieving the index of the DataFrame as a list\n",
    "index = df.index.tolist()\n",
    "print(index)  # Output: [0, 1, 2]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc4_1_7_'></a>[Unique](#toc0_)\n",
    "\n",
    "Can use the `unique()` method to retrieve the unique values of a column in a DataFrame or a Series. The `unique()` method returns a NumPy array containing the unique values of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 2, 3], 'b': [4, 5, 5, 6], 'c': [7, 8, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Retrieving the unique values of column 'a'\n",
    "unique_values = df['a'].unique()\n",
    "print(unique_values)  # Output: [1 2 3]\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the `nunique()` method which returns the number of unique elements in the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieving the number of unique values of column 'a'\n",
    "num_unique_values = df['a'].nunique()\n",
    "print(num_unique_values)  # Output: 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# <a id='toc5_'></a>[Plotting](#toc0_)\n",
    "\n",
    "Below are several examples of how to plot different types of plots in pandas without explitly calling `Matplotlib` Library. Infact, Pandas comes with a couple of plotting functionalities applicable on DataFrame or series objects that use the `Matplotlib` library under the hood. \n",
    "\n",
    "     This means any plot created by the Pandas library is a Matplotlib object. \n",
    "\n",
    "- Nice article on plotting [here](https://www.dataquest.io/blog/plot-dataframe-pandas/)\n",
    "\n",
    "Note: The `figsize` argument takes two arguments, width and height in inches, and allows us to change the size of the output figure. The default values of the width and height are 6.4 and 4.8, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_1_'></a>[Line](#toc0_)\n",
    "\n",
    "The default plot is the line plot. Use the `plot()` method with `kind` parameter set to 'line'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3, 4, 5], 'b': [5, 4, 3, 2, 1]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Plotting a line chart\n",
    "df.plot(kind='line', figsize=(15,8), title=\"Line Plot\", ylabel= \"Y-axis\", xlabel=\"X-axis\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_2_'></a>[Scatter](#toc0_)\n",
    "\n",
    "Scatter plots plot data points on the `x` and `y` axes to show the correlation between two variables. Use the `plot()` method with `kind` parameter set to 'scatter'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'x': [1, 2, 3, 4, 5], 'y': [5, 4, 3, 2, 1]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Plotting a scatter chart\n",
    "df.plot(kind='scatter', x='x', y='y', s=None, figsize=(14,7));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_3_'></a>[Bar](#toc0_)\n",
    "\n",
    "Use the `plot()` method with `kind` parameter set to 'bar'. Note, we can create horizontal bar charts by assigning the `barh` string value to the `kind` argument. We can also plot the data on the stacked vertical or horizontal bar charts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Plotting a bar chart\n",
    "df.plot(kind='bar', figsize=(20,8));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting a horizontal bar chart\n",
    "df.plot(kind='barh', figsize=(20,8));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting a stacked bar chart\n",
    "df.plot(kind='bar', stacked=True, figsize=(20,8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_4_'></a>[Pie](#toc0_)\n",
    "\n",
    "Use the `plot()` method with `kind` parameter set to 'pie'. Great proportional representation of numerical data in a column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series\n",
    "data = [1, 2, 3, 4, 5]\n",
    "s = pd.Series(data)\n",
    "\n",
    "# Plotting a pie chart\n",
    "s.plot(kind='pie', figsize=(15,5));"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new keyword argument in the code above is `autopct`, which shows the percent value on the pie chart slices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': np.random.randint(1000, size=10).tolist(), 'b': np.random.randint(1000, size=10).tolist(), 'c': np.random.randint(1000, size=10).tolist()}\n",
    "df = pd.DataFrame(data)\n",
    "s = pd.Series(df.a)\n",
    "\n",
    "# Plotting a pie chart of data from a column\n",
    "s.plot(kind='pie', figsize=(15,5));\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " If we want to represent the data of all the columns in multiple pie charts as subplots, we can assign `True` to the `subplots` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(kind='pie', legend=False, autopct='%.f', subplots=True, figsize=(20,8));\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_5_'></a>[Histogram](#toc0_)\n",
    "\n",
    "Use the `plot()` method with `kind` parameter set to 'line'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': np.random.randn(1000).tolist(), 'b': np.random.randn(1000).tolist(), 'c': np.random.randn(1000).tolist()}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# plotting histogram\n",
    "df.plot(kind='hist', bins=40, alpha=0.6, figsize=(20,8));\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_6_'></a>[Boxplot](#toc0_)\n",
    "\n",
    "Use the `plot()` method with `kind` parameter set to 'line'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'a': np.random.randn(1000).tolist(), 'b': np.random.randn(1000).tolist(), 'c': np.random.randn(1000).tolist()}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# vertical boxplot\n",
    "df.plot(kind='box', figsize=(15,7));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# horizontal boxplot\n",
    "df.plot(kind='box', vert=False, figsize=(18,7));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_7_'></a>[Area Plot](#toc0_)\n",
    "\n",
    "An extension of a line chart that fills the region between the line chart and the x-axis with a color. It creates a stacked area plot by default. Use the `plot()` method with `kind` parameter set to 'area'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create DataFrame\n",
    "data = {'a': np.random.randint(1000, size=100).tolist(), 'b': np.random.randint(1000, size=100).tolist(), 'c': np.random.randint(1000, size=100).tolist()}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# plot area chart\n",
    "df.plot(kind='area', figsize=(20,8));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# non stacked area chart\n",
    "df.plot(kind='area', stacked=False, figsize=(20,8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc5_1_8_'></a>[Scatter Matrix](#toc0_)\n",
    "\n",
    "Can create a scatter plot matrix using the `scatter_matrix` method in `pandas.plotting`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import module\n",
    "from pandas.plotting import scatter_matrix\n",
    "\n",
    "# plot scatter plot matrix\n",
    "df = pd.DataFrame(np.random.randn(100, 4), columns=[\"a\", \"b\", \"c\", \"d\"])\n",
    "scatter_matrix(df, alpha=0.2, figsize=(15, 8), diagonal=\"kde\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc5_2_'></a>[Kernel Density Estimate](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(kind='kde', figsize=(18,8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# <a id='toc6_'></a>[Change](#toc0_)\n",
    "\n",
    "In pandas,  we can change the index of a dataframe or change the values in a dataframe. We could also change an attributes (columns) datatype or fill in NA values with other values. We shall look at these below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <a id='toc6_1_1_'></a>[Index](#toc0_)\n",
    "\n",
    "We can change the index or reset the index. We look at each below. First, using the `set_index()` method: You can use the `set_index()` method to set a column as the new index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Setting column 'a' as the new index\n",
    "df = df.set_index('a')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `reset_index()` method: You can use the `reset_index()` method to reset the index to its default (0-based integers) and move the current index to a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame with a custom index\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data, index=['x', 'y', 'z'])\n",
    "\n",
    "# Resetting the index to its default\n",
    "df = df.reset_index()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc6_1_2_'></a>[Values (replace values)](#toc0_)\n",
    "\n",
    "You can change values in a DataFrame or a Series using the `replace()` method. The `replace()` method takes a dictionary or a list of values as an argument and replaces the values in the DataFrame or Series accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Replacing values\n",
    "df = df.replace({1: 10, 4: 40, 7: 70})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc6_1_3_'></a>[Data Type](#toc0_)\n",
    "\n",
    "You can change the data type of a column in a DataFrame using the `astype()` method. The `astype()` method takes the desired data type as an argument and returns a new DataFrame with the specified data type for the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7.0, 8.0, 9.0]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Changing the data type of column 'c' to int\n",
    "df['c'] = df['c'].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also change the data type of multiple columns at once, by passing a dictionary to the astype() method, where the keys are the column names and the values are the desired data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change multiple columns\n",
    "df = df.astype({'a': 'float32', 'b': 'int8'})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the data type of an entire DataFrame using the `astype()` method or the `apply()` method with `pd.to_*` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4.0, 5.0, 6.0], 'c': ['7', '8', '9']}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Changing the data type of the entire DataFrame\n",
    "df = df.apply(pd.to_numeric, errors='coerce')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc6_1_4_'></a>[NaN Values (fill in NA values)](#toc0_)\n",
    "\n",
    "You can fill in missing or NaN values in a DataFrame or a Series using the `fillna()` method. The `fillna()` method takes a value or a method as an argument and fills in missing or NaN values accordingly. There are other methods, look at `interpolate()` and `ffill` or `bfill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3, None], 'b': [4.0, 5.0, None, 6.0], 'c': ['7', '8', '9', None]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Filling in missing or NaN values with a constant value\n",
    "df.fillna(value=0, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# <a id='toc7_'></a>[Grouping](#toc0_)\n",
    "\n",
    "You can group data in a DataFrame or Series by one or more columns using the `groupby()` method. It is often useful to do this when you have categories and works well with getting several statistics per group or value counts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <a id='toc7_1_1_'></a>[GroupBy](#toc0_)\n",
    "\n",
    "Returns a a `DataFrameGroupBy` or `SeriesGroupBy` object, which can be used to perform various group-wise operations such as aggregation, transformation, and filtering. Can get several statistics as well. \n",
    "\n",
    "The `groupby()` method splits the data into groups based on the values in one or more specified columns, and then applies a function to each group independently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'A': ['foo', 'foo', 'bar', 'bar', 'foo', 'bar'],\n",
    "        'B': ['one', 'one', 'two', 'two', 'two', 'one'],\n",
    "        'C': [1, 2, 3, 4, 5, 6],\n",
    "        'D': [2.0, 3.0, 4.0, 5.0, 6.0, 7.0]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Grouping the DataFrame by column 'A'\n",
    "grouped_df = df.groupby(by='A')\n",
    "\n",
    "# apply function\n",
    "grouped_df.sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also group by multiple columns by passing multiple column names to the `groupby()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby(['A','B']).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc7_1_2_'></a>[Value Counts](#toc0_)\n",
    "\n",
    "The `value_counts()` method is used to count the number of occurrences of each unique value in a Series or a column of a DataFrame. The `value_counts()` method returns a new Series with the counts of each unique value in descending order. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a Series\n",
    "s = pd.Series([1, 2, 3, 2, 1, 3, 2, 1, 3])\n",
    "\n",
    "# Counting the occurrences of each value\n",
    "counts = s.value_counts()\n",
    "counts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'A': ['foo', 'bar', 'baz', 'foo', 'bar'],\n",
    "        'B': [1, 2, 3, 4, 5],\n",
    "        'C': [2.0, 3.0, 4.0, 5.0, 6.0]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Counting the occurrences of each value in column 'A'\n",
    "counts = df['A'].value_counts()\n",
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get percentage\n",
    "counts = df['A'].value_counts(normalize=True)\n",
    "counts\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# <a id='toc8_'></a>[Deleting](#toc0_)\n",
    "\n",
    "In pandas we can remove indexes, drop columns, remove duplicates and NaN values. We shall explore each below. We essentially make use of the  `drop()` method. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc8_1_1_'></a>[Index](#toc0_)\n",
    "\n",
    "You can delete an index from a DataFrame using the `drop()` method. The `drop()` method takes the index label or a list of index labels to be deleted as an argument, and returns a new DataFrame with the specified index(es) deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data, index=[1, 2, 3])\n",
    "\n",
    "# Deleting index 2\n",
    "df = df.drop(2)\n",
    "df\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This deletes the index 2 from the DataFrame and returns a new DataFrame with index 1 and 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop multiple indexes\n",
    "df = df.drop([1,3])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc8_1_2_'></a>[Column](#toc0_)\n",
    "\n",
    "The `drop()` method takes the column label or a list of column labels to be deleted as an argument, and returns a new DataFrame with the specified columns deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Deleting column 'a'\n",
    "df = df.drop(columns='a')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete multiple columns by passing a list\n",
    "df = df.drop(columns=['b','c'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc8_1_3_'></a>[Duplicates](#toc0_)\n",
    "\n",
    "Delete duplicate rows from a DataFrame using the `drop_duplicates()` method. The `drop_duplicates()` method returns a new DataFrame with duplicate rows removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 2, 3, 4, 4], 'b': [4, 5, 5, 6, 7, 7], 'c': [7, 8, 8, 9, 10, 10]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Deleting duplicate rows\n",
    "df = df.drop_duplicates()\n",
    "df\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will remove duplicate rows and return a new DataFrame with unique rows. You can also specify which columns to consider when identifying duplicates by passing a list of column labels to the `drop_duplicates()` method. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will consider only columns 'a' and 'b' to identify and remove duplicates.\n",
    "df = df.drop_duplicates(subset=['a','b'])\n",
    "df\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the keep parameter to keep either first occurrence of duplicate(`keep='first'`) or last occurrence of duplicate(`keep='last'`).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# will keep the last occurrence of the duplicate and remove other duplicate rows\n",
    "df = df.drop_duplicates(keep='last')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc8_1_4_'></a>[NaN Values](#toc0_)\n",
    "\n",
    "You can delete rows or columns containing NaN (Not a Number) values using the `dropna()` method. The `dropna()` method returns a new DataFrame with the NaN values removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a DataFrame\n",
    "data = {'a': [1, 2, 3, None, 5], 'b': [4.0, 5.0, None, 6.0,7.0]}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Deleting rows containing NaN values\n",
    "df = df.dropna()\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Apr  5 2022, 01:53:17) \n[Clang 12.0.0 ]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "592720dbed1256d2a23d8b00e5ee6ece89b12c02a81b97be893b807d9576d7c4"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}