03-DataVisualisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Ce7gDSKmMjW"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/shaneahmed/StatswithPython/blob/main/03-DataVisualisation.ipynb\" target=\"_blank\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "<a href=\"https://kaggle.com/kernels/welcome?src=https://github.com/shaneahmed/StatswithPython/blob/main/03-DataVisualisation.ipynb\" target=\"_blank\"><img src=\"https://kaggle.com/static/images/open-in-kaggle.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oyi6pkjpmMja"
   },
   "source": [
    "# Data Visualisation Techniques\n",
    "\n",
    "### by Shan E Ahmed Raza\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "**Name: (Please write your name and ID here prior to submission)**\n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "In this notebook we will focus on data visualisation techniques using python. We will mainly work with three python packages pandas, seaborn and matplotlib. We will also consider calculating skewness, kurtosis and correlation between multiple variables.\n",
    "\n",
    "\n",
    "Let's start with installing required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b_IDPnE9mMjb",
    "outputId": "920ff120-e679-4e7e-9e9a-d3243ad0746c"
   },
   "outputs": [],
   "source": [
    "!pip install pandas seaborn matplotlib scipy scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hOEsjhd3mMjc"
   },
   "source": [
    "Import required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HzlL1540mMjc"
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bWuMrt5wmMjd"
   },
   "source": [
    "We will start with loading builtin data set `iris` in `seaborn` package. The [iris (Fisher data set)](https://archive.ics.uci.edu/ml/datasets/iris) is perhaps the best known database to be found in the pattern recognition literature. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The data set has following attributes:\n",
    "\n",
    "1. sepal length in cm\n",
    "2. sepal width in cm\n",
    "3. petal length in cm\n",
    "4. petal width in cm\n",
    "5. class:\\\n",
    "    -- Iris Setosa\\\n",
    "    -- Iris Versicolour\\\n",
    "    -- Iris Virginica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "tLZlzQ75mbeW",
    "outputId": "55655526-a764-4184-bb2b-ae3dacaed1b8"
   },
   "outputs": [],
   "source": [
    "# you can either download the iris data from sklearn or seaborn package. \n",
    "# To download from sklearn you can use the following code\n",
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "iris = datasets.load_iris() # load the data set\n",
    "iris = pd.DataFrame(data= np.c_[iris['data'], iris['target']],\n",
    "                     columns= iris['feature_names'] + ['target']\n",
    "                    ) # convert to pandas dataframe\n",
    "iris # display the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ntGI2Z93mMje"
   },
   "outputs": [],
   "source": [
    "# seaboarn contains iris data in a dataframe with the specicies well-defined \n",
    "# so we can use this as well.\n",
    "iris = sns.load_dataset('iris') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UT1VJJOHmMjf"
   },
   "source": [
    "Let's have a look at the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G94oEElymMjf",
    "outputId": "4875251b-de55-4ad7-e621-363b7e29f244"
   },
   "outputs": [],
   "source": [
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UL51XzoWmMjg"
   },
   "source": [
    "## Distribution\n",
    "Let's plot distribution of sepal length for the whole data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f43i7prSmMjg"
   },
   "source": [
    "### Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JCfERSw3mMjh",
    "outputId": "bbfe3dfc-b67a-460b-9f0a-952189a8853a"
   },
   "outputs": [],
   "source": [
    "n, bins, patches = plt.hist(x=iris['sepal_length'], bins='auto', color='#0504aa', rwidth=0.80)\n",
    "plt.xlabel('Sepal Length')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Sepal Histogram')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oeLM3tvlmMjh"
   },
   "source": [
    "    The y-axis is the grouped frequency distribution of sepal length. Try changing the bin size to experiment with class interval."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FEQAHsGWmMjh"
   },
   "source": [
    "### Density Plot\n",
    "A density plot is a smooth continuous version of a histogram estimated from data. We discussed about skewness and kurtosis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PBz8cTPymMji",
    "outputId": "4ba2833b-6a19-4307-f33d-ec0183104fb8"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris['sepal_length'], kde=True)\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Density')\n",
    "print(\"Kurtosis = \" + str(stats.kurtosis(iris['sepal_length']))) # Display kurtosis\n",
    "print(\"Skewness = \" + str(stats.skew(iris['sepal_length']))) # Display Skewness "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nzOi8yy0mMji"
   },
   "source": [
    "    Try comparing skewness and kurtosis for the remaining variables with the shape of distribution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9KEz80bxmMji"
   },
   "source": [
    "    You can plot density plot without showing the histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2t7RMpAmmMjj",
    "outputId": "acd74da9-2be7-4915-8290-37da7bb21d39"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris['sepal_length'], kind=\"kde\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Density')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N9UypBR5mMjj"
   },
   "source": [
    "    You can also plot distribution of sepal length related to each type of iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZT-uvHbMmMjj",
    "outputId": "0bf16e5e-c7e0-45fd-ccd4-64e5c4856ee2"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris, x=\"sepal_length\", hue=\"species\", kind=\"kde\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Density')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v0ENTUrOmMjk"
   },
   "source": [
    "### Cumulative Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XcRHIzb4mMjk",
    "outputId": "63840142-6e70-46b2-9708-1c17a2bc9ef0"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris, x=\"sepal_length\", kind=\"ecdf\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Proportion')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_cGsyJBYmMjk"
   },
   "source": [
    "### Box Plot\n",
    "A box plot shows the distribution of quantitative data in a way that facilitates comparisons between variables or across levels of a categorical variable. The box shows the quartiles of the dataset while the whiskers extend to show the rest of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LZvHz_lnmMjl",
    "outputId": "2b4f3abb-022b-454c-9b2f-a1b747b980ef",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ax = sns.boxplot(x=\"species\", y=\"sepal_length\", data=iris)\n",
    "_ = ax.set(ylabel='Sepal Length', xlabel='Species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n6NwcPLHmMjm"
   },
   "source": [
    "### Violin Plots\n",
    "Unlike a box plot, in which all of the plot components correspond to actual datapoints, the violin plot features a kernel density estimation of the underlying distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H4Wz-QFfmMjm",
    "outputId": "6d6542ae-386e-473c-ac97-6818f0fcc681"
   },
   "outputs": [],
   "source": [
    "ax = sns.violinplot(x=\"species\", y=\"sepal_length\", data=iris)\n",
    "_ = ax.set(ylabel='Sepal Length', xlabel='Species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jrkSIep8mMjm"
   },
   "source": [
    "You can draw split violins to compare the across the `hue` variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VFQ7DST8mMjm",
    "outputId": "ff43e28b-02fc-4f8b-93c5-5392ece6c28d"
   },
   "outputs": [],
   "source": [
    "tips = sns.load_dataset(\"tips\")\n",
    "print(tips)\n",
    "ax = sns.violinplot(x=\"day\", y=\"total_bill\", hue=\"smoker\",\n",
    "                    data=tips, palette=\"muted\", split=True)\n",
    "_ = ax.set(ylabel='Total Bill', xlabel='Date')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lH_4sjpUmMjn"
   },
   "source": [
    "### Strip Plot\n",
    "A strip plot can be used where you would like to show all observations along with some representation of the underlying distribution. For example, with a violin plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Vd2F3mvzmMjn",
    "outputId": "4851edd5-7133-4492-95f2-794d5808c6ad"
   },
   "outputs": [],
   "source": [
    "ax = sns.violinplot(x=\"species\", y=\"sepal_length\", data=iris,\n",
    "                    inner=None, color=\".8\")\n",
    "ax = sns.stripplot(x=\"species\", y=\"sepal_length\", data=iris)\n",
    "_ = ax.set(ylabel='Sepal Length', xlabel='Species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6OcOL606mMjn"
   },
   "source": [
    "### Bar Plot\n",
    "A bar plot represents an estimate of central tendency for a numeric variable with the height of each rectangle and provides some indication of the uncertainty around that estimate using error bars. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wVC54yhxmMjn",
    "outputId": "b4683950-ce55-4110-f92e-b5557145b99e",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ax = sns.barplot(x=\"day\", y=\"tip\", hue=\"sex\", data=tips)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9xqUlBjcmMjo"
   },
   "source": [
    "### Scatter Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-evnnxcSmMjo",
    "outputId": "8851308e-980c-43e4-a23b-287787892917"
   },
   "outputs": [],
   "source": [
    "ax = sns.scatterplot(data=iris, x=\"sepal_length\", y=\"petal_length\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Petal Length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aYA2AoV6mMjo"
   },
   "source": [
    "    From the scatter plot we should expect a positive strong correlation between sepal length and petal length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_t18kUU_mMjo",
    "outputId": "76b281ec-124d-4092-a34c-efbc4013cf38"
   },
   "outputs": [],
   "source": [
    "print(stats.pearsonr(x=iris[\"sepal_length\"], y=iris[\"petal_length\"])[0]) # Person Correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iIwlb2M0mMjo",
    "outputId": "df62b44d-d59b-4ac2-826f-d40f3115b38d"
   },
   "outputs": [],
   "source": [
    "print(stats.spearmanr(a=iris[\"sepal_length\"], b=iris[\"petal_length\"])[0]) # Spearman Correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iivv5oE6mMjp",
    "outputId": "44ec8b6d-3ba3-4ae8-ea78-b28848b4cd5f"
   },
   "outputs": [],
   "source": [
    "print(stats.kendalltau(x=iris[\"sepal_length\"], y=iris[\"petal_length\"])[0]) # Kendall Tau Correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DkrB2H8SmMjp"
   },
   "source": [
    "    To identify values from different species you can use the hue variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_kL9VjEtmMjp",
    "outputId": "d9a37aa0-1264-4e29-b584-791c3ed37ddc"
   },
   "outputs": [],
   "source": [
    "ax = sns.scatterplot(data=iris, x=\"sepal_length\", y=\"petal_length\", hue=\"species\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Petal Length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gEKzOfhimMjp"
   },
   "source": [
    "### Bivariate density plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aB1hIcXAmMjp",
    "outputId": "044d7612-3352-49c2-d125-3daecce46815"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris, x=\"sepal_length\", y=\"petal_length\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Petal Length')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E0Rtfh7xmMjp",
    "outputId": "e1bd458b-b4b9-47d9-aeb4-885136314231"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris, x=\"sepal_length\", y=\"petal_length\", hue = \"species\")\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Petal Length')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mlualsH3mMjq",
    "outputId": "78cc129f-5cfb-4fc2-f82f-29c349d91b0e"
   },
   "outputs": [],
   "source": [
    "ax = sns.displot(iris, x=\"sepal_length\", y=\"petal_length\", hue = \"species\", kind=\"kde\") # kernel density estimation\n",
    "_ = ax.set(xlabel='Sepal Length', ylabel='Petal Length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xh0qGJVZmMjq"
   },
   "source": [
    "### Joint Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7J9bwgG4mMjq",
    "outputId": "3dd30e17-a100-419a-952a-bb2779277dd2"
   },
   "outputs": [],
   "source": [
    "ax = sns.jointplot(data=iris, x=\"sepal_length\", y=\"petal_length\", hue=\"species\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0a_aNQcRmMjq"
   },
   "source": [
    "### Scatter Matrix or Pair Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QPK6gX3smMjq",
    "outputId": "690e9002-7a02-44fa-d0de-3fa1c656e866"
   },
   "outputs": [],
   "source": [
    "ax = sns.pairplot(iris, hue=\"species\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jqkwanXPmMjq"
   },
   "source": [
    "### Heat Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CuhQpEYqmMjq",
    "outputId": "da6d10ad-3440-429e-dd1a-b2dfc69e3e9e"
   },
   "outputs": [],
   "source": [
    "flights = sns.load_dataset(\"flights\")\n",
    "flights = flights.pivot(\"month\", \"year\", \"passengers\")\n",
    "flights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-PNRJDaimMjq",
    "outputId": "d413a5f0-946e-4c71-a879-c7e8698111b3"
   },
   "outputs": [],
   "source": [
    "ax = sns.heatmap(flights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mS1NQkARmMjr"
   },
   "source": [
    "### Line Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ss11f86fmMjr",
    "outputId": "314dcc45-d463-43e7-8914-d714d3d62f6b"
   },
   "outputs": [],
   "source": [
    "_ = sns.lineplot(data=flights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZYT8N7S9mMjr"
   },
   "source": [
    "### Pie Chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YLp7-h0HmMjr",
    "outputId": "d5d6c0fb-4b7a-41e3-d2c0-d79483779ee9"
   },
   "outputs": [],
   "source": [
    "# Pie chart, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = ['Joe Biden', 'Donald Trump', 'others']\n",
    "\n",
    "votes = [81283495, 74223755, 2704848]\n",
    "# explode = (0.1, 0, 0) \n",
    "_ = plt.pie(votes, labels=labels, autopct='%1.1f%%', startangle=90)\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Plots\n",
    "There are various python libraries which allow interactive plots e.g., altair, plotly & bokeh. Here are a few examples below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install plotly and Bokeh\n",
    "!pip install plotly bokeh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Interactive plots using plotly\n",
    "import plotly.io as pio\n",
    "import plotly.express as px\n",
    "import plotly.offline as py\n",
    "\n",
    "df = px.data.iris()\n",
    "fig = px.scatter(df, x=\"sepal_width\", y=\"sepal_length\", color=\"species\", size=\"sepal_length\")\n",
    "fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.plotting import figure, show, output_notebook\n",
    "output_notebook()\n",
    "\n",
    "p = figure()\n",
    "p.circle(df[\"sepal_width\"], df[\"sepal_length\"], fill_color=df[\"species\"], size=df[\"sepal_length\"])\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0ctrzCyPmMjr"
   },
   "source": [
    "## Exercise\n",
    "1. Load the `flights` data set using `sns.load_dataset`. What will be the appropriate visualisation technique to visualise average passengers every year along with an indication of variability. Use an appropriate function to draw the graph [15]\n",
    "2. Load the `penguins` data set using `sns.load_dataset`. Identify the variable which will have highest correlation with `body_mass_g` and why? You do not need to calculate the correlation. [25]\n",
    "3. Using the `penguins` data set above, draw a box plot using the variable `bill_depth_mm` for different species of penguins. Identify the species you will be able to identify using `bill_depth_mm`. [20]\n",
    "4. Compute [pairwise correlation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html) (you can use default method e.g., `penguins.corr()` to calculate pairwise correlation) for the `penguins` data set and plot the heatmap. Can you identify the variables which are highly correlated? What is the correlation value along the diagonal and why? Do these results match your observation in question 2? [20]\n",
    "5. Load the `fmri` data set using `sns.load_dataset`. Choose the appropriate graph to analyse `signal` against the `timepoints` for different regions. [20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H4-UyXdLmMjr"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "03-DataVisualisation.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}