preprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "preprocessing.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyM//gTVnlVIQcW3l185ILl+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/shaneahmed/StatswithPython/blob/main/preprocessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBw2CN0PUd24"
      },
      "source": [
        "# Preprocessing\n",
        "### By Fayyaz Minhas\n",
        "---\n",
        "Name: (Please enter your name and ID here prior to submission)\n",
        "\n",
        "---\n",
        "\n",
        "In this exercise, we shall analyze the effects of different types of data transformations on the probability distribution of a random variable. \n",
        "\n",
        "We begin with generating some Gaussian distributed data with a mean of 100 and a standard deviation of 10 and plot its histogram. We then use standardization (subtraction of the mean and division by the standard deviation) preprocessing and plot the histogram. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "QOdRUa8hTYUN",
        "outputId": "54205560-355a-44de-8e1c-2174e2a8b8c0"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "N = 10000\n",
        "s = np.random.normal(100,10,N)\n",
        "plt.hist(s);plt.xlabel('x');plt.ylabel('counts');plt.show()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVVElEQVR4nO3dfbAd9X3f8ffHknkwxuZJJoDkijqqGznTAFUxiduMCw0InFjYsRmYNMg2HWU60JqOZzqyPVMcE6a4tcPYqY2HFMXCY4OJH4ISaEAhbtxMy4MgAktgyjWIII1ACo9OGFMLf/vH+amcinu1R3DPg3Tfr5kzZ/e7u2e/nMvV5+7u7+xJVSFJ0t68btwNSJImn2EhSepkWEiSOhkWkqROhoUkqdP8cTcwDMccc0wtXrx43G1I0n7lnnvu+ZuqWjDdsgMyLBYvXsyGDRvG3YYk7VeSPDbTMk9DSZI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjodkJ/glrosXn3z2Pa95cr3jG3f0qs1tCOLJIckuSvJfUk2J/ntVj8xyZ1JppJ8I8lBrX5wm59qyxf3vdbHW/2hJGcNq2dJ0vSGeRrqReD0qvoF4CRgeZLTgM8AV1XVzwLPABe19S8Cnmn1q9p6JFkKnA+8A1gOfCnJvCH2LUnaw9DConr+ts2+vj0KOB34ZquvBc5t0yvaPG35GUnS6jdU1YtV9SgwBZw6rL4lSa801AvcSeYl2QjsANYDPwSerapdbZWtwAlt+gTgcYC2/Dng6P76NNv072tVkg1JNuzcuXMY/zmSNGcNNSyq6qWqOglYSO9o4B8OcV/XVNWyqlq2YMG0t2OXJL1KIxk6W1XPAt8FfhE4IsnuUVgLgW1tehuwCKAtfzPwVH99mm0kSSMwzNFQC5Ic0aYPBX4FeJBeaHygrbYSuKlNr2vztOV/XlXV6ue30VInAkuAu4bVtyTplYb5OYvjgLVt5NLrgBur6k+SPADckOR3gL8Crm3rXwt8NckU8DS9EVBU1eYkNwIPALuAi6vqpSH2LUnaw9DCoqruB06epv4I04xmqqofAx+c4bWuAK6Y7R4lSYPxdh+SpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE7DvEW5pGksXn3zWPa75cr3jGW/OjB4ZCFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkTkMLiySLknw3yQNJNif5aKt/Ksm2JBvb45y+bT6eZCrJQ0nO6qsvb7WpJKuH1bMkaXrDvJHgLuBjVXVvksOBe5Ksb8uuqqrP9q+cZClwPvAO4Hjgz5L8g7b4i8CvAFuBu5Osq6oHhti7JKnP0MKiqrYD29v0j5I8CJywl01WADdU1YvAo0mmgFPbsqmqegQgyQ1tXcNCkkZkJNcskiwGTgbubKVLktyfZE2SI1vtBODxvs22ttpM9T33sSrJhiQbdu7cOcv/BZI0tw09LJK8EfgWcGlVPQ9cDbwNOInekcfnZmM/VXVNVS2rqmULFiyYjZeUJDVD/fKjJK+nFxRfq6pvA1TVk33Lfx/4kza7DVjUt/nCVmMvdUnSCAxzNFSAa4EHq+p3++rH9a32PmBTm14HnJ/k4CQnAkuAu4C7gSVJTkxyEL2L4OuG1bck6ZWGeWTxLuA3ge8n2dhqnwAuSHISUMAW4LcAqmpzkhvpXbjeBVxcVS8BJLkEuBWYB6ypqs1D7FuStIdhjob6SyDTLLplL9tcAVwxTf2WvW0nSRouP8EtSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSeo0tLBIsijJd5M8kGRzko+2+lFJ1id5uD0f2epJ8oUkU0nuT3JK32utbOs/nGTlsHqWJE1vmEcWu4CPVdVS4DTg4iRLgdXA7VW1BLi9zQOcDSxpj1XA1dALF+Ay4J3AqcBluwNGkjQaQwuLqtpeVfe26R8BDwInACuAtW21tcC5bXoFcF313AEckeQ44CxgfVU9XVXPAOuB5cPqW5L0SiO5ZpFkMXAycCdwbFVtb4ueAI5t0ycAj/dttrXVZqrvuY9VSTYk2bBz585Z7V+S5rqhh0WSNwLfAi6tquf7l1VVATUb+6mqa6pqWVUtW7BgwWy8pCSpGWpYJHk9vaD4WlV9u5WfbKeXaM87Wn0bsKhv84WtNlNdkjQiwxwNFeBa4MGq+t2+ReuA3SOaVgI39dUvbKOiTgOea6erbgXOTHJku7B9ZqtJkkZk/hBf+13AbwLfT7Kx1T4BXAncmOQi4DHgvLbsFuAcYAp4AfgwQFU9neRy4O623qer6ukh9i1J2sPQwqKq/hLIDIvPmGb9Ai6e4bXWAGtmrztJ0r7wE9ySpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqNFBYJPlokje1T1dfm+TeJGcOuzlJ0mQY9MjiI+0mgGcCR9L7ZPaVQ+tKkjRRBg2L3Z/EPgf4alVtZuZPZ0uSDjCDhsU9SW6jFxa3Jjkc+Onw2pIkTZJB7w11EXAS8EhVvZDkaNqN/qTXYvHqm8fdgqQBDHpksb6q7q2qZwGq6ingquG1JUmaJHs9skhyCPAG4Jj2XRK7r1O8iWm+2lSSdGDqOg31W8ClwPHAPbwcFs8D/2WIfUmSJshew6KqPg98Psm/qarfG1FPkqQJM9AF7qr6vSS/BCzu36aqrhtSX5KkCTJQWCT5KvA2YCPwUisXYFhI0hww6NDZZcDS9tWnkqQ5ZtChs5uAnxlmI5KkyTXokcUxwANJ7gJe3F2sqvcOpStJ0kQZNCw+NcwmJEmTbdDRUH8x7EYkSZNr0NFQP6I3+gngIOD1wN9V1ZuG1ZgkaXIMdIG7qg6vqje1cDgU+HXgS3vbJsmaJDuSbOqrfSrJtiQb2+OcvmUfTzKV5KEkZ/XVl7faVJLV+/xfKEl6zfb5a1Wr54+AszpW/QqwfJr6VVV1UnvcApBkKXA+8I62zZeSzEsyD/gicDawFLigrStJGqFBT0O9v2/2dfQ+d/HjvW1TVd9LsnjAPlYAN1TVi8CjSaaAU9uyqap6pPVxQ1v3gQFfV5I0CwYdDfVrfdO7gC30/tF+NS5JciGwAfhYVT1D7w62d/Sts5WX72r7+B71d77K/UqSXqVBR0PN1hcdXQ1cTu9i+eXA54CPzMYLJ1kFrAJ461vfOhsvKUlqBrpmkWRhku+0C9Y7knwrycJ93VlVPVlVL1XVT4Hf5+VTTduARX2rLmy1merTvfY1VbWsqpYtWLBgX1uTJO3FoBe4/wBYR+97LY4H/rjV9kmS4/pm30fvNiK01z4/ycFJTgSWAHcBdwNLkpyY5CB6F8HX7et+JUmvzaDXLBZUVX84fCXJpXvbIMn1wLvpfcveVuAy4N1JTqJ3GmoLvS9Xoqo2J7mR3oXrXcDFVfVSe51LgFuBecCaqto8YM+SpFkyaFg8leRfAte3+QuAp/a2QVVdME352r2sfwVwxTT1W4BbBuxTkjQEg56G+ghwHvAEsB34APChIfUkSZowgx5ZfBpY2Ya5kuQo4LPM0kgmSdJkGzQs/tHuoACoqqeTnDykniQNweLVN49t31uufM/Y9q3ZMehpqNclOXL3TDuyGDRoJEn7uUH/wf8c8L+S/GGb/yDTXIyWJB2YBv0E93VJNgCnt9L7q8r7M0nSHDHwqaQWDgaEJM1B+3yLcknS3GNYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOg0tLJKsSbIjyaa+2lFJ1id5uD0f2epJ8oUkU0nuT3JK3zYr2/oPJ1k5rH4lSTMb5pHFV4Dle9RWA7dX1RLg9jYPcDawpD1WAVdDL1yAy4B3AqcCl+0OGEnS6AwtLKrqe8DTe5RXAGvb9Frg3L76ddVzB3BEkuOAs4D1VfV0VT0DrOeVASRJGrJRX7M4tqq2t+kngGPb9AnA433rbW21meqvkGRVkg1JNuzcuXN2u5akOW5sF7irqoCaxde7pqqWVdWyBQsWzNbLSpIYfVg82U4v0Z53tPo2YFHfegtbbaa6JGmERh0W64DdI5pWAjf11S9so6JOA55rp6tuBc5McmS7sH1mq0mSRmj+sF44yfXAu4FjkmylN6rpSuDGJBcBjwHntdVvAc4BpoAXgA8DVNXTSS4H7m7rfbqq9rxoLkkasqGFRVVdMMOiM6ZZt4CLZ3idNcCaWWxNkrSP/AS3JKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKnT/HHsNMkW4EfAS8CuqlqW5CjgG8BiYAtwXlU9kyTA54FzgBeAD1XVvePo+0C2ePXN425B0gQb55HFP6+qk6pqWZtfDdxeVUuA29s8wNnAkvZYBVw98k4laY6bpNNQK4C1bXotcG5f/brquQM4Islx42hQkuaqcYVFAbcluSfJqlY7tqq2t+kngGPb9AnA433bbm21/0+SVUk2JNmwc+fOYfUtSXPSWK5ZAP+0qrYleQuwPskP+hdWVSWpfXnBqroGuAZg2bJl+7StJGnvxnJkUVXb2vMO4DvAqcCTu08vtecdbfVtwKK+zRe2miRpREYeFkkOS3L47mngTGATsA5Y2VZbCdzUptcBF6bnNOC5vtNVkqQRGMdpqGOB7/RGxDIf+HpV/WmSu4Ebk1wEPAac19a/hd6w2Sl6Q2c/PPqWJWluG3lYVNUjwC9MU38KOGOaegEXj6A1SdIMJmnorCRpQhkWkqROhoUkqZNhIUnqNK4P5UmaQ8Z1o8otV75nLPs9EHlkIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6uQ35U2QcX2bmCR18chCktTJIwtJB6xxHq0faN//vd8cWSRZnuShJFNJVo+7H0maS/aLsEgyD/gicDawFLggydLxdiVJc8f+chrqVGCqqh4BSHIDsAJ4YBg780KzpNdqXP+ODOv01/4SFicAj/fNbwXe2b9CklXAqjb7t0ke2sd9HAP8zavucHgmsa9J7Ansa19MYk8wmX1NYk8wQ1/5zGt6zb8304L9JSw6VdU1wDWvdvskG6pq2Sy2NCsmsa9J7Ansa19MYk8wmX1NYk8w+r72i2sWwDZgUd/8wlaTJI3A/hIWdwNLkpyY5CDgfGDdmHuSpDljvzgNVVW7klwC3ArMA9ZU1eZZ3s2rPoU1ZJPY1yT2BPa1LyaxJ5jMviaxJxhxX6mqUe5PkrQf2l9OQ0mSxsiwkCR1mpNhkeSIJN9M8oMkDyb5xSRHJVmf5OH2fOSIe3p7ko19j+eTXDruvlpv/y7J5iSbklyf5JA22ODOdvuVb7SBB6Ps6aOtn81JLm21kb9XSdYk2ZFkU19t2j7S84X2nt2f5JQR9/XB9n79NMmyPdb/eOvroSRnjbCn/9x+D+9P8p0kR4yyp730dXnraWOS25Ic3+pj/Rn2LftYkkpyzMj6qqo59wDWAv+qTR8EHAH8J2B1q60GPjPG/uYBT9D7gMxY+6L3gchHgUPb/I3Ah9rz+a32ZeBfj7Cnnwc2AW+gN0jjz4CfHcd7BfwycAqwqa82bR/AOcB/AwKcBtw54r5+Dng78N+BZX31pcB9wMHAicAPgXkj6ulMYH6b/kzfezWSnvbS15v6pv8t8OVJ+Bm2+iJ6g30eA44ZVV9z7sgiyZvp/RCuBaiq/1NVz9K7fcjattpa4NzxdAjAGcAPq+oxJqOv+cChSebT+wd6O3A68M0x9fVz9H4ZXqiqXcBfAO9nDO9VVX0PeHqP8kx9rACuq547gCOSHDeqvqrqwaqa7s4GK4AbqurFqnoUmKJ3i51R9HRb+xkC3EHvM1Qj62kvfT3fN3sYsHsk0Fh/hs1VwL/v62kkfc25sKD3V8pO4A+S/FWS/5rkMODYqtre1nkCOHZsHfY+R3J9mx5rX1W1Dfgs8Nf0QuI54B7g2b5f8q30jkBGZRPwz5IcneQN9P6qWsTk/Axn6mO629aM8n2byaT09RF6fx3DBPSU5IokjwO/AfyHSegryQpgW1Xdt8eiofc1F8NiPr1Du6ur6mTg7+idKvh/qndcN5Yxxe3c/3uBP9xz2Tj6aufbV9AL2ePp/ZW1fJQ97KmqHqR3yuI24E+BjcBLe6wztp/hJPYx6ZJ8EtgFfG3cvexWVZ+sqkX0erpk3P20P4w+wcvBNVJzMSy2Alur6s42/0164fHk7sO29rxjTP2dDdxbVU+2+XH39S+AR6tqZ1X9BPg28C56h7m7P9Q58tuvVNW1VfWPq+qXgWeA/83436vdZupjUm9bM9a+knwI+FXgN1q4jr2nPXwN+PU2Pc6+3kbvj7b7kmxp+743yc+Moq85FxZV9QTweJK3t9IZ9G51vg5Y2WorgZvG0B7ABbx8CgrG39dfA6cleUOS8PL79V3gA+PqK8lb2vNb6V2v+Drjf692m6mPdcCFbeTKacBzfaerxmkdcH6Sg5OcCCwB7hrFjpMsp3f+/b1V9cIk9NT6WtI3uwL4QV9fY/kZVtX3q+otVbW4qhbT+8P3lPZv2vD7GtaV/El+ACcBG4D7gT8CjgSOBm4HHqY3uuaoMfR1GPAU8Oa+2iT09dv0flk2AV+lN0Ll79P75Z2id8rs4BH39D/ohdZ9wBnjeq/oBft24Cftl/eimfqgN1Lli/RG9nyfvhFJI+rrfW36ReBJ4Na+9T/Z+noIOHuEPU3RO9e+sT2+PMqe9tLXt9r/7/cDfwycMAk/wz2Wb+Hl0VBD78vbfUiSOs2501CSpH1nWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFNAJJ/kn7noFDkhzWvlfi58fdlzQoP5QnjUiS3wEOAQ6ld3+y/zjmlqSBGRbSiLQ7Ct8N/Bj4pap6qWMTaWJ4GkoanaOBNwKH0zvCkPYbHllII5JkHXADvdtMH1dVY/+OBGlQ87tXkfRaJbkQ+ElVfT3JPOB/Jjm9qv583L1Jg/DIQpLUyWsWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6vR/AdaIVj8FGeLSAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "s9K_erJET1ZG",
        "outputId": "e39fd887-e010-46b7-a46c-26135631d94b"
      },
      "source": [
        "sp = (s-np.mean(s))/np.std(s) #standardization\n",
        "plt.hist(sp);plt.xlabel('x\\'');plt.ylabel('counts');plt.show()"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAT60lEQVR4nO3df7BfdZ3f8edLfohVKChZFpPYy+ym7cStBidFdu0fFioE2DVqVwamK1GZyf4BHZhhphN0priyTNnZVeruKh22pIJlZWmVkgotZpHW2ekiBBaRgNRbxJIMkCygYhlpg+/+8f2kfhtu8vkG7vdHkudj5s49531+vRPIfd1zzuecb6oKSZL25XXTbkCSNPsMC0lSl2EhSeoyLCRJXYaFJKnr8Gk3MA7HH398zc3NTbsNSTqg3H///X9dVUsWWnZQhsXc3BxbtmyZdhuSdEBJ8oO9LfMylCSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqeugfIJb6pnbcPvUjv3E1edM7djSqzW2M4skRyW5N8m3k2xN8jutflKSbyWZT/JnSY5s9de3+fm2fG5oX5e3+mNJzhxXz5KkhY3zMtRLwGlV9U5gFbAmyanA7wHXVNUvA88DF7b1LwSeb/Vr2nokWQmcB7wdWAN8IclhY+xbkrSHsYVFDfykzR7Rvgo4Dfj3rX4D8IE2vbbN05afniStfnNVvVRV3wfmgVPG1bck6ZXGeoM7yWFJHgR2AJuB/wH8sKp2tVW2AUvb9FLgSYC2/EfAW4brC2wzfKz1SbYk2bJz585x/HEk6ZA11rCoqperahWwjMHZwN8d47Guq6rVVbV6yZIFX8cuSXqVJjJ0tqp+CNwN/CpwbJLdo7CWAdvb9HZgOUBb/jeBZ4frC2wjSZqAcY6GWpLk2Db9BuB9wKMMQuM322rrgNva9KY2T1v+jaqqVj+vjZY6CVgB3DuuviVJrzTO5yxOBG5oI5deB9xSVV9L8ghwc5LfBf4KuL6tfz3wpSTzwHMMRkBRVVuT3AI8AuwCLqqql8fYtyRpD2MLi6p6CDh5gfrjLDCaqap+Cnx4L/u6CrhqsXuUJI3G131IkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpa5yvKJe0gLkNt0/luE9cfc5UjquDg2cWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSusYWFkmWJ7k7ySNJtia5pNU/lWR7kgfb19lD21yeZD7JY0nOHKqvabX5JBvG1bMkaWHjfJHgLuCyqnogydHA/Uk2t2XXVNUfDK+cZCVwHvB24K3Anyf5223x54H3AduA+5JsqqpHxti7JGnI2MKiqp4CnmrTLyR5FFi6j03WAjdX1UvA95PMA6e0ZfNV9ThAkpvbuoaFJE3IRO5ZJJkDTga+1UoXJ3koycYkx7XaUuDJoc22tdre6nseY32SLUm27Ny5c5H/BJJ0aBt7WCR5E/AV4NKq+jFwLfBLwCoGZx6fWYzjVNV1VbW6qlYvWbJkMXYpSWrG+uFHSY5gEBQ3VdVXAarqmaHlfwJ8rc1uB5YPbb6s1dhHXZI0AeMcDRXgeuDRqvrsUP3EodU+CDzcpjcB5yV5fZKTgBXAvcB9wIokJyU5ksFN8E3j6luS9ErjPLN4D/AR4DtJHmy1TwDnJ1kFFPAE8NsAVbU1yS0MblzvAi6qqpcBklwM3AkcBmysqq1j7FuStIdxjob6CyALLLpjH9tcBVy1QP2OfW0nSRovn+CWJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHWNLSySLE9yd5JHkmxNckmrvznJ5iTfa9+Pa/Uk+cMk80keSvKuoX2ta+t/L8m6cfUsSVrYOM8sdgGXVdVK4FTgoiQrgQ3AXVW1ArirzQOcBaxoX+uBa2EQLsAVwLuBU4ArdgeMJGkyxhYWVfVUVT3Qpl8AHgWWAmuBG9pqNwAfaNNrgRtr4B7g2CQnAmcCm6vquap6HtgMrBlX35KkV5rIPYskc8DJwLeAE6rqqbboaeCENr0UeHJos22ttrf6nsdYn2RLki07d+5c1P4l6VA39rBI8ibgK8ClVfXj4WVVVUAtxnGq6rqqWl1Vq5csWbIYu5QkNWMNiyRHMAiKm6rqq638TLu8RPu+o9W3A8uHNl/WanurS5ImZJyjoQJcDzxaVZ8dWrQJ2D2iaR1w21D9gjYq6lTgR+1y1Z3AGUmOaze2z2g1SdKEHD7Gfb8H+AjwnSQPttongKuBW5JcCPwAOLctuwM4G5gHXgQ+BlBVzyW5ErivrffpqnpujH1LkvYwtrCoqr8AspfFpy+wfgEX7WVfG4GNi9edJGl/+AS3JKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqGiksklyS5Jj2dPX1SR5Icsa4m5MkzYZRzyw+3l4CeAZwHIMns68eW1eSpJkyaljsfhL7bOBLVbWVvT+dLUk6yIwaFvcn+TqDsLgzydHAz8bXliRploz6bqgLgVXA41X1YpK30F70J70Wcxtun3YLkkYw6pnF5qp6oKp+CFBVzwLXjK8tSdIs2eeZRZKjgL8BHN8+S2L3fYpjWOCjTSVJB6feZajfBi4F3grcz8/D4sfAH4+xL0nSDNlnWFTV54DPJfmnVfVHE+pJkjRjRrrBXVV/lOTXgLnhbarqxjH1JUmaISOFRZIvAb8EPAi83MoFGBaSdAgYdejsamBl++hTSdIhZtShsw8DvzjORiRJs2vUM4vjgUeS3Au8tLtYVe8fS1eSpJkyalh8apxNSJJm26ijof7ruBuRJM2uUUdDvcBg9BPAkcARwP+qqmPG1ZgkaXaMdIO7qo6uqmNaOLwB+MfAF/a1TZKNSXYkeXio9qkk25M82L7OHlp2eZL5JI8lOXOovqbV5pNs2O8/oSTpNdvvj1Wtgf8AnNlZ9YvAmgXq11TVqvZ1B0CSlcB5wNvbNl9IcliSw4DPA2cBK4Hz27qSpAka9TLUh4ZmX8fguYuf7mubqvpmkrkR+1gL3FxVLwHfTzIPnNKWzVfV462Pm9u6j4y4X0nSIhh1NNRvDE3vAp5g8EP71bg4yQXAFuCyqnqewRts7xlaZxs/f6vtk3vU3/0qjytJepVGHQ21WB90dC1wJYOb5VcCnwE+vhg7TrIeWA/wtre9bTF2KUlqRrpnkWRZklvbDesdSb6SZNn+Hqyqnqmql6vqZ8Cf8PNLTduB5UOrLmu1vdUX2vd1VbW6qlYvWbJkf1uTJO3DqDe4/w2wicHnWrwV+I+ttl+SnDg0+0EGrxGh7fu8JK9PchKwArgXuA9YkeSkJEcyuAm+aX+PK0l6bUa9Z7GkqobD4YtJLt3XBkm+DLyXwafsbQOuAN6bZBWDy1BPMPhwJapqa5JbGNy43gVcVFUvt/1cDNwJHAZsrKqtI/YsSVoko4bFs0l+C/hymz8feHZfG1TV+QuUr9/H+lcBVy1QvwO4Y8Q+JUljMOplqI8D5wJPA08Bvwl8dEw9SZJmzKhnFp8G1rVhriR5M/AHLNJIJknSbBs1LN6xOygAquq5JCePqSdJYzC34fapHfuJq8+Z2rG1OEa9DPW6JMftnmlnFqMGjSTpADfqD/zPAH+Z5N+1+Q+zwM1oSdLBadQnuG9MsgU4rZU+VFW+n0mSDhEjX0pq4WBASNIhaL9fUS5JOvQYFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqSusYVFko1JdiR5eKj25iSbk3yvfT+u1ZPkD5PMJ3koybuGtlnX1v9eknXj6leStHfjPLP4IrBmj9oG4K6qWgHc1eYBzgJWtK/1wLUwCBfgCuDdwCnAFbsDRpI0OWMLi6r6JvDcHuW1wA1t+gbgA0P1G2vgHuDYJCcCZwKbq+q5qnoe2MwrA0iSNGaTvmdxQlU91aafBk5o00uBJ4fW29Zqe6u/QpL1SbYk2bJz587F7VqSDnFTu8FdVQXUIu7vuqpaXVWrlyxZsli7lSQx+bB4pl1eon3f0erbgeVD6y1rtb3VJUkTNOmw2ATsHtG0DrhtqH5BGxV1KvCjdrnqTuCMJMe1G9tntJokaYIOH9eOk3wZeC9wfJJtDEY1XQ3ckuRC4AfAuW31O4CzgXngReBjAFX1XJIrgfvaep+uqj1vmkuSxmxsYVFV5+9l0ekLrFvARXvZz0Zg4yK2JknaTz7BLUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqOnwaB03yBPAC8DKwq6pWJ3kz8GfAHPAEcG5VPZ8kwOeAs4EXgY9W1QPT6PtgNrfh9mm3IGmGTfPM4h9W1aqqWt3mNwB3VdUK4K42D3AWsKJ9rQeunXinknSIm6XLUGuBG9r0DcAHhuo31sA9wLFJTpxGg5J0qJpWWBTw9ST3J1nfaidU1VNt+mnghDa9FHhyaNttrfb/SbI+yZYkW3bu3DmuviXpkDSVexbAP6iq7Ul+Adic5LvDC6uqktT+7LCqrgOuA1i9evV+bStJ2repnFlU1fb2fQdwK3AK8Mzuy0vt+462+nZg+dDmy1pNkjQhEw+LJG9McvTuaeAM4GFgE7CurbYOuK1NbwIuyMCpwI+GLldJkiZgGpehTgBuHYyI5XDgT6vqPye5D7glyYXAD4Bz2/p3MBg2O89g6OzHJt+yJB3aJh4WVfU48M4F6s8Cpy9QL+CiCbQmSdqLWRo6K0maUYaFJKnLsJAkdRkWkqSuaT2UJ+kQMq0XVT5x9TlTOe7ByDMLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpeflDdDpvVpYpLU45mFJKnLMwtJB61pnq0fbJ//fcCcWSRZk+SxJPNJNky7H0k6lBwQYZHkMODzwFnASuD8JCun25UkHToOlMtQpwDzVfU4QJKbgbXAI+M4mDeaJb1W0/o5Mq7LXwdKWCwFnhya3wa8e3iFJOuB9W32J0keexXHOR7461fV4fjMYk8wm33NYk8wm33NYk8wm30dUD3l917Tfv/W3hYcKGHRVVXXAde9ln0k2VJVqxeppUUxiz3BbPY1iz3BbPY1iz3BbPZlTwMHxD0LYDuwfGh+WatJkibgQAmL+4AVSU5KciRwHrBpyj1J0iHjgLgMVVW7klwM3AkcBmysqq1jONRruow1JrPYE8xmX7PYE8xmX7PYE8xmX/YEpKomfUxJ0gHmQLkMJUmaIsNCktRlWOxFksuSVJLjZ6CXK5M8lOTBJF9P8tZp9wSQ5PeTfLf1dmuSY2egpw8n2ZrkZ0mmOtxxFl9Rk2Rjkh1JHp52L7slWZ7k7iSPtP92l0y7J4AkRyW5N8m3W1+/M+2edktyWJK/SvK1SR3TsFhAkuXAGcD/nHYvze9X1TuqahXwNeCfT7uhZjPwK1X1DuC/A5dPuR+Ah4EPAd+cZhMz/IqaLwJrpt3EHnYBl1XVSuBU4KIZ+bt6CTitqt4JrALWJDl1yj3tdgnw6CQPaFgs7BrgnwEzcfe/qn48NPtGZqevr1fVrjZ7D4PnX6aqqh6tqlfz9P5i+3+vqKmq/w3sfkXNVFXVN4Hnpt3HsKp6qqoeaNMvMPghuHS6XUEN/KTNHtG+pv5vL8ky4BzgX0/yuIbFHpKsBbZX1ben3cuwJFcleRL4J8zOmcWwjwP/adpNzJCFXlEz9R+Asy7JHHAy8K3pdjLQLvc8COwANlfVLPT1Lxn8MvuzSR70gHjOYrEl+XPgFxdY9EngEwwuQU3Uvnqqqtuq6pPAJ5NcDlwMXDELfbV1PsngUsJNs9KTDjxJ3gR8Bbh0j7Ppqamql4FV7X7crUl+paqmdr8nya8DO6rq/iTvneSxD8mwqKp/tFA9yd8DTgK+nQQGl1UeSHJKVT09jZ4WcBNwBxMKi15fST4K/Dpwek3ooZ39+LuaJl9Rsx+SHMEgKG6qqq9Ou589VdUPk9zN4H7PNAcHvAd4f5KzgaOAY5L826r6rXEf2MtQQ6rqO1X1C1U1V1VzDC4dvGvcQdGTZMXQ7Frgu9PqZViSNQxOh99fVS9Ou58Z4ytqRpTBb2bXA49W1Wen3c9uSZbsHuGX5A3A+5jyv72quryqlrWfT+cB35hEUIBhcaC4OsnDSR5icIlsJoYWAn8MHA1sbsN6/9W0G0rywSTbgF8Fbk9y5zT6aDf+d7+i5lHgljG9oma/JPky8JfA30myLcmF0+6JwW/LHwFOa/8fPdh+c562E4G727+7+xjcs5jYUNVZ4+s+JEldnllIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJCmIMlckv8y7T6kURkWkqQuw0IasyR/v33mx1FJ3phkK4NXgczU21+lffGhPGkCkvwug3f5vAHYVlX/YsotSfvFsJAmoL0f6j7gp8CvtbeZSgcML0NJk/EW4E0M3qV11JR7kfabZxbSBCTZxODT8k4CTqyqi6fckrRfDsnPs5AmKckFwP+pqj9tn83935KcVlXfmHZv0qg8s5AkdXnPQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdf1fjcubwMolR4wAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "toR2cZsTVqKx"
      },
      "source": [
        "## Questions\n",
        "1. What changes do you observe in the histogram after standardization? [5]\n",
        "2. What is the mean of the data after standardization? [5]\n",
        "3. What is the variance of the data after standardization? [5]\n",
        "4. Preprocess the data in accordance with the following preprocessing schemes and report your observations on and reasoning for the differences between the histograms before and after the preprocessing: a). Log transform, b.)Binarization, c). Discretization into 3 levels, d). Min-Max Scaling, e). Rank Transformation [20]\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KXLSKA8ca7kQ"
      },
      "source": [
        "#Write your code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5i-zC4e1aneb"
      },
      "source": [
        "## Introducing sklearn\n",
        "\n",
        "Scikit-learn is a commonly used machine learning package in Python. It is pretty simple to use and has some basic data sets for us to get started. We shall be using sklearn for studying multivariate analysis, regression, clustering and linar models. \n",
        "\n",
        "We begin by loading the iris dataset (https://en.wikipedia.org/wiki/Iris_flower_data_set) which consists of 150 examples of flowers each with 4 features (sepal length, sepal width, petal length, petal width) and a corresponding species identifier (0: setosa,1: versicolor,2: virginica). The code for loading the data is given below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cech9C8aaCYx",
        "outputId": "39971033-b68a-4216-a30a-d08e7274300a"
      },
      "source": [
        "from sklearn import datasets\n",
        "iris = datasets.load_iris()\n",
        "X = iris.data\n",
        "Y = iris.target\n",
        "print('IRIS data dimensions',X.shape)\n",
        "print('IRIS targets',Y)\n",
        "print('IRIS target names',iris.target_names)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "IRIS data dimensions (150, 4)\n",
            "IRIS targets [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
            " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
            " 2 2]\n",
            "IRIS target names ['setosa' 'versicolor' 'virginica']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oNVkIK4mcuGy"
      },
      "source": [
        "## Questions:\n",
        "\n",
        "5. Generate the scatter matrix of the Iris dataset. [15]\n",
        "6. State your general observations about the general associative patterns between different features. [10]\n",
        "7. State your general observations about the differences in terms of feature values for different species of these flowers. [10]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ya3iwvgOb12G"
      },
      "source": [
        "# your code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LRIlkj4Teo1F"
      },
      "source": [
        "## Preprocessing and Data Imputation with sklearn\n",
        "\n",
        "Sklearn provides data scaling and preprocessing support for multivariable data. Familiarize yourself with the builtin standardscaler (https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html). Below is a simple example. If you want to go in more depth, you may also want to look at: https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5FYp9T_kfU3N",
        "outputId": "510948fc-1af3-473c-faea-80df5d0eaee4"
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "data = [[0, 0], [0, 0], [1, 1], [1, 1]]\n",
        "scaler = StandardScaler()\n",
        "scaler.fit(data)\n",
        "tdata = scaler.transform(data)\n",
        "print('Original Data\\n',data)\n",
        "print('Transformed Data\\n',tdata)\n",
        "print('Mean of each feature',scaler.mean_)\n",
        "print('Variance of each feature',scaler.var_)\n"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original Data\n",
            " [[0, 0], [0, 0], [1, 1], [1, 1]]\n",
            "Transformed Data\n",
            " [[-1. -1.]\n",
            " [-1. -1.]\n",
            " [ 1.  1.]\n",
            " [ 1.  1.]]\n",
            "Mean of each feature [0.5 0.5]\n",
            "Variance of each feature [0.25 0.25]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6Og7R1WgW4x"
      },
      "source": [
        "Sklearn also provides support for data imputation (see: https://scikit-learn.org/stable/modules/impute.html) with a minimalistic example on data imputation below. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "08CzHsIlfcpX",
        "outputId": "a9560aa1-deb7-46b8-eb4d-8a63a4fac4e7"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.impute import SimpleImputer\n",
        "imp = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
        "data = np.array([[1, 2], [np.nan, 3], [7, 6]])\n",
        "imp.fit(data)\n",
        "idata = imp.transform(data)\n",
        "print('Imputed Data \\n',idata)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Imputed Data \n",
            " [[1. 2.]\n",
            " [4. 3.]\n",
            " [7. 6.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HPWqLa0chGMj"
      },
      "source": [
        "## Questions:\n",
        "\n",
        "8. Use the standard scaler and minmax scaler in sklearn to transform the iris data and verify (through scatter plots) the impact of the transformation. [30]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0_CICEc6gxyB"
      },
      "source": [
        "#your code here"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}