Course_2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a href=\"https://colab.research.google.com/github/neworldemancer/DSF5/blob/master/Course_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Introduction to machine learning & Data Analysis\n",
        "\n",
        "Basic introduction on how to perform typical machine learning tasks with Python.\n",
        "\n",
        "Prepared by Mykhailo Vladymyrov & Aris Marcolongo,\n",
        "Data Science Lab, University Of Bern, 2023\n",
        "\n",
        "This work is licensed under <a href=\"https://creativecommons.org/share-your-work/public-domain/cc0/\">CC0</a>.\n",
        "\n",
        "# Part 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b-fYVr8NTLeK"
      },
      "source": [
        "# Load libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T14:05:44.613282Z",
          "start_time": "2022-09-27T14:05:43.538999Z"
        },
        "id": "hVJn0ilgOS8F",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "from sklearn import tree\n",
        "from sklearn import ensemble\n",
        "\n",
        "from sklearn.datasets import make_blobs\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import metrics\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "from matplotlib import  pyplot as plt\n",
        "import seaborn as sns\n",
        "#sns.set()\n",
        "\n",
        "from time import time as timer\n",
        "from imageio import imread\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import os\n",
        "\n",
        "from sklearn.manifold import TSNE\n",
        "\n",
        "import tensorflow as tf\n",
        "import tarfile\n",
        "\n",
        "%matplotlib inline\n",
        "from matplotlib import animation\n",
        "from IPython.display import HTML"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zV3YwuW4L6Yy",
        "outputId": "b067b97a-6cb6-4c47-f07b-83586d5844a8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting umap-learn\n",
            "  Downloading umap-learn-0.5.4.tar.gz (90 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.8/90.8 kB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from umap-learn) (1.23.5)\n",
            "Requirement already satisfied: scipy>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from umap-learn) (1.11.2)\n",
            "Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.10/dist-packages (from umap-learn) (1.2.2)\n",
            "Requirement already satisfied: numba>=0.51.2 in /usr/local/lib/python3.10/dist-packages (from umap-learn) (0.56.4)\n",
            "Collecting pynndescent>=0.5 (from umap-learn)\n",
            "  Downloading pynndescent-0.5.10.tar.gz (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from umap-learn) (4.66.1)\n",
            "Requirement already satisfied: tbb>=2019.0 in /usr/local/lib/python3.10/dist-packages (from umap-learn) (2021.10.0)\n",
            "Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba>=0.51.2->umap-learn) (0.39.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from numba>=0.51.2->umap-learn) (67.7.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.10/dist-packages (from pynndescent>=0.5->umap-learn) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.22->umap-learn) (3.2.0)\n",
            "Building wheels for collected packages: umap-learn, pynndescent\n",
            "  Building wheel for umap-learn (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for umap-learn: filename=umap_learn-0.5.4-py3-none-any.whl size=86770 sha256=a56264314d501e5153477fdecc82b30fdf7d3cd1229824ed01ac933ecaef0e6c\n",
            "  Stored in directory: /root/.cache/pip/wheels/fb/66/29/199acf5784d0f7b8add6d466175ab45506c96e386ed5dd0633\n",
            "  Building wheel for pynndescent (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pynndescent: filename=pynndescent-0.5.10-py3-none-any.whl size=55615 sha256=45c2fb85ca5863c8a0cbc4d7d6d2b889905bd9ea28c5ad503c3716b3963cc82e\n",
            "  Stored in directory: /root/.cache/pip/wheels/4a/38/5d/f60a40a66a9512b7e5e83517ebc2d1b42d857be97d135f1096\n",
            "Successfully built umap-learn pynndescent\n",
            "Installing collected packages: pynndescent, umap-learn\n",
            "Successfully installed pynndescent-0.5.10 umap-learn-0.5.4\n"
          ]
        }
      ],
      "source": [
        "pip install umap-learn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T14:05:49.051625Z",
          "start_time": "2022-09-27T14:05:46.022444Z"
        },
        "id": "oes8tZkKXS1S"
      },
      "outputs": [],
      "source": [
        "import umap"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T14:05:50.662474Z",
          "start_time": "2022-09-27T14:05:50.657850Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7xQIE54NmqJs",
        "outputId": "b816a504-e059-401f-8b4f-176be800ad71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://github.com/neworldemancer/DSF5/raw/master/colab_material.tgz\n",
            "261349/261349 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "if not os.path.exists('data'):\n",
        "    path = os.path.abspath('.')+'/colab_material.tgz'\n",
        "    tf.keras.utils.get_file(path, 'https://github.com/neworldemancer/DSF5/raw/master/colab_material.tgz')\n",
        "    tar = tarfile.open(path, \"r:gz\")\n",
        "    tar.extractall()\n",
        "    tar.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:22:00.451788Z",
          "start_time": "2022-09-26T19:22:00.401516Z"
        },
        "id": "atch-QcelhRy"
      },
      "outputs": [],
      "source": [
        "from utils.routines import *"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cBxkWZaLCUcR"
      },
      "source": [
        "# Datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MssLYOiCUcX"
      },
      "source": [
        "In this course we will use several synthetic and real-world datasets to illustrate the behavior of the models and exercise our skills."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UQgU5I-lEll"
      },
      "source": [
        "## 1. Synthetic linear"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jGfWOWRjlWPa"
      },
      "outputs": [],
      "source": [
        "def get_linear(n_d=1, n_points=10, w=None, b=None, sigma=5):\n",
        "  x = np.random.uniform(0, 10, size=(n_points, n_d))\n",
        "\n",
        "  w = w or np.random.uniform(0.1, 10, n_d)\n",
        "  b = b or np.random.uniform(-10, 10)\n",
        "  y = np.dot(x, w) + b + np.random.normal(0, sigma, size=n_points)\n",
        "\n",
        "  print('true slopes: w =', w, ';  b =', b)\n",
        "\n",
        "  return x, y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5RLYxGy_nBZG"
      },
      "outputs": [],
      "source": [
        "x, y = get_linear(n_d=1, sigma=0)\n",
        "plt.plot(x[:, 0], y, '*')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "10ODDOp4nX4S"
      },
      "outputs": [],
      "source": [
        "n_d = 2\n",
        "x, y = get_linear(n_d=n_d, n_points=100)\n",
        "\n",
        "fig = plt.figure(figsize=(8,8))\n",
        "ax = fig.add_subplot(111, projection='3d')\n",
        "ax.scatter(x[:,0], x[:,1], y, marker='x', color='b',s=40)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-2uMw5C4CUcj"
      },
      "source": [
        "## 2. House prices"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qi_Q6YTNCUcj"
      },
      "source": [
        "Subset of the Ames Houses dataset: http://jse.amstat.org/v19n3/decock.pdf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Uj2LFlahCUcj"
      },
      "outputs": [],
      "source": [
        "def house_prices_dataset(return_df=False, price_max=400000, area_max=40000):\n",
        "  path = 'data/AmesHousing.csv'\n",
        "\n",
        "  df = pd.read_csv(path, na_values=('NaN', ''), keep_default_na=False)\n",
        "\n",
        "  rename_dict = {k:k.replace(' ', '').replace('/', '') for k in df.keys()}\n",
        "  df.rename(columns=rename_dict, inplace=True)\n",
        "\n",
        "  useful_fields = ['LotArea',\n",
        "                  'Utilities', 'OverallQual', 'OverallCond',\n",
        "                  'YearBuilt', 'YearRemodAdd', 'ExterQual', 'ExterCond',\n",
        "                  'HeatingQC', 'CentralAir', 'Electrical',\n",
        "                  '1stFlrSF', '2ndFlrSF','GrLivArea',\n",
        "                  'FullBath', 'HalfBath',\n",
        "                  'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual', 'TotRmsAbvGrd',\n",
        "                  'Functional','PoolArea',\n",
        "                  'YrSold', 'MoSold'\n",
        "                  ]\n",
        "  target_field = 'SalePrice'\n",
        "\n",
        "  df.dropna(axis=0, subset=useful_fields+[target_field], inplace=True)\n",
        "\n",
        "  cleanup_nums = {'Street':      {'Grvl': 0, 'Pave': 1},\n",
        "                  'LotFrontage': {'NA':0},\n",
        "                  'Alley':       {'NA':0, 'Grvl': 1, 'Pave': 2},\n",
        "                  'LotShape':    {'IR3':0, 'IR2': 1, 'IR1': 2, 'Reg':3},\n",
        "                  'Utilities':   {'ELO':0, 'NoSeWa': 1, 'NoSewr': 2, 'AllPub': 3},\n",
        "                  'LandSlope':   {'Sev':0, 'Mod': 1, 'Gtl': 3},\n",
        "                  'ExterQual':   {'Po':0, 'Fa': 1, 'TA': 2, 'Gd': 3, 'Ex':4},\n",
        "                  'ExterCond':   {'Po':0, 'Fa': 1, 'TA': 2, 'Gd': 3, 'Ex':4},\n",
        "                  'BsmtQual':    {'NA':0, 'Po':1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex':5},\n",
        "                  'BsmtCond':    {'NA':0, 'Po':1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex':5},\n",
        "                  'BsmtExposure':{'NA':0, 'No':1, 'Mn': 2, 'Av': 3, 'Gd': 4},\n",
        "                  'BsmtFinType1':{'NA':0, 'Unf':1, 'LwQ': 2, 'Rec': 3, 'BLQ': 4, 'ALQ':5, 'GLQ':6},\n",
        "                  'BsmtFinType2':{'NA':0, 'Unf':1, 'LwQ': 2, 'Rec': 3, 'BLQ': 4, 'ALQ':5, 'GLQ':6},\n",
        "                  'HeatingQC':   {'Po':0, 'Fa': 1, 'TA': 2, 'Gd': 3, 'Ex':4},\n",
        "                  'CentralAir':  {'N':0, 'Y': 1},\n",
        "                  'Electrical':  {'':0, 'NA':0, 'Mix':1, 'FuseP':2, 'FuseF': 3, 'FuseA': 4, 'SBrkr': 5},\n",
        "                  'KitchenQual': {'Po':0, 'Fa': 1, 'TA': 2, 'Gd': 3, 'Ex':4},\n",
        "                  'Functional':  {'Sal':0, 'Sev':1, 'Maj2': 2, 'Maj1': 3, 'Mod': 4, 'Min2':5, 'Min1':6, 'Typ':7},\n",
        "                  'FireplaceQu': {'NA':0, 'Po':1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex':5},\n",
        "                  'PoolQC':      {'NA':0, 'Fa': 1, 'TA': 2, 'Gd': 3, 'Ex':4},\n",
        "                  'Fence':       {'NA':0, 'MnWw': 1, 'GdWo': 2, 'MnPrv': 3, 'GdPrv':4},\n",
        "                  }\n",
        "\n",
        "  df_X = df[useful_fields].copy()\n",
        "  df_X.replace(cleanup_nums, inplace=True)  # convert continous categorial variables to numerical\n",
        "  df_Y = df[target_field].copy()\n",
        "\n",
        "  x = df_X.to_numpy().astype(np.float32)\n",
        "  y = df_Y.to_numpy().astype(np.float32)\n",
        "\n",
        "  if price_max>0:\n",
        "    idxs = y<price_max\n",
        "    x = x[idxs]\n",
        "    y = y[idxs]\n",
        "\n",
        "  if area_max>0:\n",
        "    idxs = x[:,0]<area_max\n",
        "    x = x[idxs]\n",
        "    y = y[idxs]\n",
        "\n",
        "  return (x, y, df) if return_df else (x,y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-jcX7oYDFwzQ"
      },
      "outputs": [],
      "source": [
        "def house_prices_dataset_normed():\n",
        "    x, y = house_prices_dataset(return_df=False, price_max=-1, area_max=-1)\n",
        "\n",
        "    scaler=StandardScaler()\n",
        "    features_scaled=scaler.fit_transform(x)\n",
        "\n",
        "    return features_scaled"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YqWU0eHts1RM"
      },
      "outputs": [],
      "source": [
        "x, y, df = house_prices_dataset(return_df=True)\n",
        "print(x.shape, y.shape)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91nj7znzMEpA"
      },
      "outputs": [],
      "source": [
        "plt.plot(x[:, 0], y, '.')\n",
        "plt.xlabel('area, sq.ft')\n",
        "plt.ylabel('price, $');\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q7CNxkPdNB4L"
      },
      "source": [
        "## 3. Blobs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j8wXhleONKgZ"
      },
      "outputs": [],
      "source": [
        "x, y = make_blobs(n_samples=1000, centers=[[0,0], [5,5], [10, 0]])\n",
        "colors = \"ygr\"\n",
        "for i, color in enumerate(colors):\n",
        "    idx = y == i\n",
        "    plt.scatter(x[idx, 0], x[idx, 1], c=color, edgecolor='gray', s=25)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8S1jwU4cXQX4"
      },
      "source": [
        "## 4. MNIST"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2u82UQ5XQX4"
      },
      "source": [
        "The MNIST database of handwritten digits has a training set of 60,000 examples, and a test set of 10,000 examples. The digits have been size-normalized and centered in a fixed-size image.\n",
        "It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting (taken from http://yann.lecun.com/exdb/mnist/). Each example is a 28x28 grayscale image and the dataset can be readily downloaded from Tensorflow."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JaNaGGOkXQX5"
      },
      "outputs": [],
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlUY5gl8XQX7"
      },
      "source": [
        "Let's check few samples:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qtYtGEDdXQX8"
      },
      "outputs": [],
      "source": [
        "n = 3\n",
        "fig, ax = plt.subplots(n, n, figsize=(2*n, 2*n))\n",
        "ax = [ax_xy for ax_y in ax for ax_xy in ax_y]\n",
        "for axi, im_idx in zip(ax, np.random.choice(len(train_images), n**2)):\n",
        "  im = train_images[im_idx]\n",
        "  im_class = train_labels[im_idx]\n",
        "  axi.imshow(im, cmap='gray')\n",
        "  axi.text(1, 4, f'{im_class}', color='r', size=16)\n",
        "  axi.grid(False)\n",
        "plt.tight_layout(0,0,0)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5MXcNBOBCUcy"
      },
      "source": [
        "## 5. Fashion MNIST"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rANzstLwCUcy"
      },
      "source": [
        "`Fashion-MNIST` is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. (from https://github.com/zalandoresearch/fashion-mnist)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Woc5ld-HCUcz"
      },
      "outputs": [],
      "source": [
        "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FH2G-FAHCUc1"
      },
      "source": [
        "Let's check few samples:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "90EhR8nmCUc1"
      },
      "outputs": [],
      "source": [
        "n = 3\n",
        "fig, ax = plt.subplots(n, n, figsize=(2*n, 2*n))\n",
        "ax = [ax_xy for ax_y in ax for ax_xy in ax_y]\n",
        "for axi, im_idx in zip(ax, np.random.choice(len(train_images), n**2)):\n",
        "  im = train_images[im_idx]\n",
        "  im_class = train_labels[im_idx]\n",
        "  axi.imshow(im, cmap='gray')\n",
        "  axi.text(1, 4, f'{im_class}', color='r', size=16)\n",
        "  axi.grid(False)\n",
        "plt.tight_layout(0,0,0)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xeaJnq9hCUc3"
      },
      "source": [
        "Each of the training and test examples is assigned to one of the following labels:\n",
        "\n",
        "| Label | Description |\n",
        "| --- | --- |\n",
        "| 0 | T-shirt/top |\n",
        "| 1 | Trouser |\n",
        "| 2 | Pullover |\n",
        "| 3 | Dress |\n",
        "| 4 | Coat |\n",
        "| 5 | Sandal |\n",
        "| 6 | Shirt |\n",
        "| 7 | Sneaker |\n",
        "| 8 | Bag |\n",
        "| 9 | Ankle boot |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_wxOrdWko8W"
      },
      "source": [
        "In this course we will use several synthetic and real-world datasets to illustrate the behavior of the models and exercise our skills."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F9ARiWSS91kO"
      },
      "source": [
        "# 1. Unsupervised Learning Techniques\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "diTExyW-L6Y0"
      },
      "source": [
        "Unsupervised learning technique different from supervised ones from the fact that data are not labelled (no supervision).\n",
        "\n",
        "We do not aim at fitting a mapping from $X$ to $Y$, but to understand pattern in the data cloud $X$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afROKfT591kS"
      },
      "source": [
        "## 1. Principal Component Analysis (PCA)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0iypmtDL6Y1"
      },
      "source": [
        "Let's start with the example that we will use to make the theory more concrete. We will take a dataset from from kaggle https://www.kaggle.com/datasets/miroslavsabo/young-people-survey?resource=download (already downloaded for you in the folder `data`)\n",
        "\n",
        "The datasets consists of the results of a survey about the music preferences of several students, arriving at the following dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T14:09:42.322377Z",
          "start_time": "2022-09-27T14:09:42.283713Z"
        },
        "id": "J7ep6OIJL6Y1"
      },
      "outputs": [],
      "source": [
        "data=pd.read_csv(\"data/responses.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 687
        },
        "id": "vuOinT9cL6Y4",
        "outputId": "364e82a5-b5e0-45dc-fc67-0e66c36285a0"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-6c180c05-a58c-4b96-a62a-26441c5e182a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Music</th>\n",
              "      <th>Slow songs or fast songs</th>\n",
              "      <th>Dance</th>\n",
              "      <th>Folk</th>\n",
              "      <th>Country</th>\n",
              "      <th>Classical music</th>\n",
              "      <th>Musical</th>\n",
              "      <th>Pop</th>\n",
              "      <th>Rock</th>\n",
              "      <th>Metal or Hardrock</th>\n",
              "      <th>...</th>\n",
              "      <th>Age</th>\n",
              "      <th>Height</th>\n",
              "      <th>Weight</th>\n",
              "      <th>Number of siblings</th>\n",
              "      <th>Gender</th>\n",
              "      <th>Left - right handed</th>\n",
              "      <th>Education</th>\n",
              "      <th>Only child</th>\n",
              "      <th>Village - town</th>\n",
              "      <th>House - block of flats</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>20.0</td>\n",
              "      <td>163.0</td>\n",
              "      <td>48.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>college/bachelor degree</td>\n",
              "      <td>no</td>\n",
              "      <td>village</td>\n",
              "      <td>block of flats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>...</td>\n",
              "      <td>19.0</td>\n",
              "      <td>163.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>college/bachelor degree</td>\n",
              "      <td>no</td>\n",
              "      <td>city</td>\n",
              "      <td>block of flats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>...</td>\n",
              "      <td>20.0</td>\n",
              "      <td>176.0</td>\n",
              "      <td>67.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>secondary school</td>\n",
              "      <td>no</td>\n",
              "      <td>city</td>\n",
              "      <td>block of flats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>22.0</td>\n",
              "      <td>172.0</td>\n",
              "      <td>59.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>college/bachelor degree</td>\n",
              "      <td>yes</td>\n",
              "      <td>city</td>\n",
              "      <td>house/bungalow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>20.0</td>\n",
              "      <td>170.0</td>\n",
              "      <td>59.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>secondary school</td>\n",
              "      <td>no</td>\n",
              "      <td>village</td>\n",
              "      <td>house/bungalow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1005</th>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>...</td>\n",
              "      <td>20.0</td>\n",
              "      <td>164.0</td>\n",
              "      <td>57.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>secondary school</td>\n",
              "      <td>no</td>\n",
              "      <td>city</td>\n",
              "      <td>house/bungalow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1006</th>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>27.0</td>\n",
              "      <td>183.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>male</td>\n",
              "      <td>left handed</td>\n",
              "      <td>masters degree</td>\n",
              "      <td>no</td>\n",
              "      <td>village</td>\n",
              "      <td>house/bungalow</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1007</th>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>18.0</td>\n",
              "      <td>173.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>secondary school</td>\n",
              "      <td>yes</td>\n",
              "      <td>city</td>\n",
              "      <td>block of flats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1008</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>25.0</td>\n",
              "      <td>173.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>female</td>\n",
              "      <td>right handed</td>\n",
              "      <td>college/bachelor degree</td>\n",
              "      <td>no</td>\n",
              "      <td>city</td>\n",
              "      <td>block of flats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1009</th>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>...</td>\n",
              "      <td>21.0</td>\n",
              "      <td>185.0</td>\n",
              "      <td>72.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>male</td>\n",
              "      <td>right handed</td>\n",
              "      <td>secondary school</td>\n",
              "      <td>no</td>\n",
              "      <td>village</td>\n",
              "      <td>house/bungalow</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1010 rows × 150 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6c180c05-a58c-4b96-a62a-26441c5e182a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6c180c05-a58c-4b96-a62a-26441c5e182a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6c180c05-a58c-4b96-a62a-26441c5e182a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a99129d1-4afb-452c-a182-b22fcd7eb790\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a99129d1-4afb-452c-a182-b22fcd7eb790')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a99129d1-4afb-452c-a182-b22fcd7eb790 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      Music  Slow songs or fast songs  Dance  Folk  Country  Classical music  \\\n",
              "0       5.0                       3.0    2.0   1.0      2.0              2.0   \n",
              "1       4.0                       4.0    2.0   1.0      1.0              1.0   \n",
              "2       5.0                       5.0    2.0   2.0      3.0              4.0   \n",
              "3       5.0                       3.0    2.0   1.0      1.0              1.0   \n",
              "4       5.0                       3.0    4.0   3.0      2.0              4.0   \n",
              "...     ...                       ...    ...   ...      ...              ...   \n",
              "1005    5.0                       2.0    5.0   2.0      2.0              5.0   \n",
              "1006    4.0                       4.0    5.0   1.0      3.0              4.0   \n",
              "1007    4.0                       3.0    1.0   1.0      2.0              2.0   \n",
              "1008    5.0                       3.0    3.0   3.0      1.0              3.0   \n",
              "1009    5.0                       5.0    4.0   3.0      2.0              3.0   \n",
              "\n",
              "      Musical  Pop  Rock  Metal or Hardrock  ...   Age  Height  Weight  \\\n",
              "0         1.0  5.0   5.0                1.0  ...  20.0   163.0    48.0   \n",
              "1         2.0  3.0   5.0                4.0  ...  19.0   163.0    58.0   \n",
              "2         5.0  3.0   5.0                3.0  ...  20.0   176.0    67.0   \n",
              "3         1.0  2.0   2.0                1.0  ...  22.0   172.0    59.0   \n",
              "4         3.0  5.0   3.0                1.0  ...  20.0   170.0    59.0   \n",
              "...       ...  ...   ...                ...  ...   ...     ...     ...   \n",
              "1005      4.0  4.0   4.0                3.0  ...  20.0   164.0    57.0   \n",
              "1006      1.0  4.0   1.0                1.0  ...  27.0   183.0    80.0   \n",
              "1007      2.0  3.0   4.0                1.0  ...  18.0   173.0    75.0   \n",
              "1008      1.0  3.0   4.0                1.0  ...  25.0   173.0    58.0   \n",
              "1009      3.0  4.0   1.0                1.0  ...  21.0   185.0    72.0   \n",
              "\n",
              "      Number of siblings  Gender  Left - right handed  \\\n",
              "0                    1.0  female         right handed   \n",
              "1                    2.0  female         right handed   \n",
              "2                    2.0  female         right handed   \n",
              "3                    1.0  female         right handed   \n",
              "4                    1.0  female         right handed   \n",
              "...                  ...     ...                  ...   \n",
              "1005                 1.0  female         right handed   \n",
              "1006                 5.0    male          left handed   \n",
              "1007                 0.0  female         right handed   \n",
              "1008                 1.0  female         right handed   \n",
              "1009                 1.0    male         right handed   \n",
              "\n",
              "                    Education  Only child  Village - town  \\\n",
              "0     college/bachelor degree          no         village   \n",
              "1     college/bachelor degree          no            city   \n",
              "2            secondary school          no            city   \n",
              "3     college/bachelor degree         yes            city   \n",
              "4            secondary school          no         village   \n",
              "...                       ...         ...             ...   \n",
              "1005         secondary school          no            city   \n",
              "1006           masters degree          no         village   \n",
              "1007         secondary school         yes            city   \n",
              "1008  college/bachelor degree          no            city   \n",
              "1009         secondary school          no         village   \n",
              "\n",
              "      House - block of flats  \n",
              "0             block of flats  \n",
              "1             block of flats  \n",
              "2             block of flats  \n",
              "3             house/bungalow  \n",
              "4             house/bungalow  \n",
              "...                      ...  \n",
              "1005          house/bungalow  \n",
              "1006          house/bungalow  \n",
              "1007          block of flats  \n",
              "1008          block of flats  \n",
              "1009          house/bungalow  \n",
              "\n",
              "[1010 rows x 150 columns]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vKZDi4ngL6Y4",
        "outputId": "87bcddad-0d73-4ab6-a873-f2eee03e8484"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Index(['Music', 'Slow songs or fast songs', 'Dance', 'Folk', 'Country',\n",
            "       'Classical music', 'Musical', 'Pop', 'Rock', 'Metal or Hardrock',\n",
            "       'Punk', 'Hiphop, Rap', 'Reggae, Ska', 'Swing, Jazz', 'Rock n roll',\n",
            "       'Alternative', 'Latino', 'Techno, Trance', 'Opera'],\n",
            "      dtype='object')\n"
          ]
        }
      ],
      "source": [
        "music_columns=data.columns[:19]\n",
        "print(music_columns)\n",
        "music_data=data[music_columns].dropna()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pqkUey6GL6Y4"
      },
      "source": [
        "The answers are of course correlated and we expect to have typical patterns recurring, that we define as people liking similar types of songs.\n",
        "\n",
        "The patterns may be also mixing, for e.g. a class of people may like classic `Pop` and `Reggae`, but not `Latino`. An other class may like `Latino` and `Reggae`, but not `Pop`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eC_1UsLZL6Y4"
      },
      "source": [
        "PCA will help to find these typical patterns and their number in a data driven fashion. As we will see these patterns will naturally appear when trying to compress data in a lower dimensional space."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cXqNWQWT91kU"
      },
      "source": [
        "### Theory overview."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YydjP9pLL6Y4"
      },
      "source": [
        "We will look at PCA from the point of view of `dimensionaliy reduction`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xVw0KP4_91kW"
      },
      "source": [
        "**Objective:** PCA is used for dimensionality reduction when we have a large number $D$ of features with non-trivial intercorrelation ( data redundancy ) and to isolate relevant features. The number of features $D$ defines the original dimension of the dataset. Each sample defines a vector of dimensionality $D$.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6iH35oOL6Y4"
      },
      "source": [
        "    QUESTION: what are the starting vectors in our survey dataset? How many do we have?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T12:13:53.622053Z",
          "start_time": "2022-09-27T12:13:53.611428Z"
        },
        "id": "x74cyGHiL6Y4"
      },
      "source": [
        "PCA provides a new set of $M$ uncorrelated features for every data point, with $M \\le D$. The new features are:\n",
        "\n",
        "- a linear combination of the original ones ;\n",
        "- uncorrelated between each other ;\n",
        "\n",
        "If $M \\ll D$ we get an effective dimensionality reduction."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T12:17:23.351734Z",
          "start_time": "2022-09-27T12:17:23.345972Z"
        },
        "id": "DWaB15hBL6Y4"
      },
      "source": [
        "    QUESTION: Does the number of data points changes after applying PCA?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6GHjtsrNaDA8"
      },
      "source": [
        "***MOTIVATION***\n",
        "\n",
        "What do we need from a dimensionality reduction?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "in46SnZcRfdu"
      },
      "source": [
        "<img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/pca-theory.png\" width=\"100%\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnnFBIaf91kX"
      },
      "source": [
        "***REPRESENTATION***\n",
        "\n",
        "Each data point indexed by $p=1..N$ starts from an original $D-$dimensional space:\n",
        "$$x_p=(x_{p,1},...,x_{p,D})$$\n",
        "\n",
        "We want to find another more economical representation with $M$ components:\n",
        "$$x'_p=(x_{p,1},...,x_{p,M})$$\n",
        "\n",
        "PCA does that by defining $D$ auxiliar vectors, called the `principal components` or `principal vectors`:\n",
        "\n",
        "$$k_1=(k_{1,1},...,k_{1,D})$$\n",
        "$$...$$\n",
        "$$k_D=(k_{D,1},...,k_{D,D})$$\n",
        "\n",
        "$$\\mathbf k_{i'} \\cdot \\mathbf k_{j'} = \\delta_{i',j'}$$\n",
        "\n",
        "such that:\n",
        "\n",
        "$$x'_{p,i'}=\\mathbf x_p \\cdot \\mathbf k_{i'}, i'=1...D$$\n",
        "\n",
        "How are these principal components chosen? How can prove that:\n",
        "\n",
        "$$S^2=\\frac{1}{N}\\sum_{p=1}^{N}  |\\mathbf{x}_p - \\mathbf{\\overline{x}}|^2=\\sum_{i'=1}^{D}\\left( \\frac{1}{N}\\sum_{p=1}^{N}  |x_{p,i'} - \\overline{x'}_{i'}|^2\\right) $$  \n",
        "and therefore:\n",
        "\n",
        "$$S^2=\\epsilon_1+\\epsilon_2+...+\\epsilon_D$$\n",
        "\n",
        "PCA finds principal vectors $\\mathbf k$ such that $\\epsilon_1$ is as largest as possible, $\\epsilon_2<\\epsilon_1$ is such that $\\epsilon_1+\\epsilon_2$ is as largest as possible, and so on. Therefore truncating the summation the error is as small as possible among all possible choices all $k$-vectors. The $\\epsilon$ are called the `explained variance` values.\n",
        "\n",
        "The ratios:\n",
        "\n",
        "$$\\rho_1=\\epsilon_1/S^2,\\rho_2=\\epsilon_2/S^2,...$$\n",
        "\n",
        "are called the `explained variance ratio` values, and their cumulated brothers:\n",
        "\n",
        "$$r_1=\\epsilon_1/S^2,r_2=(\\epsilon_1+\\epsilon_2)/S^2,...,r_D=1$$\n",
        "\n",
        "is the `cumulated explained variance ratio`.\n",
        "\n",
        "The number of components is chosen selecting an optimal number of components (M) to keep. The plot of the explained variance ratio as a function of k is called a *scree plot* as serves to select an optimal value of $M$ (if it exists). If $M$ is small we have obtained a new data representation.\n",
        "\n",
        "NAMES TO REMEMBER:\n",
        "\n",
        "- `Principal components`: A sequence of orthonormal vectors $k_1,..,k_n$. We can interpret these vectors as the typical patterns found in the data, from increasing to decreasing probability of appearance.\n",
        "\n",
        "- `Scores`: For every sample-point $p$, the new features are called scores are given by the component of $p$ along the $k$ vectors: $$x'_{p,i}=\\mathbf x_p \\cdot \\mathbf k_i$$, sometimes also denoted with the letter $s_{p,i}$.  \n",
        "\n",
        "- `Explained variance`: For every k, the ratio between the variance of the reconstructed vectors and total variance. The number of components is chosen selecting an optimal k. The plot of the explained variance as a function of k is called a *scree plot*.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w_HMFXfknEdL"
      },
      "source": [
        "\n",
        "***INTERPRETATION: RECONSTRUCTED VECTOR***\n",
        "\n",
        "The scores can be used to derive a best approximation of the original vector, if needed, using only the first $M$ principal components:\n",
        "\n",
        "$$\\mathbf x_{rec}=\\sum_{i=1}^M s_{p,i} \\mathbf k_i$$\n",
        "\n",
        "When doing this operation, PCA acts like a filter to remove noise. This also explains the interpretation of the principal component as a pattern."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8acY1KzsL6Y4"
      },
      "source": [
        "    QUESTION:\n",
        "    What are the main differences between a vector $x_p$ and a principal component $k_n$? In particular:\n",
        "    1) how many do we have ?\n",
        "    2) what are their dimensions ?\n",
        "    \n",
        "    QUESTION:\n",
        "    What are the differences between $x_p$ and a score vector $s_p$? In particular:\n",
        "    1) how many do we have ?\n",
        "    2) what are their dimensions ?\n",
        "    \n",
        "    QUESTION:\n",
        "    1) How many reconstructed vectors do we have? What is their dimension ? Can we have a reconstructed vector with zero error?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hF-FTX_P91ka"
      },
      "source": [
        "### Sklearn: implementation and usage of PCA.\n",
        "\n",
        "\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGwCp73k91kb"
      },
      "source": [
        "We start showing a two-dimensional example that can be easy visualized."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LoEWiYnL91ko"
      },
      "source": [
        "We load the datasets that we are going to use for the examples:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-0_OW5ECL6Y4"
      },
      "outputs": [],
      "source": [
        "data=load_sample_data_pca()\n",
        "\n",
        "n_samples,n_dim=data.shape\n",
        "\n",
        "print('We have ',n_samples, 'samples of dimension ', n_dim)\n",
        "\n",
        "plt.figure(figsize=((5,5)))\n",
        "plt.grid()\n",
        "plt.plot(data[:,0],data[:,1],'o')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wzvgzzfA91k8"
      },
      "source": [
        "The data set is almost one dimensional. PCA will confirm this result.\n",
        "\n",
        "As with most of sklearn functionalities, we need first to create a PCA object. We will use the object methods to perform PCA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:12:10.956955Z",
          "start_time": "2022-09-26T19:12:10.953963Z"
        },
        "id": "l6poli6o91k_"
      },
      "outputs": [],
      "source": [
        "pca=PCA(n_components=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ju0UCpkP91lK"
      },
      "source": [
        "A call to the pca.fit method computes the principal components"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:12:12.057604Z",
          "start_time": "2022-09-26T19:12:12.034521Z"
        },
        "id": "ifWK32ME91lM",
        "outputId": "1ae523bb-a49c-4555-874b-b9be1051f22c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "PCA(n_components=2)"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pca.fit(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZdfZbLv91lW"
      },
      "source": [
        "Now the pca.components_ attribute contains the principal components. We can print them alongside with the data and check that they constitute an orthonormal basis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I1Zwy37ZL6Y5"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=((5,5)))\n",
        "plt.grid()\n",
        "plt.plot(data[:,0],data[:,1],'o')\n",
        "\n",
        "circle=plt.Circle((0, 0), 1.0, linestyle='--', color='red',fill=False)\n",
        "ax=plt.gca()\n",
        "ax.add_artist(circle)\n",
        "\n",
        "for vec in pca.components_:\n",
        "    plt.quiver([0], [0], [vec[0]], [vec[1]], angles='xy', scale_units='xy', scale=1)\n",
        "\n",
        "plt.xlim(-2,2)\n",
        "plt.ylim(-2,2)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q48o3r3t91li"
      },
      "source": [
        "The pca.explained_variance_ratio_ attribute contains the explained variance. In this case we see that already the first reconstructed vector explains 95% of the variance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:12:22.269942Z",
          "start_time": "2022-09-26T19:12:22.263717Z"
        },
        "id": "TwEopdTN91lk",
        "outputId": "ae5178f3-5a2e-4063-b54d-747698d779eb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[0.95140729 0.04859271]\n"
          ]
        }
      ],
      "source": [
        "print(pca.explained_variance_ratio_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfaA1wo391ls"
      },
      "source": [
        "To compute the reconstructed vectors for k=1 we first need to compute the scores and then multiply by the basis vectors:\n",
        "\n",
        "$$\\mathbf x_{rec}=\\sum_{i<M} (\\mathbf x \\cdot \\mathbf k_i) \\mathbf k_i$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:12:24.347892Z",
          "start_time": "2022-09-26T19:12:24.343929Z"
        },
        "id": "enRkHTyO91lu"
      },
      "outputs": [],
      "source": [
        "k=1\n",
        "scores=pca.transform(data)\n",
        "res=np.dot(scores[:,:k], pca.components_[:k,:] )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BMX6z3QKL6Y5"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=((5,5)))\n",
        "plt.plot(res[:,0],res[:,1],'o')\n",
        "plt.plot(data[:,0],data[:,1],'o')\n",
        "\n",
        "for a,b,c,d in zip(data[:,0],data[:,1],res[:,0],res[:,1]) :\n",
        "    plt.plot([a,c],[b,d],'-', linestyle = '--', color='red')\n",
        "\n",
        "plt.grid()\n",
        "\n",
        "plt.xlim(-1.0,1.0)\n",
        "plt.ylim(-1.0,1.0)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZlmDioD91mD"
      },
      "source": [
        "The same procedure is followed for high dimensional datasets. Here we generate random data which lies almost on a 6-dimensional subspace. The resulting scree plot can be used to find this result in a semi-automatic fashion."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tzC3iXjoL6Y5"
      },
      "source": [
        "Let's redo the same now with our survey dataset, to review the concepts again and think \"high\"-dimensionally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T14:44:10.677074Z",
          "start_time": "2022-09-27T14:44:10.652371Z"
        },
        "id": "FeaMbSd9L6Y5",
        "outputId": "17ac3312-5ad1-406a-a80b-bb638658130a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "PCA()"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pca=PCA()\n",
        "pca.fit(music_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "scW5XkGvL6Y5"
      },
      "outputs": [],
      "source": [
        "plt.plot(pca.explained_variance_ratio_,'-o')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ASRvM6OKL6Y6"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(pca.components_.transpose(),\n",
        "                  columns = [f'V_{i+1}' for i in range(len(music_columns))],\n",
        "                  index=music_columns)\n",
        "df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7qiQtvp4L6Y6"
      },
      "outputs": [],
      "source": [
        "for vector in ['V_1','V_2','V_3','V_4']:\n",
        "    plt.figure()\n",
        "    plt.title(vector)\n",
        "    plt.plot(np.arange(len(music_columns)),list(df[vector]),'-o')\n",
        "    _=plt.xticks(np.arange(len(music_columns)),music_columns, rotation=90)\n",
        "    plt.grid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T12:40:33.906396Z",
          "start_time": "2022-09-27T12:40:33.902364Z"
        },
        "id": "UmMogS2sL6Y6"
      },
      "source": [
        "### EXERCISE 1 : Find the dimensionality of the hidden dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Tcjhv_FEL6Y6"
      },
      "outputs": [],
      "source": [
        "# In this exercise you will take a custom high dimensional dataset and try to find its dimensionality.\n",
        "\n",
        "# STORY: you suppose you have 20 sensors measuring blood pressure in different parts of the bad during the day,\n",
        "# reporting measurments every second.\n",
        "#\n",
        "# We assume that some particular movements of the body will produce a correlated response in blood pressure.\n",
        "# Furthermore, these body movements do not happen independently but are superposing each other,\n",
        "#so that the final pressure measured is a sum of all of them.\n",
        "#\n",
        "# You are given a dataset of recording. Your task is to use PCA to discover, under these assumptions,\n",
        "# how many are the body movements producing these resposes, and to characterize the blood pressure response along these 20 sensors.\n",
        "\n",
        "# To do that:\n",
        "\n",
        "# 1. Load the data using the function data=load_multidimensional_data_pca() , check the dimensionality of the data and plot them.\n",
        "data=...\n",
        "n_samples,n_dim=...\n",
        "print('We have ',n_samples, 'samples of dimension ', n_dim)\n",
        "\n",
        "# 2. Define a PCA object and perform the PCA fitting.\n",
        "pca=PCA()\n",
        "pca....\n",
        "\n",
        "# 3. Check the explained variance ratio and select best number of components. How many are the typical body movements?\n",
        "print(pca...)\n",
        "plt.plot(...)\n",
        "\n",
        "# 3. Plot the pattern response associated to the typical body movements by exploring the principal vectors.\n",
        "for vec in pca.components_[:...]:\n",
        "    ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6nJsEk2R91nM"
      },
      "source": [
        "### EXERCISE 2 : Find the hidden drawing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H7agKcnqL6Y6"
      },
      "outputs": [],
      "source": [
        "### In this exercise you will take a high dimensional dataset, find the optimal number of principal components\n",
        "# and see what plot do we get if we visualize in a 2D plot the first two scores.\n",
        "\n",
        "#\n",
        "## STORY: there is a man of a parallel universe where strings generated a very high dimensional space\n",
        "#instead of our 3D one. He wants to communicate with you making a drawing that you can access as a set of\n",
        "#points in a multidimensional letter, given to you as a numpy array. The guy tried to do hist best to communicate\n",
        "#with you. He cannot write in a 2D space, but he can write on a 2-D subspace of this multidimensional one.\n",
        "#Mathematically he can link in a subspace generated by the multidimensional vectors k1 and k2, such a subspace\n",
        "#is the subset of points s1 * k_1 + s_2 * k_2 .\n",
        "\n",
        "# Can you decode the message?\n",
        "#\n",
        "\n",
        "# 1. Load the data using the function data=load_ex2_data_pca(seed=1235) , check the dimensionality of the data to\n",
        "#get a feeling of how high dimensional this letter is.\n",
        "data=...\n",
        "n_samples,n_dim=...\n",
        "print('We have ',n_samples, 'samples of dimension ', n_dim)\n",
        "\n",
        "# 2. Define a PCA object and perform the PCA fitting.\n",
        "pca=PCA()\n",
        "pca....\n",
        "\n",
        "# 3. Check the explained variance ratio and select best number of components. Did the guy do a good job?\n",
        "print(pca...)\n",
        "plt.plot(...)\n",
        "\n",
        "# 4. Try to plot the first score against the second one, for each of the data point. What does the alien want to tell you?\n",
        "k=...\n",
        "data_transformed=...\n",
        "plt.plot(data_transformed[:,...],data_transformed[:,...],'o')\n",
        "plt.xlabel('First component')\n",
        "plt.ylabel('Second component')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-sYehIF9L6Y6"
      },
      "source": [
        "### EXERCISE 3 (bonus) :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T15:16:35.115619Z",
          "start_time": "2022-09-27T15:16:35.112616Z"
        },
        "id": "1Apt5eBGL6Y6"
      },
      "outputs": [],
      "source": [
        "# Explore more the survey dataset. Suppose you have studied the first vectors and found\n",
        "# some 4 types of music advertisment, targetted to the 4 classes of clients.\n",
        "\n",
        "# Given a customer, how would you choose which advertisement to send him?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3AtpLmQef_R"
      },
      "source": [
        "### Final comments:\n",
        "\n",
        "PCA is able therefore to make this mapping:\n",
        "\n",
        "$(x_1,...,x_D) \\rightarrow (y_1,..,y_M)$\n",
        "\n",
        "Here we focused on data compression, but it is also very important that $y_1,...,y_M$ are uncorrolated for interpratibility purposes. Being uncorrelated means (roughly) that in our dataset we can change one variable without affecting the others. The dimensions 1,...,M are often therefore more interpretable and providing more information.\n",
        "\n",
        "See e.g. a similar application here:\n",
        "\n",
        "\"Principal component analysis of dietary and lifestyle patterns in relation to risk of subtypes of esophageal and gastric cancer\", https://pubmed.ncbi.nlm.nih.gov/21435900/\n",
        "\n",
        ", where each data point $x$ is an answer from a questionnaire of food. The principal components are than typical \"patterns\" of answers that are uncorrlated, have a look at table 2, and if you want read the whole data story :)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEK5-Ksg91nw"
      },
      "source": [
        "## 2. Data visualization and embedding in low dimensions ( t-SNE / UMAP )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4rP48tTL91nz"
      },
      "source": [
        "### Theory overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_Jer5L-91n0"
      },
      "source": [
        "PCA is a linear embedding technique where the scores are a linear function of the original variables. This forces the number of principal components to be used to be high, if the manifold is highly non-linear. Curved manifolds need to be embedded in higher dimensions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iX4zvhaN91n2"
      },
      "source": [
        "Other `non-linear` embedding techniques start from a local description of the environment of each sample point in the original space:\n",
        "\n",
        "- `t-Sne` uses a `statistical description` of the environment of a sample point ;\n",
        "- `UMAP` describes the `topology` of the environment through a generalized \"triangulation\" (simplex decomposition) ;\n",
        "\n",
        "The projection on the low-dimensional space is optimized in order to match as much as possible the description of the local environment.\n",
        "\n",
        "It is not the goal of this introduction to discuss the derivation of such approaches, which can be found in the references:\n",
        "\n",
        "https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf\n",
        "\n",
        "https://arxiv.org/pdf/1802.03426.pdf\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KhP5EIc791n7"
      },
      "source": [
        "Instead, the following, we will show how to apply practically these dimensionality reductions techniques. Keep in mind that the embedding is given by an iterative solution of a minimization problem and therefore the results may depend on the value of the random seed, especially for t-SNE visualizazions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DwIggNFN91n8"
      },
      "source": [
        "### Utilization in Python and examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RP5RbzRa91n9"
      },
      "source": [
        "To begin with, we create a t-SNE object that we are going to use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:17:17.498928Z",
          "start_time": "2022-09-26T19:17:17.495285Z"
        },
        "id": "BMZtUAyR91n9"
      },
      "outputs": [],
      "source": [
        "tsne_model = TSNE(perplexity=30, n_components=2, learning_rate=200, early_exaggeration=4.0,init='pca',\n",
        "                      n_iter=2000, random_state=2233212, metric='euclidean', verbose=100 )\n",
        "\n",
        "umap_model = umap.UMAP(n_neighbors=30, n_components=2, random_state=1711)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D-iq56Yn91oD"
      },
      "source": [
        "### Example 1: Exercise 3 Cont'd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OFZ_RAwj91oE"
      },
      "source": [
        "We will first visualize our multi-dimensional heart using t-SNE:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:26:26.119196Z",
          "start_time": "2022-09-26T19:26:25.619284Z"
        },
        "id": "8TFExSbC91oF",
        "outputId": "129a0689-e50e-4227-84d3-5017d7be68d8",
        "scrolled": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] Computing 91 nearest neighbors...\n",
            "[t-SNE] Indexed 651 samples in 0.001s...\n",
            "[t-SNE] Computed neighbors for 651 samples in 0.009s...\n",
            "[t-SNE] Computed conditional probabilities for sample 651 / 651\n",
            "[t-SNE] Mean sigma: 0.060286\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/arismarcolongo/miniconda3/envs/selective_search/lib/python3.9/site-packages/sklearn/manifold/_t_sne.py:982: FutureWarning: The PCA initialization in TSNE will change to have the standard deviation of PC1 equal to 1e-4 in 1.2. This will ensure better convergence.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] KL divergence after 250 iterations with early exaggeration: 10.369261\n",
            "[t-SNE] KL divergence after 300 iterations: 0.602112\n"
          ]
        }
      ],
      "source": [
        "data= load_ex2_data_pca(seed=1235, n_add=20)\n",
        "\n",
        "tsne_model = TSNE(perplexity=30, n_components=2, learning_rate=200, early_exaggeration=4.0,init='pca',\n",
        "                      n_iter=300, random_state=2233212, metric='euclidean', verbose=1 )\n",
        "\n",
        "tsne_heart = tsne_model.fit_transform(data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:25:22.637451Z",
          "start_time": "2022-09-26T19:25:22.473134Z"
        },
        "id": "Oc_97AnqL6Y6",
        "outputId": "2687b2fc-7e22-4303-fe33-a1f37de06eb9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] Computing 3 nearest neighbors...\n",
            "[t-SNE] Indexed 4 samples in 0.000s...\n",
            "[t-SNE] Computed neighbors for 4 samples in 0.002s...\n",
            "[t-SNE] Computed conditional probabilities for sample 4 / 4\n",
            "[t-SNE] Mean sigma: 12.094863\n",
            "[t-SNE] KL divergence after 250 iterations with early exaggeration: 33.738800\n",
            "[t-SNE] KL divergence after 1000 iterations: 0.047805\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "(4, 2)"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "from sklearn.manifold import TSNE\n",
        "from utils.routines import *\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])\n",
        "X_embedded = TSNE(n_components=2, learning_rate='auto',\n",
        "                   init='random', perplexity=3, verbose=1).fit_transform(X)\n",
        "X_embedded.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-26T19:25:31.237944Z",
          "start_time": "2022-09-26T19:25:30.300387Z"
        },
        "id": "5I-83Di_L6Y7",
        "outputId": "8acc87b4-3180-4b05-a44f-780ff3be079c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[t-SNE] Computing 10 nearest neighbors...\n",
            "[t-SNE] Indexed 651 samples in 0.001s...\n",
            "[t-SNE] Computed neighbors for 651 samples in 0.003s...\n",
            "[t-SNE] Computed conditional probabilities for sample 651 / 651\n",
            "[t-SNE] Mean sigma: 0.025506\n",
            "[t-SNE] KL divergence after 250 iterations with early exaggeration: 69.112373\n",
            "[t-SNE] KL divergence after 1000 iterations: 0.924472\n"
          ]
        }
      ],
      "source": [
        "data= load_ex2_data_pca(seed=1235, n_add=20)\n",
        "\n",
        "tsne_model = TSNE(n_components=2, learning_rate='auto', init='random', perplexity=3, verbose=1)\n",
        "\n",
        "tsne_heart = tsne_model.fit_transform(data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cuad41J_L6Y7"
      },
      "outputs": [],
      "source": [
        "plt.scatter(tsne_heart[:,0],tsne_heart[:,1])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O0R8p_0E91oR"
      },
      "source": [
        "And using UMAP :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BUU2YauU91oT"
      },
      "outputs": [],
      "source": [
        "umap_model = umap.UMAP(n_neighbors=30, n_components=2, random_state=1711)\n",
        "\n",
        "umap_hart = umap_model.fit_transform(data)\n",
        "plt.scatter(umap_hart[:, 0], umap_hart[:, 1])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kot3Prcv91oi"
      },
      "source": [
        "### Example 2: Mnist dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kWohsqFvL6Y7"
      },
      "outputs": [],
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
        "\n",
        "n_examples = 5000\n",
        "data=train_images[:n_examples,:].reshape(n_examples,-1)\n",
        "data=data/255\n",
        "\n",
        "labels=train_labels[:n_examples]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TDeL_V2a91op",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# not to run on COLAB\n",
        "\n",
        "# tsne_model = TSNE(perplexity=10, n_components=2, learning_rate=200,\n",
        "#                   early_exaggeration=4.0,init='pca',\n",
        "#                   n_iter=2000, random_state=2233212,\n",
        "#                   metric='euclidean', verbose=100, n_jobs=1)\n",
        "\n",
        "# tsne_mnist = tsne_model.fit_transform(data)\n",
        "\n",
        "# plt.scatter(tsne_mnist[:,0],tsne_mnist[:,1],c=labels,s=10)\n",
        "# plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZRwwHbqpt9n0"
      },
      "source": [
        " | <img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/t_sne_mnist.png\" width=\"100%\"/> | <img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/mnist.png\" width=\"100%\"/> |\n",
        " |  -----:| -----:|"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T_KKO8vQL6Y7"
      },
      "outputs": [],
      "source": [
        "umap_model = umap.UMAP(n_neighbors=10, n_components=2, random_state=1711)\n",
        "umap_mnist = umap_model.fit_transform(data)\n",
        "plt.scatter(umap_mnist[:, 0], umap_mnist[:, 1], c=labels, s=10)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPOM-hCV91pB"
      },
      "source": [
        "### Example 3: Fashion_Mnist dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Uq3DPRiV91pF"
      },
      "outputs": [],
      "source": [
        "fmnist = tf.keras.datasets.fashion_mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = fmnist.load_data()\n",
        "\n",
        "n_examples = 5000\n",
        "data=train_images[:n_examples,:].reshape(n_examples,-1)\n",
        "data=data/255\n",
        "\n",
        "labels=train_labels[:n_examples]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ytrl7jyC91pR",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# not to run on COLAB\n",
        "\n",
        "# tsne_model = TSNE(perplexity=50, n_components=2, learning_rate=200, early_exaggeration=4.0,init='pca',\n",
        "#                      n_iter=1000, random_state=2233212, metric='euclidean', verbose=100 )\n",
        "\n",
        "# tsne_fmnist = tsne_model.fit_transform(data)\n",
        "\n",
        "# plt.scatter(tsne_fmnist[:,0],tsne_fmnist[:,1],c=labels,s=10)\n",
        "# plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-27T15:07:26.743568Z",
          "start_time": "2022-09-27T15:07:26.738108Z"
        },
        "id": "UUnsyxnet9n9"
      },
      "source": [
        " | <img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/t_sne_fmnist.png\" width=\"100%\"/> | <img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/fmnist.png\" width=\"100%\"/>\n",
        " |  -----:| -----:|"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xq3vZFJy91p8"
      },
      "outputs": [],
      "source": [
        "umap_model = umap.UMAP(n_neighbors=50, n_components=2, random_state=1711)\n",
        "umap_fmnist = umap_model.fit_transform(data)\n",
        "plt.scatter(umap_fmnist[:, 0], umap_fmnist[:, 1], c=labels, s=10)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bz2LEGDP91qM"
      },
      "source": [
        "### Example 4: House prices"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eycvatmq91qO"
      },
      "outputs": [],
      "source": [
        "data=house_prices_dataset_normed()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "82secxju91qZ",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# not to run on COLAB\n",
        "\n",
        "#tsne_model = TSNE(perplexity=30, n_components=2, learning_rate=200,\n",
        "#                  early_exaggeration=4.0,init='pca', n_iter=1000,\n",
        "#                  random_state=2233212, metric='euclidean', verbose=100)\n",
        "\n",
        "#tsne_houses = tsne_model.fit_transform(data)\n",
        "\n",
        "#plt.scatter(tsne_houses[:,0],tsne_houses[:,1],s=20)\n",
        "#plt.savefig('t_sne_houses.png')\n",
        "# plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESjUplVMt9oF"
      },
      "source": [
        "<img src=\"https://github.com/neworldemancer/DSF5/raw/master/figures/t_sne_houses.png\" width=\"50%\"/>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FWeCP6pN91qr"
      },
      "outputs": [],
      "source": [
        "umap_model = umap.UMAP(n_neighbors=30, n_components=2, random_state=1711)\n",
        "umap_houses = umap_model.fit_transform(data)\n",
        "plt.scatter(umap_houses[:, 0], umap_houses[:, 1], s=20)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1B3wGs2Q91rE"
      },
      "source": [
        "**Message:** Visualization techniques are useful for having an initial grasp of multi-dimensional datasets and guide further analysis and the choice of the modelling data strategy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "78xdYF9-B-jA"
      },
      "source": [
        "**Caveats:**\n",
        "- available t-SNE implementations may vary a lot in terms of performance. Computational time can be reduced performing PCA before a t-SNE projection\n",
        "\n",
        "- UMAP, thanks to the algorithm being amanable to clever initializations and optimization schemes, offers great stability and scaling properties\n",
        "\n",
        "- UMAP, even if starting from a local picture, is generally more able to spread apart different clusters\n",
        "\n",
        "- The result of an embedding may depend on the values of the metaparameters. One should try to see how the final embedding changes in order to get to a complete picture"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "8UQgU5I-lEll",
        "-2uMw5C4CUcj",
        "q7CNxkPdNB4L",
        "8S1jwU4cXQX4",
        "5MXcNBOBCUcy"
      ],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}