VOiCES_workshop/Machine Learning with audio data-answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with audio data using pandas and sci-kit learn. \n",
    "#### Nina Lopatina, Ph.D.   |  InQTel, Lab41  |  January 18 2019\n",
    "In this workshop, we will learn how to classify a speaker's gender from realistic audio data using pandas and sci-kit learn. We will go through a brief description of VOiCES data, how to view and manipulate data in Pandas, and how to train a simple model in sci-kit learn. There are exercises throughout the workshop, with a longer exercise available at the end for those who wish to continue developing these skills or work with VOiCES data in the hackathon.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup: \n",
    "\n",
    "Directions in repo README"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Intro to VOiCES\n",
    "\n",
    "[VOiCES](https://voices18.github.io/) is an audio dataset put together in collaboration between Lab41 and SRI. \n",
    "\n",
    "Clean speech was recorded in rooms of different sizes, each having distinct room acoustic profiles, with background noise played concurrently. \n",
    "\n",
    "These recordings provide audio data that better represents real-use scenarios. \n",
    "\n",
    "These data are provided in wav files, but we have extracted some features from a subset of the data for this workshop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup to play example audio files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import librosa as lr\n",
    "import librosa.display\n",
    "import IPython.display\n",
    "\n",
    "filename_clean = 'data/Lab41-SRI-VOiCES-rm1-none-sp0083-ch003054-sg0005-mc01-stu-clo-dg090.wav'\n",
    "filename_babb = 'data/Lab41-SRI-VOiCES-rm1-babb-sp0083-ch003054-sg0005-mc12-lav-wal-dg090.wav'\n",
    "filename_tele = 'data/Lab41-SRI-VOiCES-rm1-tele-sp0083-ch003054-sg0005-mc03-stu-mid-dg090.wav'\n",
    "filename_musi = 'data/Lab41-SRI-VOiCES-rm1-musi-sp0083-ch003054-sg0005-mc07-stu-beh-dg090.wav'\n",
    "\n",
    "def player(fname):\n",
    "    #     # Read in the signal and sample rate\n",
    "    s0, sample_rate = lr.core.load(fname, sr=None, mono=True)\n",
    "    IPython.display.display(IPython.display.Audio(data=s0, rate=sample_rate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean audio file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "player(filename_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Same clip with television:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "player(filename_tele)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Same clip with music"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "player(filename_musi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Same clip with babble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "player(filename_babb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the data\n",
    "\n",
    "### Waveform amplitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "sp_x, sp_sr = lr.load(filename_tele) #Noisy\n",
    "src_x, src_sr = lr.load(filename_clean) #Without noise\n",
    "\n",
    "plt.figure(figsize = (10,5))\n",
    "lr.display.waveplot(src_x, src_sr, color = 'blue', alpha = 0.6, label = 'Source')\n",
    "lr.display.waveplot(sp_x, sr = sp_sr, alpha = 0.5, color = 'orange',label = 'Noisy Speech')\n",
    "plt.legend()\n",
    "plt.ylabel('Waveform amplitude', size = 16)\n",
    "plt.xlabel('Time (seconds)', size = 16)\n",
    "plt.title('Noisy and clean speech waveform amplitude', size = 20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fourier transformed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ft(x,sr,title):# source \n",
    "    ft = lr.stft(x)\n",
    "    db = lr.amplitude_to_db(abs(ft))\n",
    "    plt.figure(figsize=(14, 4))\n",
    "    plt.title(title, size = 20)\n",
    "    lr.display.specshow(db, sr=sr, x_axis='time', y_axis='hz')\n",
    "    plt.ylabel('Hz', size = 16)\n",
    "    plt.xlabel('Time (seconds)', size = 16)\n",
    "    clb = plt.colorbar()\n",
    "    clb.set_label('Decibels')\n",
    "    return db\n",
    "\n",
    "src_db = plot_ft(src_x,src_sr,'Fourier transformed power spectrum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mel spectogram transformed data \n",
    "\n",
    "Mel-Frequency analysis of speech is based on human perception experiments\n",
    "* It is observed that human ear acts as filter: it concentrates on only certain frequency components\n",
    "\n",
    "* These filters are non-uniformly spaced on the frequency axis: More filters in the low frequency regions & fewer filters in high frequency regions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ms(x,sr,title):# source \n",
    "    S = lr.feature.melspectrogram(x,sr)\n",
    "    db = lr.power_to_db(S)\n",
    "    plt.figure(figsize=(14, 4))\n",
    "    plt.title(title)\n",
    "    lr.display.specshow(db, sr=sr, x_axis='time', y_axis='mel')\n",
    "    plt.ylabel('Hz', size = 16)\n",
    "    plt.xlabel('Time (seconds)', size = 16)\n",
    "    clb = plt.colorbar()\n",
    "    clb.set_label('Decibels')\n",
    "    \n",
    "# Source\n",
    "src_db = plot_ms(src_x,src_sr,'Mel transformed power spectrum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Intro to Pandas\n",
    "\n",
    "### Pandas uses cases:\n",
    "\n",
    "- Finding trends in data\n",
    "- Business analytics\n",
    "- Cleaning data\n",
    "- Blending multiple data sources\n",
    "- Easy data manipulation to make awesome models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import some packages -- these are modules with specific functionalities to make your life easier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices = pd.read_csv('./data/VOiCES_90deg_features.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Structures and Viewing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the Dataframe and Series data structures in pandas. You can think of dataframes as a spreadsheet or table, and Series as columns. \n",
    "\n",
    "### What types of data are in this data frame?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data properties\n",
    "* mic_id:           Microphone #\n",
    "* mic_type:         studio or lavalier\n",
    "* location:         Distance from subject, see https://voices18.github.io/Lab41-SRI-VOiCES_README/\n",
    "* spk_angle:        all 90* here\n",
    "\n",
    "Statistical values\n",
    "* Centroid:        2D mean of audio data\n",
    "* variance:        Dispersion of samples around centroid\n",
    "* skewness:        symmetry of the probability density function of the amplitude of a time series. Positive skewness with more large than small values. \n",
    "* kurtosis:        measures the peakedness of the PDF of a time series. A kurtosis value close to three indicates a Gaussian-like peakedness. PDFs with relatively sharp peaks have kurtosis greater than three. PDFs with relatively flat peaks have kurtosis less than three\n",
    "* roll_off_min:    min & max frequency   \n",
    "\n",
    "Mel Frequency Cepstral Coefficient (MFCC) transformed data\n",
    "* mfcc 1-12: Features extracted from data generated into 12 Mel bands "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Most of the times the data we load can be large so we look at a subset\n",
    "# Default is first 5 entries\n",
    "voices.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many items?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(voices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick data statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices['noise']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1:\n",
    "### Select the microphone id column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Answer:\n",
    "voices['mic_id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find types of noise conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voices['noise'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: \n",
    "\n",
    "### Find the types of microphones used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer:\n",
    "voices['mic_id'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data visualization I\n",
    "\n",
    "We want to get to know our data\n",
    "\n",
    "### Plot all the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = voices.columns\n",
    "r = 6\n",
    "c = 5\n",
    "label_size = 22\n",
    "tick_size = 18\n",
    "title_size = 28\n",
    "\n",
    "fig = plt.figure(figsize = (5*c,5*r))\n",
    "\n",
    "for i,var in enumerate(features):\n",
    "    ax = fig.add_subplot(r,c,i+1)\n",
    "    x = voices.index\n",
    "    y = voices[var]\n",
    "    ax.scatter(x,y,c='m')\n",
    "\n",
    "    ax.set_title(var,size=title_size)\n",
    "    ax.set_xlabel('speaker', size = label_size)\n",
    "    ax.set_ylabel(var,size = label_size)\n",
    "    ax.tick_params(labelsize=tick_size)\n",
    "\n",
    "plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sorted plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = voices.columns\n",
    "r = 6\n",
    "c = 5\n",
    "label_size = 22\n",
    "tick_size = 18\n",
    "title_size = 28\n",
    "\n",
    "fig = plt.figure(figsize = (5*c,5*r))\n",
    "\n",
    "for i,var in enumerate(features):\n",
    "    ax = fig.add_subplot(r,c,i+1)\n",
    "    x = voices.index\n",
    "    y = voices.sort_values(by=var)[var] # Added sort_values()\n",
    "    ax.scatter(x,y,c='m')\n",
    "\n",
    "    ax.set_title(var,size=title_size)\n",
    "    ax.set_xlabel('sample', size = label_size) # Changed speaker to sample b/c the # no longer corresponds\n",
    "    ax.set_ylabel(var,size = label_size)\n",
    "    ax.tick_params(labelsize=tick_size)\n",
    "\n",
    "plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data visualization II\n",
    "\n",
    "We want to build a model that will classify the gender of the speaker. First let's check that there are differences between male and female speakers that can be observed in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split data by gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dataframes with Male & Female speakers\n",
    "M_DF = voices[voices['Gender']=='M']\n",
    "F_DF = voices[voices['Gender']=='F']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "F_DF.Gender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: \n",
    "\n",
    "### Create a dataframe with only samples with television background noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Answer:\n",
    "tele = voices[voices['noise']=='tele']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important consideration: How many data points in each split?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print length of each dataframe\n",
    "print(len(M_DF), \" male speakers\")\n",
    "print(len(F_DF), \" female speakers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Any concerns with the number of samples per class?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution: Trim data frame to have even data for each label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M_DF = M_DF[:len(F_DF)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(M_DF), \" male speakers\")\n",
    "print(len(F_DF), \" female speakers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select features useful for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(M_DF.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = M_DF.columns[9:]\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot data split by gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "r = 4\n",
    "c = 5\n",
    "label_size = 22\n",
    "tick_size = 18\n",
    "title_size = 28\n",
    "\n",
    "fig = plt.figure(figsize = (5*c,5*r))\n",
    "\n",
    "for i,var in enumerate(features):\n",
    "    ax = fig.add_subplot(r,c,i+1)\n",
    "    x = range(len(M_DF))\n",
    "    y = M_DF[var]\n",
    "    ax.scatter(x,y,c='g')\n",
    "    \n",
    "    x = range(len(F_DF))\n",
    "    y = F_DF[var]\n",
    "    ax.scatter(x,y,c='r')\n",
    "    \n",
    "    ax.set_title(var + ' by gender',size=title_size)\n",
    "    ax.set_xlabel('sample', size = label_size)\n",
    "    ax.set_ylabel(var,size = label_size)\n",
    "    ax.tick_params(labelsize=tick_size)\n",
    "\n",
    "plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.2)\n",
    "\n",
    "# Red is female, green is male"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: Modify the below block of code to more informatively display the grouped data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = 4\n",
    "c = 5\n",
    "label_size = 22\n",
    "tick_size = 18\n",
    "title_size = 28\n",
    "\n",
    "fig = plt.figure(figsize = (5*c,5*r))\n",
    "\n",
    "for i,var in enumerate(features):\n",
    "    ax = fig.add_subplot(r,c,i+1)\n",
    "    x = range(len(M_DF))\n",
    "    y = M_DF[var]\n",
    "    ax.scatter(x,y,c='g')\n",
    "    \n",
    "    x = range(len(F_DF))\n",
    "    y = F_DF[var]\n",
    "    ax.scatter(x,y,c='r')\n",
    "    \n",
    "    ax.set_title(var + ' by gender',size=title_size)\n",
    "    ax.set_xlabel('sample', size = label_size)\n",
    "    ax.set_ylabel(var,size = label_size)\n",
    "    ax.tick_params(labelsize=tick_size)\n",
    "\n",
    "plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.2)\n",
    "\n",
    "# Red is female, green is male"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = 4\n",
    "c = 5\n",
    "label_size = 22\n",
    "tick_size = 18\n",
    "title_size = 28\n",
    "\n",
    "fig = plt.figure(figsize = (5*c,5*r))\n",
    "\n",
    "for i,var in enumerate(features):\n",
    "    ax = fig.add_subplot(r,c,i+1)\n",
    "    x = range(len(M_DF))\n",
    "    y = M_DF.sort_values(by=var)[var]\n",
    "    ax.scatter(x,y,c='g')\n",
    "    \n",
    "    x = range(len(F_DF))\n",
    "    y = F_DF.sort_values(by=var)[var]\n",
    "    ax.scatter(x,y,c='r')\n",
    "    \n",
    "    ax.set_title(var + ' by gender',size=title_size)\n",
    "    ax.set_xlabel('sample', size = label_size)\n",
    "    ax.set_ylabel(var,size = label_size)\n",
    "    ax.tick_params(labelsize=tick_size)\n",
    "\n",
    "plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.2)\n",
    "\n",
    "# Red is female, green is male"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Any guesses for which features would be most informative for classifying gender?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Gender classification with Sci-kit learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix,accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Interleave rows from tables:\n",
    "\n",
    "# Initialize table\n",
    "df = pd.DataFrame(M_DF.loc[M_DF.index[0]])\n",
    "df = df.transpose()\n",
    "df = df.append(F_DF.loc[F_DF.index[0]])\n",
    "\n",
    "# Add rest of rows interleaved\n",
    "for i in range(1,len(M_DF)):\n",
    "    df = df.append(M_DF.loc[M_DF.index[i]])\n",
    "    df = df.append(F_DF.loc[F_DF.index[i]])\n",
    "\n",
    "# Remove redundant rows from index reseting\n",
    "df = df.reset_index()\n",
    "df = df.drop('index',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split into data and label\n",
    "\n",
    "X = df.drop('Gender',axis=1)\n",
    "X = X[X.columns[8:]] # Data\n",
    "\n",
    "y = df.Gender # Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_size = 0.3\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size,shuffle = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up to track accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc = pd.DataFrame(columns = ['model','train_accuracy','test_accuracy'])\n",
    "   \n",
    "def get_accuracy(model,model_name,df_acc,X_train, y_train,X_test,y_test):\n",
    "    mdl = model()\n",
    "    mdl.fit(X_train, y_train)\n",
    "    preds_test = mdl.predict(X_test)\n",
    "    preds_train = mdl.predict(X_train)\n",
    "    #add to next row:\n",
    "    nr = len(df_acc)\n",
    "    df_acc.loc[nr,'model'] = model_name\n",
    "    df_acc.loc[nr,'test_accuracy'] = round(accuracy_score(y_test,preds_test)*100,1)\n",
    "    df_acc.loc[nr,'train_accuracy'] = round(accuracy_score(y_train,preds_train)*100,1)\n",
    "    return df_acc, mdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 1: Logistic regression\n",
    "\n",
    "Simplest supervised binary classification algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc,logreg = get_accuracy(LogisticRegression,'LogisticRegression',df_acc,X_train, y_train,X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 2: Random Forest Classifier\n",
    "\n",
    "A random forest is supervised classification algorithm. It is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset, and uses averaging to improve the predictive accuracy and control over-fitting. The output is the mode of the class output of all the trees in the forest. If there are enough trees in the forest, the classifier won’t overfit the data. Lastly, RFC will identify and select the most important features from the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc, rdf = get_accuracy(RandomForestClassifier,'RandomForestClassifier',df_acc,X_train, y_train,X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improving accuracy with a hyperparameter search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert categorical to one hot\n",
    "y_encoded = pd.get_dummies(y)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y_encoded, test_size=test_size,shuffle = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create param grid for Randomized Hyperparameter Search\n",
    "\n",
    "param_grid = {'n_estimators': (5,10,50,100),\n",
    "              'max_features': (5,8, 12, 16)\n",
    "             }\n",
    "clf = RandomForestClassifier()\n",
    "grid = RandomizedSearchCV(clf, param_grid, cv=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat = grid.predict(X_test)\n",
    "accuracy = metrics.accuracy_score(y_test,y_hat)\n",
    "print('Accuracy for gender id ', round(accuracy*100,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Ideas for further improving accuracy?\n",
    "\n",
    "\n",
    "### Classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_rep = classification_report(y_true = y_test, y_pred = y_hat)\n",
    "\n",
    "#this report is just text, so needs to be converted to a more easily readable form\n",
    "def to_table(clf_rep):\n",
    "    report = clf_rep.splitlines()\n",
    "    res = []\n",
    "    res.append(['']+report[0].split())\n",
    "    for row in report[2:-2]:\n",
    "        res.append(row.split())\n",
    "    lr = report[-1].split()\n",
    "    res.append([' '.join(lr[:3])]+lr[3:])\n",
    "    output = np.array(res)\n",
    "    return output\n",
    "\n",
    "cols = []\n",
    "cols.append('Gender: 0F/1M')\n",
    "\n",
    "output = to_table(clf_rep)\n",
    "for item in output[0][1:]:\n",
    "    cols.append(item)\n",
    "    \n",
    "out_df = pd.DataFrame(output[1:],columns = cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We know 0 is F and 1 is M because\n",
    "\n",
    "y_test.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can we interpret our model: How is the decision being made?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get feature importance\n",
    "clf = RandomForestClassifier(n_estimators = 100)\n",
    "clf.fit(X_train,y_train)\n",
    "df_features = pd.DataFrame(columns = ['variable', 'contribution'])\n",
    "\n",
    "df_features['contribution'] = pd.Series(clf.feature_importances_)\n",
    "df_features['variable'] = pd.Series(X.columns)\n",
    "\n",
    "# How much do each of the features contribute? \n",
    "df_features = df_features.sort_values(by='contribution',ascending = False)\n",
    "p = df_features.plot.bar(x = 'variable',legend= False, color = 'c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is it easier to identify gender for specific noise conditions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Accuracy for the clean data\n",
    "\n",
    "# Create dataframe including noise\n",
    "features = ['noise','Gender', 'Centroid', 'variance', 'skewness',\n",
    "       'kurtosis', 'mfcc1', 'mfcc2', 'mfcc3', 'mfcc4', 'mfcc5', 'mfcc6',\n",
    "       'mfcc7', 'mfcc8', 'mfcc9', 'mfcc10', 'mfcc11', 'mfcc12', 'roll_off_max',\n",
    "       'roll_off_min']\n",
    "X = df[features]\n",
    "X = X.drop('Gender',axis=1)\n",
    "X.columns\n",
    "\n",
    "# Split\n",
    "_, X_test_noise, _, _ = train_test_split(X, y, test_size=test_size, shuffle = False)\n",
    "\n",
    "# Calculate accuracy\n",
    "noise_type = 'none'\n",
    "metrics.accuracy_score(y_test[X_test_noise['noise']==noise_type],y_hat[X_test_noise['noise']==noise_type])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5: Show accuracy by noise condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer:\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6:  Can you improve on this model using sklearn or make any interesting inferences about the data using pandas?\n",
    "\n",
    "Open ended or try one of the suggested exercises below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6a: Change hyperparameters of RFC from the below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "'''\n",
    "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
    "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
    "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "            min_samples_leaf=1, min_samples_split=2,\n",
    "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
    "            oob_score=False, random_state=None, verbose=0,\n",
    "            warm_start=False)\n",
    "'''\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6b: Analysis of incorrectly classified data points "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Addendum: Code for continuing to work with VOiCES data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A. Speech preprocessing\n",
    "\n",
    "To modify the preprocessing we worked with above to other data subsets or to change the preprocessing: \n",
    "\n",
    "git clone https://github.com/Lab41/VOiCES-subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B. Speaker id with torch\n",
    "\n",
    "1.  Clone the cyphercat repo with \n",
    "\n",
    "git clone https://github.com/ninalopatina/cyphercat\n",
    "\n",
    "2. pip install torch\n",
    "\n",
    "3. Modify & insert the below code into a cell\n",
    "\n",
    "sys.path.insert(0, '{path to cyphercat repo}')\n",
    "\n",
    "import cyphercat as cc\n",
    "\n",
    "import torch\n",
    "\n",
    "4. Utilize the functions below\n",
    "\n",
    "#### Let the Lab41 members know if you have any questions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class tensorToMFCC:\n",
    "    def __call__(self, y):\n",
    "#         y = y.numpy()\n",
    "        dims = y.shape\n",
    "        y = libr.feature.melspectrogram(np.reshape(y, (dims[1],)), 16000, n_mels=number_of_mels,\n",
    "                               fmax=8000)\n",
    "        y = libr.feature.mfcc(S = libr.power_to_db(y))\n",
    "        y = torch.from_numpy(y)                           \n",
    "        return y.float()\n",
    "\n",
    "class STFT:\n",
    "    def __call__(self,y):\n",
    "        dims = y.shape\n",
    "        y = np.abs(libr.core.stft(np.reshape(y, (dims[1],))))\n",
    "        y = torch.from_numpy(y).permute(1,0)\n",
    "        return y.float()\n",
    "\n",
    "transform_type = 'MFCC'\n",
    "if transform_type == 'SFTF':\n",
    "    target_net_type = cc.ft_cnn_classifer\n",
    "    shadow_net_type = cc.ft_cnn_classifer\n",
    "    in_size = 94# 20 forMFCC,  94 for STFT\n",
    "    transform  = STFT() ## STFT or MFCC\n",
    "elif transform_type == 'MFCC':\n",
    "    transform  = tensorToMFCC()\n",
    "    target_net_type = cc.MFCC_cnn_classifier\n",
    "    shadow_net_type = cc.MFCC_cnn_classifier\n",
    "    in_size = 20\n",
    "    \n",
    "# To load data:\n",
    "subset = 'room-1'\n",
    "[speaker_df, sample_df] = cc.Voices_preload_and_split(subset = subset)\n",
    "\n",
    "valid_sequence_train_target = cc.Voices_dataset(df=dfs[0], transform = transform)\n",
    "valid_sequence_test_target = cc.Voices_dataset(df=dfs[1], transform = transform)\n",
    "\n",
    "target_train_loader = DataLoader(valid_sequence_train_target,\n",
    "                      batch_size=batch_size,\n",
    "                      shuffle=True,\n",
    "                      num_workers=8,\n",
    "                    drop_last = True\n",
    "                     # pin_memory=True # CUDA only\n",
    "                     )\n",
    "\n",
    "\n",
    "target_test_loader = DataLoader(valid_sequence_test_target,\n",
    "                      batch_size=batch_size,\n",
    "                      shuffle=True,\n",
    "                      num_workers=8\n",
    "                     # pin_memory=True # CUDA only\n",
    "                     )\n",
    "# Set up the model:\n",
    "\n",
    "#in_size defined above\n",
    "n_hidden = 512\n",
    "n_classes = valid_sequence_test_target.num_speakers\n",
    "print(n_classes,' speakers')\n",
    "df.at[df_idx,'# speakers']=n_classes\n",
    "\n",
    "\n",
    "target_net = target_net_type(n_classes).to(device)\n",
    "target_net.apply(models.weights_init)\n",
    "\n",
    "target_loss = nn.CrossEntropyLoss()\n",
    "target_optim = optim.Adam(target_net.parameters(), lr=.001)\n",
    "\n",
    "# Train the model\n",
    "train_accuracy, test_accuracy = cc.train(target_net, target_train_loader, target_test_loader, target_optim, target_loss, n_epochs, verbose = False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}