05_model_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Validation\n",
    "In this notebook, I take the mp3s of videos from Wes's and Scott's youtube channels to verify the speaker prediction model. Wes' video is [CSS GRID: Full Bleed Blog Layout Exercise — 25 of 25](https://www.youtube.com/watch?v=z9p4ctpvmTs) and Scott's video is [Code Blog - How I Fixed A Very Weird Bug](https://www.youtube.com/watch?v=Jt5zb94F-bI). The idea here is that the model should predict most of the one second segments of the audio are the author of the video speaking.\n",
    "\n",
    "In this notebook we:\n",
    "1. Split the video mp3s into one second secgments\n",
    "2. Make predictions for the two videos\n",
    "3. Spot check that most of the segments are predicted as the author of the video speaking.\n",
    "\n",
    "Our model is quite accurate with both videos, with `95%` accuracy for Scott and `97%` accuracy for Wes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This directory contains two folders, `clips` and `mp3-files`. The `mp3-files` folder contains downloaded mp3s from YouTube videos of Wes and Scott. Wes' video is [CSS GRID: Full Bleed Blog Layout Exercise — 25 of 25](https://www.youtube.com/watch?v=z9p4ctpvmTs) and Scott's video is [Code Blog - How I Fixed A Very Weird Bug](https://www.youtube.com/watch?v=Jt5zb94F-bI). The `clips` folder contains the one second clips of both episodes for validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split MP3s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import jsonlines\n",
    "import librosa\n",
    "import numpy as np\n",
    "import pydub\n",
    "from sklearn.externals import joblib\n",
    "from tqdm import tqdm\n",
    "\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_audio(segment, fname):\n",
    "    \"\"\"Takes an episode segment and splits it into 1s chunks\"\"\"\n",
    "    slices_1s = segment[::1000]\n",
    "    for i, chunk in enumerate(slices_1s):\n",
    "        second_zfill = str(i).zfill(4)\n",
    "        try:\n",
    "            with open(f\"validation-data/clips/{fname}-{second_zfill}.mp3\", \"xb\") as f:\n",
    "                chunk.export(f, format=\"mp3\")\n",
    "        except FileExistsError:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "wes_mp3 = \"./validation-data/mp3-files/CSS GRID Full Bleed Blog Layout Exercise 25 of 25.mp3\"\n",
    "wes_mp3_segment = pydub.AudioSegment.from_mp3(wes_mp3).set_channels(1)\n",
    "split_audio(wes_mp3_segment, \"wes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "scott_mp3 = \"./validation-data/mp3-files/Code Blog - How I Fixed A Very Weird Bug.mp3\"\n",
    "scott_mp3_segment = pydub.AudioSegment.from_mp3(scott_mp3).set_channels(1)\n",
    "split_audio(scott_mp3_segment, \"scott\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_model = joblib.load(\"syntax-speaker-predictor.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features(mp3_path):\n",
    "    y, sr = librosa.load(mp3_path)\n",
    "    mel_spec = librosa.feature.melspectrogram(y=y, sr=sr)\n",
    "    mel_ravel = mel_spec.ravel()\n",
    "    return mel_ravel\n",
    "\n",
    "\n",
    "def make_prediction(mp3_path):\n",
    "    try:\n",
    "        x = get_features(mp3_path)\n",
    "    except EOFError:\n",
    "        return 2\n",
    "    if x.shape != (5888,):\n",
    "        return 2  # other (under 1s)\n",
    "    else:\n",
    "        pred = full_model.predict([x])\n",
    "        return pred[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 657/657 [01:22<00:00,  7.92it/s]\n"
     ]
    }
   ],
   "source": [
    "wes_clips = list(sorted(Path(\"./validation-data/clips/\").glob(f\"wes-*.mp3\")))\n",
    "wes_predictions = []\n",
    "for clip in tqdm(wes_clips, total=len(wes_clips)):\n",
    "    wes_predictions.append((clip.stem, make_prediction(str(clip))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 638 0.97\n",
      "0 16 0.02\n",
      "2 3 0.00\n"
     ]
    }
   ],
   "source": [
    "# Should be mostly `1` for Wes\n",
    "# Some `2` for \"Other\" Intro Music, etc\n",
    "\n",
    "speaker_counts_wes = Counter(i[1] for i in wes_predictions).most_common()\n",
    "for speaker, count in speaker_counts_wes:\n",
    "    print(speaker, count, f\"{count / len(wes_predictions):.2f}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 641/641 [01:11<00:00,  8.92it/s]\n"
     ]
    }
   ],
   "source": [
    "scott_clips = list(sorted(Path(\"./validation-data/clips/\").glob(f\"scott-*.mp3\")))\n",
    "scott_predictions = []\n",
    "for clip in tqdm(scott_clips, total=len(scott_clips)):\n",
    "    scott_predictions.append((clip.stem, make_prediction(str(clip))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 608 0.95\n",
      "1 32 0.05\n",
      "2 1 0.00\n"
     ]
    }
   ],
   "source": [
    "# Should be mostly `0` for Scott\n",
    "# Some `2` for \"Other\" Intro Music, etc\n",
    "\n",
    "speaker_counts_scott = Counter(i[1] for i in scott_predictions).most_common()\n",
    "for speaker, count in speaker_counts_scott:\n",
    "    print(speaker, count, f\"{count / len(scott_predictions):.2f}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}