Tutorials/DeepLearningForAudio/Deep Learning for Audio Part 1 - Audio Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "6d42e5f8-dbda-4ae7-9d2d-ed20e91c7d4d"
    }
   },
   "source": [
    "# Deep Learning for Audio Part 1 - Audio Processing\n",
    "\n",
    "The purpose of this notebook is to outline how to preprocess audio data into features for machine learning, including for deep learning approaches. We also cover some background on audio processing theory which is needed to perform such featurization.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:100% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# change notebook settings for wider screen\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Featurization\n",
    "\n",
    "Before we can perform data science on audio signals we have to first convert them to a format which is useful - this process is called featurization, i.e. we create features from each sound file. We later combine multiple sound files together into train, validation and test sets used by the machine learning algorithm.\n",
    "\n",
    "### Audio features demystified\n",
    "\n",
    "Audio by itself comes in amplitude representation, where amplitude of the sound changes at different frequencies over time. What we need to do is extract which frequencies are present in each unit of time - those frequencies, when combined, create sounds. Think of playing piano notes - each note resonates at a particular frequency and those frequencies combine to create a particular tune. If we know what notes are being played, we can attempt to classify a particular piano solo. Hence we need a mechanism of breaking down amplitude over time into frequencies over time: such representation is also commonly called a _spectrogram_.\n",
    "\n",
    "Luckily for us, there is a [Fast Fourier Transform algorithm](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (FFT) which does just that: it converts amplitude over each time segment into corresponding frequencies. There is another [Nyquist-Shannon sampling theorem](https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem) which states that if we sample the incoming sound signal at a certain rate, we can achieve what's commonly called **\"lossless\"** audio, i.e. we can convert amplitude into frequencies over time and then recover the original amplitude with no error at any point of time from the broken down frequencies.\n",
    "\n",
    "### Nyquist Theorem and frequencies\n",
    "\n",
    "If the highest frequency component in a signal is f<sub>max</sub>, then the sampling rate must be at least 2f<sub>max</sub>. The higher the highest frequency, the higher the _bandwidth_ of the signal. \n",
    "\n",
    "### Toy Example\n",
    "\n",
    "We simulate the simplest type of audio signal there is - a simple sine wave at a particular frequency. To make things interesting, we generate two such sine waves at different frequencies and add them together - we expect to recover just those two frequencies in the spectrogram.\n",
    "\n",
    "You'll see in the next notebook that real world examples are usually far more complex than what we're showing here - there are multiple overlapping frequencies. Given the Nyquist Theorem, we have to make sure that we sample at at least twice the rate of the highest frequency that we intend to detect in the signal. To do this, the [bitrate](https://en.wikipedia.org/wiki/Bit_rate#Audio) of the sample audio has to be high enough to allow us to do this, i.e. the number of discrete time points per 1 second of audio length. \n",
    "\n",
    "Since we're generating sample audio in this notebook, we get to control all these parameters. However when working with different audio formats you should be aware of all the theory behind audio storage: for example, reduction in bandwitch affects the sampling frequency which in turn affects the maximum frequency (and bandwidth) which you can train on; other audio formats cut out non-dominant frequencies from the file entirely to reduce storage size.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the two frequencies below to whichever number you want - as long as you make sure that the bitrate is higher than the maximum frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get this Jupyter Notebook to work properly, you need to install a few packages. You can run the following command in the terminal:\n",
    "\n",
    "`sudo apt-get install python-pyaudio`\n",
    "\n",
    "`sudo apt-get install libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools`\n",
    "\n",
    "`pip install numpy pyaudio wave librosa scipy matplotlib`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "\n",
    "# in Hertz, so 32kHz\n",
    "bitrate = 32000\n",
    "\n",
    "# note frequencies https://pages.mtu.edu/~suits/notefreqs.html\n",
    "freq1 = 512.\n",
    "freq2 = 1024.\n",
    "assert(freq1 > 0 and freq2 > 0)\n",
    "\n",
    "sound_clip_duration = 1     #seconds to play sound\n",
    "\n",
    "# increase sound quality rate to represent underlying frequencies\n",
    "if max(freq1, freq2) > bitrate:\n",
    "    bitrate = max(freq1, freq2) + 100.\n",
    "\n",
    "# number of time series points in the array\n",
    "n = int(bitrate * sound_clip_duration)\n",
    "# pad the sound time series\n",
    "n_pad = n % bitrate\n",
    "\n",
    "# pad the series\n",
    "x = np.arange(n)\n",
    "# use np.sin instead of vectorizing sin\n",
    "# vector_sin = np.vectorize(lambda arg: math.sin(arg))\n",
    "wave_fun = lambda freq: np.sin(x/((bitrate/freq)/math.pi))\n",
    "# superposition of two frequencies\n",
    "data_array = wave_fun(freq1) + wave_fun(freq2)\n",
    "# renormalize the data (to guarantee sine wave for playback later on)\n",
    "data_array /= np.abs(data_array).max()\n",
    "# pad the array with zeros if needed\n",
    "data_array = np.hstack((data_array, np.zeros(n_pad)))\n",
    "\n",
    "# final sanity check\n",
    "assert(n + n_pad == len(data_array))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've generated the audio stream, we can try playing it (if your machine has audio access - if not, the next section records a .wav file which you can play on any machine). In Windows DSVM this will not throw an error message, and in Linux DSVM the below cell will say something like `No Default Output Device Available` which you can ignore - the test.wav file will still be written to the file system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No PyAudio device available - skip to the next cell to save the audio file and play it from disc.\n"
     ]
    }
   ],
   "source": [
    "# run the following commands if PyAudio is not installed on your machine\n",
    "# sudo apt-get install python-pyaudio\n",
    "# sudo apt install libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools\n",
    "# pip install pyaudio\n",
    "import pyaudio\n",
    "import sys\n",
    "PyAudio = pyaudio.PyAudio     #initialize pyaudio\n",
    "\n",
    "data = ''\n",
    "#generating waves\n",
    "for i in range(n + n_pad):    \n",
    "    data += chr(int(data_array[i]*127+128))\n",
    "\n",
    "try:\n",
    "    p = PyAudio()\n",
    "    stream = p.open(format = p.get_format_from_width(1), \n",
    "                channels = 1, \n",
    "                rate = bitrate, \n",
    "                output = True)\n",
    "\n",
    "    stream.write(data)\n",
    "    stream.stop_stream()\n",
    "    stream.close()\n",
    "    p.terminate()\n",
    "\n",
    "except OSError:\n",
    "    print (\"No PyAudio device available - skip to the next cell to save the audio file and play it from disc.\")\n",
    "except:\n",
    "    print(\"Unknown Error:\", sys.exc_info()[0])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "[Errno -9996] Invalid output device (no default output device)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-066fbedd1d15>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m                 \u001b[0mchannels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m                 \u001b[0mrate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbitrate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m                 output = True)\n\u001b[0m",
      "\u001b[0;32m~/python3env/lib/python3.5/site-packages/pyaudio.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    748\u001b[0m         \"\"\"\n\u001b[1;32m    749\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 750\u001b[0;31m         \u001b[0mstream\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mStream\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    751\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_streams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    752\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/python3env/lib/python3.5/site-packages/pyaudio.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, PA_manager, rate, channels, format, input, output, input_device_index, output_device_index, frames_per_buffer, start, input_host_api_specific_stream_info, output_host_api_specific_stream_info, stream_callback)\u001b[0m\n\u001b[1;32m    439\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    440\u001b[0m         \u001b[0;31m# calling pa.open returns a stream object\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 441\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stream\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpa\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    442\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    443\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_input_latency\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stream\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minputLatency\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: [Errno -9996] Invalid output device (no default output device)"
     ]
    }
   ],
   "source": [
    "stream = p.open(format = p.get_format_from_width(1), \n",
    "                channels = 1, \n",
    "                rate = bitrate, \n",
    "                output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with different audio formats\n",
    "\n",
    "Not all audio formats are lossless - as described in the previous section, some try to reduce the overall file size by the use of algorithms which may intentionally degrade sound quality, thereby affecting the featurization and the quality of your predictions on the test set. For example, one common approach that can reduce the size of the audio file is to eliminate non-dominant frequencies and store the file as a sparser spectrogram - the decoder converts the sparser spectrogram back into audio waveform when playing the file. One can further reduce file size by downsampling the original waveform before computing the spectrogram - as we know from the previous section, both approaches will degrade sound quality and some events might become undetectable from the audio data (no matter which algorithm you use).\n",
    "\n",
    "We detail how to work with .wav file format in this notebook, which is the most common lossless file format for audio work. We recommend you convert any other file format to .wav if you want to re-apply this notebook to other datasets on DS VM.\n",
    "\n",
    "[Python Audio Tools](http://audiotools.sourceforge.net/install.html) provide a great way to handle conversions between various file formats (and can even rip entire CDs of music).\n",
    "\n",
    "We use the wave library to save the python audio stream as a sequence of byte frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wave\n",
    "import os\n",
    "test_fname = os.getcwd() + \"/test.wav\"\n",
    "print (\"Saving file to \" + test_fname)\n",
    "\n",
    "wave_file = wave.open(test_fname, 'wb')\n",
    "# mono audio\n",
    "wave_file.setnchannels(1)\n",
    "wave_file.setsampwidth(p.get_sample_size(pyaudio.paInt16))\n",
    "wave_file.setframerate(bitrate)\n",
    "wave_file.writeframesraw(bytes(data, 'UTF-8'))\n",
    "wave_file.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now download the sample audio file and play it on your computer.\n",
    "\n",
    "Next, we need to perform the FFT decomposition and recover the original frequencies which we've introduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this has to be at least twice of max frequency which we've entered\n",
    "# but you can play around with different sample rates and see how this\n",
    "# affects the results;\n",
    "# since we generated this audio, the sample rate is the bitrate\n",
    "sample_rate = bitrate\n",
    "\n",
    "# size of audio FFT window relative to sample_rate\n",
    "n_window = 1024\n",
    "# overlap between adjacent FFT windows\n",
    "n_overlap = 360\n",
    "# number of mel frequency bands to generate\n",
    "n_mels = 64\n",
    "\n",
    "# fmin and fmax for librosa filters in Hz - used for visualization purposes only\n",
    "fmax = max(freq1, freq2) + 1000.\n",
    "fmin = 0.\n",
    "\n",
    "# stylistic change to the notebook\n",
    "fontsize = 14\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The section below visualizes the sound which we generated (and saved to disk), plots the mel-scaled FFT (also known as [Mel spectrogram](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) or Short Mel FFT) and then plots an alternative spectrogram in log-domain which we use for featurization for the neural network.\n",
    "\n",
    "#### Wave plot\n",
    "\n",
    "This shows two normalized and added sine waves which we generated. You see predominantly two oscillations which we've introduced - pretty simple.\n",
    "\n",
    "#### Mel spectrogram\n",
    "\n",
    "This is simply the FFT of each audio window mapped to [mel scale](https://en.wikipedia.org/wiki/Mel_scale), which perceptually makes pitches to be of equal distance from one another (human ear focuses on certain frequencies, so our perception is that the mel frequencies are of equal distance from each other and not the raw FFT frequencies) - think of changing the FFT frequency scale to make sure all features (frequencies) are equidistant. __Here you can clearly see that we recovered the original frequencies which we entered at the beginning of this notebook with no loss__.\n",
    "\n",
    "We need to add a few extra steps to use the spectrogram for [audio event detection](http://www.cs.tut.fi/sgn/arg/dcase2017/index):\n",
    "- after mapping the frequencies to mel-scale, we use the [Hamming window](https://en.wikipedia.org/wiki/Window_function#Hamming_window) in the FFT: the assumption is that the time domain signal is periodic which results in discontinuity at the edges of the FFT window. Window functions are designed to avoid this, by making sure that the data at the edges is zero (no discontinuity). This is achieved by multiplying the signal by the window function (Hamming in this case) which gradually decays the signal towards zero at the edges.\n",
    " - we use a mel filter matrix to combine FFT bins into mel frequency bins: this step is equivalent to the previous step where use used a canned _librosa_ routine, given the same windowing function is used by librosa\n",
    " - we further make sure that there are no numerical problems in taking the logarithm of the FFT by adding a small positive number (offset) to the FFT before taking the log. The logarithm of the mel-scaled spectrogram is used directly for acoustic event detection.\n",
    " \n",
    "This is the raw featurization which is needed to detect acoustic events in audio (e.g. dog barking, water boiling, tires screeching), which is our ultimate goal. For human speech recognition systems this featurization is not usually used directly.\n",
    "\n",
    "#### Speech recognition approaches\n",
    "\n",
    "In some applications such as sound event detection, speech recognition, and speaker detection, human brains tend to focus on lower-frequency patterns in audio, so the featurization approach needs to go a step further and compute [Mel Frequency Cepstral Coefficients (MFCC)](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum). The idea is that we we want to fit an envelope function across the log of the mel-scaled frequencies. To do this, we use the [Discrete Cosine Transform (DCT)](https://en.wikipedia.org/wiki/Discrete_cosine_transform):\n",
    " - we compute the DCT of the log of mel-scaled frequency spectrogram\n",
    " - we then use Cepstral Analysis to focus on the lower frequencies, which give us the envelope and the MFCCs which we then directly train on for speech recognition - MFCCs are the amplitudes of the resulting spectrum\n",
    " \n",
    "Basically the log mel frequency spectrogram can be thought of as the sum of the spectral envelope (slow and steady function) and spectral details (remaining residual). Since humans tend to focus on patterns in lower frequencies, we are more interested in the former - the spectral envelope. Because we have access to the sum and not each element, we use the DCT to obtain the cepstrum and then focus on the coefficients which are responsible for modelling lower frequencies - more details are provided [here](http://www.speech.cs.cmu.edu/15-492/slides/03_mfcc.pdf).\n",
    "\n",
    "It should also be noted that there are other approaches for speech recognition - in the next notebook we actually obtain great performance on speech commands dataset without the use of MFCCs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set plotting parameters\n",
    "fontsize_blog = 48\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "# third-party sounds processing and visualization library\n",
    "import librosa\n",
    "import librosa.display\n",
    "# signal processing library\n",
    "from scipy import signal\n",
    "\n",
    "plt.rcParams['font.family'] = 'serif'\n",
    "plt.rcParams['font.serif'] = 'Ubuntu'\n",
    "plt.rcParams['font.monospace'] = 'Ubuntu Mono'\n",
    "plt.rcParams['font.size'] = fontsize\n",
    "plt.rcParams['axes.labelsize'] = fontsize\n",
    "plt.rcParams['axes.labelweight'] = 'bold'\n",
    "plt.rcParams['axes.titlesize'] = fontsize\n",
    "plt.rcParams['xtick.labelsize'] = fontsize\n",
    "plt.rcParams['ytick.labelsize'] = fontsize\n",
    "plt.rcParams['legend.fontsize'] = fontsize\n",
    "plt.rcParams['figure.titlesize'] = fontsize\n",
    "\n",
    "# Make a new figure\n",
    "plt.figure(figsize=(128, 16), dpi= 80, facecolor='w', edgecolor='k')\n",
    "plt.subplot(131)\n",
    "# display the raw waveform\n",
    "librosa.display.waveplot(data_array, int(sample_rate), max_sr = int(sample_rate))\n",
    "plt.title('Raw audio waveform with %d Hz bitrate sampled at @ %d Hz' % (int(bitrate), int(sample_rate)), fontsize = fontsize_blog)\n",
    "plt.xlabel(\"Time (s)\", fontsize = fontsize_blog)\n",
    "plt.ylabel(\"Amplitude\", fontsize = fontsize_blog)\n",
    "\n",
    "plt.subplot(132)\n",
    "S = librosa.feature.melspectrogram(data_array, sr = sample_rate, n_mels=n_mels, fmin = fmin, fmax = fmax)\n",
    "#S = librosa.feature.spectral.fft_frequencies(data_array, sample_rate)\n",
    "# Convert to log scale (dB). We'll use the peak power as reference.\n",
    "# log_S = librosa.logamplitude(S, ref_power=np.max)\n",
    "# Display the spectrogram on a mel scale\n",
    "# sample rate and hop length parameters are used to render the time axis\n",
    "librosa.display.specshow(S, sr = sample_rate, x_axis = 'time', y_axis='mel',\\\n",
    "                         x_coords=np.linspace(0, 1, S.shape[1]))\n",
    "# optional colorbar plot\n",
    "plt.colorbar(format='%+02.0f')\n",
    "plt.title('Un-normalized librosa Mel spectrogram', fontsize = fontsize_blog)\n",
    "plt.xlabel(\"Time (s)\", fontsize = fontsize_blog)\n",
    "plt.ylabel(\"Hz\", fontsize = fontsize_blog)\n",
    "\n",
    "plt.subplot(133)\n",
    "melW = librosa.filters.mel(sr=sample_rate, n_fft=n_window, n_mels=64, fmin=fmin, fmax=fmax)\n",
    "ham_win = np.hamming(n_window)\n",
    "[f, t, x] = signal.spectral.spectrogram(\n",
    "    x=data_array,\n",
    "    window=ham_win,\n",
    "    nperseg=n_window,\n",
    "    noverlap=n_overlap,\n",
    "    detrend=False,\n",
    "    return_onesided=True,\n",
    "    mode='magnitude')\n",
    "x = np.dot(x.T, melW.T)\n",
    "x = np.log(x + 1e-8)\n",
    "x = x.astype(np.float32)\n",
    "librosa.display.specshow(x.T, sr=sample_rate, x_axis='time', y_axis='mel', x_coords=np.linspace(0, 1, x.shape[0]))\n",
    "plt.xlabel(\"Time (s)\", fontsize = fontsize_blog)\n",
    "plt.ylabel(\"Hz\", fontsize = fontsize_blog)\n",
    "plt.title(\"Mel power spectrogram used in DCASE 2017 (dB)\", fontsize = fontsize_blog)\n",
    "# optional colorbar plot\n",
    "plt.colorbar(format='%+02.0f dB')\n",
    "\n",
    "# Make the figure layout compact\n",
    "# plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "66409e63-e6e2-4d97-b63f-9fd3860d60c3"
    }
   },
   "source": [
    "## Featurization on DS VM\n",
    "\n",
    "The next section shows how to handle input .wav files and featurize them to useful spectrograms. There are two plots:\n",
    "1. Shows the original amplitude plot of the .wav file. The Urban Sounds data has already been converted to mono sound representation, so instead of two amplitudes over time (stereo) we get only one single audio amplitude (sound) as a function of time over a 1-second interval.\n",
    "2. We re-apply the approach from the winning DCASE 2016 Task 4 solution - this shows the approach which is used to featurize the data for machine learning in the rest of the notebook. As you can see, the featurized dataset isn't very different from the out-of-the-box solution provided by generic application of the librosa library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "930ce9ea-c567-4778-b15d-e00ecd7af298"
    }
   },
   "outputs": [],
   "source": [
    "audio_path = os.getcwd() + \"/data/zero_0.wav\"\n",
    "sample_rate = 16000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# audio is a 1D time series of the sound\n",
    "# can also use (audio, fs) = soundfile.read(audio_path)\n",
    "(audio, fs) = librosa.load(audio_path, sr = None, duration = 1)\n",
    "# check that native bitrate matches our assumed sample rate\n",
    "assert(int(fs) == int(sample_rate))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "9c3cc1cb-356a-411b-8492-135bbc43f97a"
    }
   },
   "outputs": [],
   "source": [
    "# Make a new figure\n",
    "plt.figure(figsize=(18, 16), dpi= 60, facecolor='w', edgecolor='k')\n",
    "plt.subplot(211)\n",
    "# Display the spectrogram on a mel scale\n",
    "librosa.display.waveplot(audio, int(sample_rate), max_sr = int(sample_rate))\n",
    "plt.title('Raw audio waveform @ %d Hz' % sample_rate, fontsize = fontsize) \n",
    "plt.xlabel(\"Time (s)\")\n",
    "plt.ylabel(\"Amplitude\")\n",
    "\n",
    "plt.subplot(212)\n",
    "melW =librosa.filters.mel(sr=sample_rate, n_fft=n_window, n_mels=n_mels, fmin=fmin, fmax=fmax)\n",
    "ham_win = np.hamming(n_window)\n",
    "[f, t, x] = signal.spectral.spectrogram(\n",
    "    x=audio,\n",
    "    window=ham_win,\n",
    "    nperseg=n_window,\n",
    "    noverlap=n_overlap,\n",
    "    detrend=False,\n",
    "    return_onesided=True,\n",
    "    mode='magnitude')\n",
    "x = np.dot(x.T, melW.T)\n",
    "x = np.log(x + 1e-8)\n",
    "x = x.astype(np.float32)\n",
    "librosa.display.specshow(x.T, sr=sample_rate, x_axis='time', y_axis='mel', x_coords=np.linspace(0, 1, x.shape[0]))\n",
    "plt.xlabel(\"Time (s)\")\n",
    "plt.title(\"Mel power spectrogram used in DCASE 2017 (dB)\", fontsize = fontsize)\n",
    "# optional colorbar plot\n",
    "plt.colorbar(format='%+02.0f dB')\n",
    "\n",
    "# Make the figure layout compact\n",
    "# plt.tight_layout()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "nbpresent": {
   "slides": {
    "0579c4f1-93c1-4c53-a6ea-499cce4a986f": {
     "id": "0579c4f1-93c1-4c53-a6ea-499cce4a986f",
     "prev": "daac3759-c6b6-45a9-9999-bc86c0641528",
     "regions": {
      "8ef90291-271a-4696-9d7c-5ed111272697": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "9c3cc1cb-356a-411b-8492-135bbc43f97a",
        "part": "whole"
       },
       "id": "8ef90291-271a-4696-9d7c-5ed111272697"
      }
     }
    },
    "08c1da6e-a649-4aeb-81ec-548af9a7839e": {
     "id": "08c1da6e-a649-4aeb-81ec-548af9a7839e",
     "prev": "7268d961-6a7a-4085-a41d-b4ff29be7624",
     "regions": {
      "12da4945-3a6d-48a4-b692-aec49288db6a": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "930ce9ea-c567-4778-b15d-e00ecd7af298",
        "part": "whole"
       },
       "id": "12da4945-3a6d-48a4-b692-aec49288db6a"
      }
     }
    },
    "212782cc-97cb-4646-bd77-a4c58be15243": {
     "id": "212782cc-97cb-4646-bd77-a4c58be15243",
     "prev": "b3a8bb8b-0a44-4cbf-97cc-fc0f2ec2bb08",
     "regions": {
      "049524de-de30-438d-bd7b-2bf5f85dd400": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "2ff9b5a1-a527-4c8f-988d-00a3e5824000",
        "part": "whole"
       },
       "id": "049524de-de30-438d-bd7b-2bf5f85dd400"
      }
     }
    },
    "28178124-f27a-4120-9cdf-ce87086a7f7d": {
     "id": "28178124-f27a-4120-9cdf-ce87086a7f7d",
     "prev": "0579c4f1-93c1-4c53-a6ea-499cce4a986f",
     "regions": {
      "d442229d-8e62-4295-bd97-f27ebce650b8": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "2ff9b5a1-a527-4c8f-988d-00a3e5824000",
        "part": "whole"
       },
       "id": "d442229d-8e62-4295-bd97-f27ebce650b8"
      }
     }
    },
    "2bb56f92-7fb3-4f2e-9088-1ab04cf11358": {
     "id": "2bb56f92-7fb3-4f2e-9088-1ab04cf11358",
     "prev": "9bfc47c5-2590-4161-a2e6-e0af644e3f23",
     "regions": {
      "f4c33145-57e6-48ec-a18f-2247de100faa": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "48316aba-a15f-4366-b9c9-a6da91e26226",
        "part": "whole"
       },
       "id": "f4c33145-57e6-48ec-a18f-2247de100faa"
      }
     }
    },
    "2d6f82a4-33fd-4732-a8fe-f9f5df47db1c": {
     "id": "2d6f82a4-33fd-4732-a8fe-f9f5df47db1c",
     "prev": "28178124-f27a-4120-9cdf-ce87086a7f7d",
     "regions": {
      "19d29332-bce0-40e6-8498-554009bbe7fc": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "274ea58f-2e30-4946-8586-b2099f0ff74f",
        "part": "whole"
       },
       "id": "19d29332-bce0-40e6-8498-554009bbe7fc"
      }
     }
    },
    "5feabd9c-2b31-4304-bdd8-df5fb4e441fa": {
     "id": "5feabd9c-2b31-4304-bdd8-df5fb4e441fa",
     "prev": "99946a3a-c2d7-472c-9e63-118117f216df",
     "regions": {
      "ca306aa3-1692-4ecd-b063-89414ca4b1a8": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "1cbba0ae-2a23-4a9c-992f-e9601db2f8b8",
        "part": "whole"
       },
       "id": "ca306aa3-1692-4ecd-b063-89414ca4b1a8"
      }
     }
    },
    "6993b049-b4b2-4aa8-bef3-bbe695515f9d": {
     "id": "6993b049-b4b2-4aa8-bef3-bbe695515f9d",
     "prev": "70e66fdf-da61-4fd7-b1a7-2432ed92f7ab",
     "regions": {
      "37cca9dc-1d03-4736-af08-76007e500da2": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "3f54d560-e8c3-4ead-88b6-7ae323c10709",
        "part": "whole"
       },
       "id": "37cca9dc-1d03-4736-af08-76007e500da2"
      }
     }
    },
    "70e66fdf-da61-4fd7-b1a7-2432ed92f7ab": {
     "id": "70e66fdf-da61-4fd7-b1a7-2432ed92f7ab",
     "prev": "5feabd9c-2b31-4304-bdd8-df5fb4e441fa",
     "regions": {
      "5eb9c5a1-a6ed-4790-b5e0-03a3a4672204": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "70ca7992-be9b-43f1-ba0f-de825860af99",
        "part": "whole"
       },
       "id": "5eb9c5a1-a6ed-4790-b5e0-03a3a4672204"
      }
     }
    },
    "7268d961-6a7a-4085-a41d-b4ff29be7624": {
     "id": "7268d961-6a7a-4085-a41d-b4ff29be7624",
     "prev": "2bb56f92-7fb3-4f2e-9088-1ab04cf11358",
     "regions": {
      "4e8862c4-f268-4c9b-beb3-53efb26a411f": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "32f230a2-c644-4581-b2dc-0941cf9c75b6",
        "part": "whole"
       },
       "id": "4e8862c4-f268-4c9b-beb3-53efb26a411f"
      }
     }
    },
    "764d0c2b-88d0-4853-b54b-714ebe85212c": {
     "id": "764d0c2b-88d0-4853-b54b-714ebe85212c",
     "prev": "9b5e2607-093b-4796-826d-f510fcf748a4",
     "regions": {
      "5655168e-2467-48c7-9714-81c96ca2254e": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "533a3954-b6c2-44fe-b62e-d8a146c2ebe8",
        "part": "whole"
       },
       "id": "5655168e-2467-48c7-9714-81c96ca2254e"
      }
     }
    },
    "99946a3a-c2d7-472c-9e63-118117f216df": {
     "id": "99946a3a-c2d7-472c-9e63-118117f216df",
     "prev": "2d6f82a4-33fd-4732-a8fe-f9f5df47db1c",
     "regions": {
      "ef1890a6-de94-4c49-8357-f9c9f2861f74": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "533a3954-b6c2-44fe-b62e-d8a146c2ebe8",
        "part": "whole"
       },
       "id": "ef1890a6-de94-4c49-8357-f9c9f2861f74"
      }
     }
    },
    "9b5e2607-093b-4796-826d-f510fcf748a4": {
     "id": "9b5e2607-093b-4796-826d-f510fcf748a4",
     "prev": "212782cc-97cb-4646-bd77-a4c58be15243",
     "regions": {
      "cebf1367-e975-4ac6-8413-81eaef7b9231": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "274ea58f-2e30-4946-8586-b2099f0ff74f",
        "part": "whole"
       },
       "id": "cebf1367-e975-4ac6-8413-81eaef7b9231"
      }
     }
    },
    "9bfc47c5-2590-4161-a2e6-e0af644e3f23": {
     "id": "9bfc47c5-2590-4161-a2e6-e0af644e3f23",
     "prev": "dcc6465d-176f-42a8-ae6c-5fa531242607",
     "regions": {
      "84b322da-1b3b-414c-8be0-5a02788c201a": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "66409e63-e6e2-4d97-b63f-9fd3860d60c3",
        "part": "whole"
       },
       "id": "84b322da-1b3b-414c-8be0-5a02788c201a"
      }
     }
    },
    "aff55c0d-6544-45ed-912b-68df537f6311": {
     "id": "aff55c0d-6544-45ed-912b-68df537f6311",
     "prev": "f2a1baa3-64d9-4fad-9c44-ba2f34bad758",
     "regions": {
      "7529ed8b-c15b-4d23-a982-c6709abb7d6f": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "66409e63-e6e2-4d97-b63f-9fd3860d60c3",
        "part": "whole"
       },
       "id": "7529ed8b-c15b-4d23-a982-c6709abb7d6f"
      }
     }
    },
    "b3a8bb8b-0a44-4cbf-97cc-fc0f2ec2bb08": {
     "id": "b3a8bb8b-0a44-4cbf-97cc-fc0f2ec2bb08",
     "prev": "08c1da6e-a649-4aeb-81ec-548af9a7839e",
     "regions": {
      "2aec1831-bc17-4242-8432-01b11ecc2f33": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "9c3cc1cb-356a-411b-8492-135bbc43f97a",
        "part": "whole"
       },
       "id": "2aec1831-bc17-4242-8432-01b11ecc2f33"
      }
     }
    },
    "c10b6670-a23e-4ca3-93b9-77d14d557c92": {
     "id": "c10b6670-a23e-4ca3-93b9-77d14d557c92",
     "prev": "fac9b98d-2107-4379-936a-8bf5baa5347c",
     "regions": {
      "79a9690f-a245-423e-8ce1-108f37f0b05d": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "3f54d560-e8c3-4ead-88b6-7ae323c10709",
        "part": "whole"
       },
       "id": "79a9690f-a245-423e-8ce1-108f37f0b05d"
      }
     }
    },
    "ce2af2ed-e637-48ed-a07f-ce8e92a55306": {
     "id": "ce2af2ed-e637-48ed-a07f-ce8e92a55306",
     "prev": "aff55c0d-6544-45ed-912b-68df537f6311",
     "regions": {
      "014928ff-04c2-4597-b6a5-56e39dea28f9": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "32f230a2-c644-4581-b2dc-0941cf9c75b6",
        "part": "whole"
       },
       "id": "014928ff-04c2-4597-b6a5-56e39dea28f9"
      }
     }
    },
    "daac3759-c6b6-45a9-9999-bc86c0641528": {
     "id": "daac3759-c6b6-45a9-9999-bc86c0641528",
     "prev": "ce2af2ed-e637-48ed-a07f-ce8e92a55306",
     "regions": {
      "11362faa-aff3-49d4-945e-63b619327112": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "930ce9ea-c567-4778-b15d-e00ecd7af298",
        "part": "whole"
       },
       "id": "11362faa-aff3-49d4-945e-63b619327112"
      }
     }
    },
    "dcc6465d-176f-42a8-ae6c-5fa531242607": {
     "id": "dcc6465d-176f-42a8-ae6c-5fa531242607",
     "prev": null,
     "regions": {
      "57c67d73-a546-4476-990d-fe1d3ff7cad1": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "6d42e5f8-dbda-4ae7-9d2d-ed20e91c7d4d",
        "part": "whole"
       },
       "id": "57c67d73-a546-4476-990d-fe1d3ff7cad1"
      }
     }
    },
    "f2a1baa3-64d9-4fad-9c44-ba2f34bad758": {
     "id": "f2a1baa3-64d9-4fad-9c44-ba2f34bad758",
     "prev": null,
     "regions": {
      "e9d276ab-2c96-4f25-840d-8a9d605685dc": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "6d42e5f8-dbda-4ae7-9d2d-ed20e91c7d4d",
        "part": "whole"
       },
       "id": "e9d276ab-2c96-4f25-840d-8a9d605685dc"
      }
     }
    },
    "f4ef0435-8a10-4dc9-b39e-56fafce0f65a": {
     "id": "f4ef0435-8a10-4dc9-b39e-56fafce0f65a",
     "prev": "764d0c2b-88d0-4853-b54b-714ebe85212c",
     "regions": {
      "2b9153ca-e580-4fbe-b7f9-80285673b946": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "1cbba0ae-2a23-4a9c-992f-e9601db2f8b8",
        "part": "whole"
       },
       "id": "2b9153ca-e580-4fbe-b7f9-80285673b946"
      }
     }
    },
    "fac9b98d-2107-4379-936a-8bf5baa5347c": {
     "id": "fac9b98d-2107-4379-936a-8bf5baa5347c",
     "prev": "f4ef0435-8a10-4dc9-b39e-56fafce0f65a",
     "regions": {
      "e75a9597-ab17-45fa-8c7d-f76235e75f9b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "70ca7992-be9b-43f1-ba0f-de825860af99",
        "part": "whole"
       },
       "id": "e75a9597-ab17-45fa-8c7d-f76235e75f9b"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}