2-Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model\n",
    "In this notebook we'll put together the ideas for building a model for training our voice embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import regularizers, optimizers\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Audio Trigger Model\n",
    "Here's the model I used in a class assignment building a model for detetecting a trigger word (\"activate\") from audio samples. This model was used to monitor an audio channel for the trigger word, and output $\\hat{y}$ at each time step with 1 if the trigger word had just finished being spoken, and 0 otherwise.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't use this exact model.  This is because the $\\hat{y}$ predictions in that model were one per time step.\n",
    "\n",
    "Instead what we want is a single $\\hat{y}$ of shape ${n}$ where ${n}$ is the length of the desired word embedding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Here is the model we'll use for producing an embedding for an audio clip:\n",
    "\n",
    "![Model](images/model.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    " * Architecture diagram drawn with draw.io\n",
    " * Trigger Word Network architecture from Course 4 (sequential nets) week 4 Deep Learning Specialization, Sequential Models, Deep Learning.ai, Coursera"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Triplet Loss on Images\n",
    "\n",
    "For an image $x$, we denote its encoding $f(x)$, where $f$ is the function computed by the neural network.\n",
    "\n",
    "<!--\n",
    "We will also add a normalization step at the end of our model so that $\\mid \\mid f(x) \\mid \\mid_2 = 1$ (means the vector of encoding should be of norm 1).\n",
    "!-->\n",
    "\n",
    "Training will use triplets of images $(A, P, N)$:  \n",
    "\n",
    "- A is an \"Anchor\" image--a picture of a person. \n",
    "- P is a \"Positive\" image--a picture of the same person as the Anchor image.\n",
    "- N is a \"Negative\" image--a picture of a different person than the Anchor image.\n",
    "\n",
    "These triplets are picked from our training dataset. We will write $(A^{(i)}, P^{(i)}, N^{(i)})$ to denote the $i$-th training example. \n",
    "\n",
    "You'd like to make sure that an image $A^{(i)}$ of an individual is closer to the Positive $P^{(i)}$ than to the Negative image $N^{(i)}$) by at least a margin $\\alpha$:\n",
    "\n",
    "$$\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 + \\alpha < \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$$\n",
    "\n",
    "You would thus like to minimize the following \"triplet cost\":\n",
    "\n",
    "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\underbrace{\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2}_\\text{(1)} - \\underbrace{\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2}_\\text{(2)} + \\alpha \\large ] \\small_+ \\tag{3}$$\n",
    "\n",
    "Here, we are using the notation \"$[z]_+$\" to denote $max(z,0)$.  \n",
    "\n",
    "Notes:\n",
    "- The term (1) is the squared distance between the anchor \"A\" and the positive \"P\" for a given triplet; you want this to be small. \n",
    "- The term (2) is the squared distance between the anchor \"A\" and the negative \"N\" for a given triplet, you want this to be relatively large. It has a minus sign preceding it because minimizing the negative of the term is the same as maximizing that term.\n",
    "- $\\alpha$ is called the margin. It is a hyperparameter that you pick manually. We will use $\\alpha = 0.2$. \n",
    "\n",
    "Most implementations also rescale the encoding vectors to haven L2 norm equal to one (i.e., $\\mid \\mid f(img)\\mid \\mid_2$=1); you won't have to worry about that in this assignment.\n",
    "\n",
    "**Exercise**: Implement the triplet loss as defined by formula (3). Here are the 4 steps:\n",
    "1. Compute the distance between the encodings of \"anchor\" and \"positive\": $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$\n",
    "2. Compute the distance between the encodings of \"anchor\" and \"negative\": $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$\n",
    "3. Compute the formula per training example: $ \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2 + \\alpha$\n",
    "3. Compute the full formula by taking the max with zero and summing over the training examples:\n",
    "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 - \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2+ \\alpha \\large ] \\small_+ \\tag{3}$$\n",
    "\n",
    "#### Hints\n",
    "* Useful functions: `tf.reduce_sum()`, `tf.square()`, `tf.subtract()`, `tf.add()`, `tf.maximum()`.\n",
    "* For steps 1 and 2, you will sum over the entries of $\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2$ and $\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$.  \n",
    "* For step 4 you will sum over the training examples.\n",
    "\n",
    "#### Additional Hints\n",
    "* Recall that the square of the L2 norm is the sum of the squared differences: $||x - y||_{2}^{2} = \\sum_{i=1}^{N}(x_{i} - y_{i})^{2}$\n",
    "* Note that the `anchor`, `positive` and `negative` encodings are of shape `(m,128)`, where m is the number of training examples and 128 is the number of elements used to encode a single example.\n",
    "* For steps 1 and 2, you will maintain the number of `m` training examples and sum along the 128 values of each encoding. \n",
    "[tf.reduce_sum](https://www.tensorflow.org/api_docs/python/tf/math/reduce_sum) has an `axis` parameter.  This chooses along which axis the sums are applied.  \n",
    "* Note that one way to choose the last axis in a tensor is to use negative indexing (`axis=-1`).\n",
    "* In step 4, when summing over training examples, the result will be a single scalar value.\n",
    "* For `tf.reduce_sum` to sum across all axes, keep the default value `axis=None`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simpler CNN Model\n",
    "\n",
    "My research indicates that convolutional neural networks can perform many tasks pretty well if the samples are small.   CNN's train faster than RNN's using GRU's or LSTMs. So maybe we can succeed with a simpler CNN model.  We'll be using a sliding window of about 1 second of audio, so maybe we can succeed with a CNN model.\n",
    "\n",
    "Let's try it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to determine the shape we want for the inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 64, 64, 32)        896       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 64, 64, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 62, 62, 64)        18496     \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 62, 62, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 31, 31, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 31, 31, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 31, 31, 64)        36928     \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 31, 31, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 29, 29, 64)        36928     \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 29, 29, 64)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 14, 14, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 14, 14, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 14, 14, 128)       73856     \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 14, 14, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 12, 12, 128)       147584    \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 4608)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 512)               2359808   \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 2,679,626\n",
      "Trainable params: 2,679,626\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "def create_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, (3, 3), padding='same',\n",
    "                     input_shape=(64,64,3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(64, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(64, (3, 3), padding='same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(64, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Conv2D(128, (3, 3), padding='same'))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Conv2D(128, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(512))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    model.compile(optimizers.rmsprop(lr=0.0005, decay=1e-6),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])\n",
    "    model.summary()\n",
    "\n",
    "model = create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I want to just get a 1 second window connected to a neural net and have it do computations.\n",
    "\n",
    "Then after that I'll get an RNN connected to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1, 2, 3],\n",
       "        [4, 5, 6]]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a=np.array([[[1,2,3],[4,5,6]]])\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Squeeze_1:0' shape=(2, 3) dtype=int32>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras.backend as K\n",
    "K.squeeze(a, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}