labs/03_neural_recsys/Short_Intro_to_Embeddings_with_Keras_rendered.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Categorical Embeddings\n",
    "\n",
    "We will use the embeddings through the whole lab. They are simply represented by a matrix of tunable parameters (weights).\n",
    "\n",
    "Let us assume that we are given a pre-trained embedding matrix for an vocabulary of size 10. Each embedding vector in that matrix has dimension 4. Those dimensions are too small to be realistic and are only used for demonstration purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  1.  2.  3.]\n",
      " [ 4.  5.  6.  7.]\n",
      " [ 8.  9. 10. 11.]\n",
      " [12. 13. 14. 15.]\n",
      " [16. 17. 18. 19.]\n",
      " [20. 21. 22. 23.]\n",
      " [24. 25. 26. 27.]\n",
      " [28. 29. 30. 31.]\n",
      " [32. 33. 34. 35.]\n",
      " [36. 37. 38. 39.]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "embedding_size = 4\n",
    "vocab_size = 10\n",
    "\n",
    "embedding_matrix = np.arange(embedding_size * vocab_size, dtype='float32')\n",
    "embedding_matrix = embedding_matrix.reshape(vocab_size, embedding_size)\n",
    "print(embedding_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the embedding for a given integer (ordinal) symbol $i$, you may either:\n",
    " - simply index (slice) the embedding matrix by $i$, using numpy integer indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12. 13. 14. 15.]\n"
     ]
    }
   ],
   "source": [
    "i = 3\n",
    "print(embedding_matrix[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - compute a one-hot encoding vector $\\mathbf{v}$ of $i$, then compute a dot product with the embedding matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "def onehot_encode(dim, label):\n",
    "    return np.eye(dim)[label]\n",
    "\n",
    "\n",
    "onehot_i = onehot_encode(vocab_size, i)\n",
    "print(onehot_i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12. 13. 14. 15.]\n"
     ]
    }
   ],
   "source": [
    "embedding_vector = np.dot(onehot_i, embedding_matrix)\n",
    "print(embedding_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Embedding layer in Keras\n",
    "\n",
    "In Keras, embeddings have an extra parameter, `input_length` which is typically used when having a sequence of symbols as input (think sequence of words). In our case, the length will always be 1.\n",
    "\n",
    "```py\n",
    "Embedding(output_dim=embedding_size, input_dim=vocab_size,\n",
    "          input_length=sequence_length, name='my_embedding')\n",
    "```\n",
    "\n",
    "furthermore, we load the fixed weights from the previous matrix instead of using a random initialization:\n",
    "\n",
    "```py\n",
    "Embedding(output_dim=embedding_size, input_dim=vocab_size,\n",
    "          weights=[embedding_matrix],\n",
    "          input_length=sequence_length, name='my_embedding')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "embedding_layer = Embedding(\n",
    "    output_dim=embedding_size, input_dim=vocab_size,\n",
    "    weights=[embedding_matrix],\n",
    "    input_length=1, name='my_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use it as part of a Keras model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Input\n",
    "from tensorflow.keras.models import Model\n",
    "\n",
    "x = Input(shape=[1], name='input')\n",
    "embedding = embedding_layer(x)\n",
    "model = Model(inputs=x, outputs=embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of an embedding layer is then a 3-d tensor of shape `(batch_size, sequence_length, embedding_size)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, 1, 4)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.output_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`None` is a marker for dynamic dimensions.\n",
    "\n",
    "The embedding weights can be retrieved as model parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[ 0.,  1.,  2.,  3.],\n",
       "        [ 4.,  5.,  6.,  7.],\n",
       "        [ 8.,  9., 10., 11.],\n",
       "        [12., 13., 14., 15.],\n",
       "        [16., 17., 18., 19.],\n",
       "        [20., 21., 22., 23.],\n",
       "        [24., 25., 26., 27.],\n",
       "        [28., 29., 30., 31.],\n",
       "        [32., 33., 34., 35.],\n",
       "        [36., 37., 38., 39.]], dtype=float32)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `model.summary()` method gives the list of trainable parameters per layer in the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input (InputLayer)           [(None, 1)]               0         \n",
      "_________________________________________________________________\n",
      "my_embedding (Embedding)     (None, 1, 4)              40        \n",
      "=================================================================\n",
      "Total params: 40\n",
      "Trainable params: 40\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `predict` method of the Keras embedding model to project a single integer label into the matching embedding vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[12., 13., 14., 15.]]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_to_encode = np.array([[3]])\n",
    "model.predict(labels_to_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same for a batch of integers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[12., 13., 14., 15.]],\n",
       "\n",
       "       [[12., 13., 14., 15.]],\n",
       "\n",
       "       [[ 0.,  1.,  2.,  3.]],\n",
       "\n",
       "       [[36., 37., 38., 39.]]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_to_encode = np.array([[3], [3], [0], [9]])\n",
    "model.predict(labels_to_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of an embedding layer is then a 3-d tensor of shape `(batch_size, sequence_length, embedding_size)`.\n",
    "To remove the sequence dimension, useless in our case, we use the `Flatten()` layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Flatten\n",
    "\n",
    "x = Input(shape=[1], name='input')\n",
    "y = Flatten()(embedding_layer(x))\n",
    "model2 = Model(inputs=x, outputs=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, 4)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.output_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[12., 13., 14., 15.]], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.predict(np.array([3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** how many trainable parameters does `model2` have? Check your answer with `model2.summary()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we re-used the same `embedding_layer` instance in both `model` and `model2`: therefore **the two models share exactly the same weights in memory**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2.set_weights([np.ones(shape=(vocab_size, embedding_size))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_to_encode = np.array([[3]])\n",
    "model2.predict(labels_to_encode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1., 1., 1., 1.]]], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(labels_to_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Home assignment**:\n",
    "\n",
    "\n",
    "The previous model definitions used the [function API of Keras](https://keras.io/getting-started/functional-api-guide/). Because the embedding and flatten layers are just stacked one after the other it is possible to instead use the [Sequential model API](https://keras.io/getting-started/sequential-model-guide/).\n",
    "\n",
    "Defined a third model named `model3` using the sequential API and that also reuses the same embedding layer to share parameters with `model` and `model2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "\n",
    "# TODO\n",
    "model3 = None\n",
    "\n",
    "# print(model3.predict(labels_to_encode))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 1. 1. 1.]]\n"
     ]
    }
   ],
   "source": [
    "# %load solutions/embeddings_sequential_model.py\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "model3 = Sequential([\n",
    "    embedding_layer,\n",
    "    Flatten(),\n",
    "])\n",
    "\n",
    "labels_to_encode = np.array([[3]])\n",
    "print(model3.predict(labels_to_encode))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}