MachineTranslation_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prove the soft max is invariant to constant offset in the input\n",
    "\n",
    "$$ softmax(x) = softmax(x+c) $$\n",
    "\n",
    "#### Answer\n",
    "\n",
    "We can write softmax function as below,\n",
    "\n",
    "$$ softmax(x)_j = \\frac{e^{x_j}}{\\sum_{k=1}^D e^{x_k}} $$\n",
    "\n",
    "Then, after adding constant $c$\n",
    "\n",
    "$$ softmax(x+c)_j = \\frac{e^{x_j + c}}{\\sum_{k=1}^D e^{x_k + c}} = \\frac{e^{x_j}  e^{c}}{\\sum_{k=1}^D e^{x_k}e^{c}} = \\frac{e^{x_j}}{\\sum_{k=1}^D e^{x_k}} = softmax(x)_j$$\n",
    "\n",
    "### Derive the gradient of sigmoid\n",
    "\n",
    "\n",
    "#### Answer\n",
    "We can write sigmoid function as below,\n",
    "\n",
    "$$ \\sigma(x) = sigmoid(x) = \\frac{1}{1+e^{-x}} $$\n",
    "\n",
    "And, we know below\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial x} \\frac{1}{1+e^{-x}} = \\frac{0\\cdot e^{-x} - 1 \\cdot - e^{-x}}{(1+e^{-x})^2} = \\frac{e^{-x}}{(1+e^{-x})^2} = \\frac{1}{(1+e^{-x})}\\frac{e^{-x}}{(1+e^{-x})} = \\frac{1}{(1+e^{-x})}\\left(\\frac{1+e^{-x}}{(1+e^{-x})} -\\frac{1}{(1+e^{-x})}\\right) = \\sigma(x)(1-\\sigma(x))$$\n",
    "\n",
    "$$\\sigma'(x) =  \\sigma(x)(1-\\sigma(x)) $$\n",
    "\n",
    "\n",
    "### Derive the gradient w.r.t the inputs of a softmax function when cross entropy loss is used for evaluation\n",
    "\n",
    "Find the gradients with respect to the softmax input vector $\\theta$, when the prediction is made by $\\hat{y} = softmax(\\theta)$. The cross entropy function is \n",
    "\n",
    "$$ C(y,\\hat{y}) = - \\sum_i y_i log(\\hat{y}_i) $$\n",
    "\n",
    "#### Answer\n",
    "\n",
    "$$\\frac{\\partial C}{\\partial \\theta} = \\frac{\\partial}{\\partial \\theta} \\left[ - \\sum_i y_i log(\\hat{y}_i) \\right] = \\frac{\\partial}{\\partial \\theta} \\left[ - \\sum_i y_i log \\left(\\frac{e^{\\theta_i}}{\\sum_{k=1}^D e^{\\theta_k}} \\right) \\right] = \\dots $$\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial \\theta} C = \\hat{y} - y $$\n",
    "\n",
    "\n",
    "### Derive the gradient with respect to the input $x$, $\\frac{\\partial J}{\\partial x}$ where $J$ is the cost function. We use cross entropy cost $C$ for cost function $J$\n",
    "\n",
    "Note taht below equations\n",
    "\n",
    "$$ h = sigmoid(x W_1 + b_1) $$\n",
    "$$ \\hat{y} = softmax(h W_2 + b_2) $$\n",
    "\n",
    "We now derive below with chain rules\n",
    "$$\\frac{\\partial J}{\\partial x} = \\frac{\\partial J}{\\partial \\theta}\\frac{\\partial \\theta}{\\partial h}\\frac{\\partial h}{\\partial a}\\frac{\\partial a}{\\partial x}$$\n",
    "\n",
    "We now compute one by one\n",
    "\n",
    "* $\\frac{\\partial J}{\\partial \\theta}$\n",
    "\n",
    "We derived in previous question, $\\partial J / \\partial \\theta = \\hat{y} - y$\n",
    "\n",
    "* $\\frac{\\partial \\theta}{\\partial h}$\n",
    "\n",
    "$\\theta = h W_2 + b_2$ and $\\frac{\\partial \\theta}{\\partial h} = W_2^T$\n",
    "\n",
    "* $\\frac{\\partial h}{\\partial a}$\n",
    "\n",
    "$h = sigmoid(x W_1 + b_1)$ and put $a = x W_1 + b_1$ then, $h = sigmoid (a)$\n",
    "$$\\frac{\\partial h}{\\partial a} = sigmoid'(a) = h (1-h)$$\n",
    "\n",
    "* $\\frac{\\partial a}{\\partial x}$\n",
    "\n",
    " $a = x W_1 + b_1$ and $\\frac{\\partial a}{\\partial x} = W_1^T$\n",
    "\n",
    "\n",
    "Then,\n",
    "\n",
    "$$\\frac{\\partial J}{\\partial x} = \\frac{\\partial J}{\\partial \\theta}\\frac{\\partial \\theta}{\\partial h}\\frac{\\partial h}{\\partial a}\\frac{\\partial a}{\\partial x} = (\\hat{y} - y)W_2^T h(1-h)W_1^T$$\n",
    "\n",
    "### How many parameters in NN\n",
    "\n",
    "Input is $D_x$-dim, output is $D_y$-dim, there are H hidden\n",
    "\n",
    "#### Answer\n",
    "$D_x * H + H * D_y + H + D_y$ parameters, $H$ and $D_y$ are bias terms for hidden layer and output layer respectively.\n",
    "\n",
    "### Assume you are given a predicted word vector \\hat^r (v_{w_I} in the lecture notes in the case of skip-gram), and word prediction is made with the softmax function found in **word2vec** models\n",
    "\n",
    "$$P(\\textrm{word}_i | \\hat{r}, w) = \\frac{\\exp(w_i^T \\hat{r})}{\\sum_{j=1}^|V| \\exp(w_j^T \\hat{r})}$$\n",
    "\n",
    "Derive the gradients of cost function w.r.t $\\hat{r}$. Cross entropy cost is applied.\n",
    "\n",
    "#### Answer\n",
    "\n",
    "We need to compute $\\partial J / \\partial \\hat{r} $. And,\n",
    "\n",
    "$$J = -log P(\\textrm{word}_i | \\hat{r}, w) = -log\\frac{exp(w_i^T \\hat{r})}{\\sum_{j=1}^{|V|} exp(w_j^T \\hat{r})} = - (w_i^T \\hat{r} - \\log \\sum_{j=1}^{|V|} \\exp(w_j^T \\hat{r})) = - w_i^T \\hat{r} + \\log \\sum_{j=1}^{|V|} \\exp(w_j^T \\hat{r}) $$\n",
    "\n",
    "Then,\n",
    "\n",
    "$$\\frac{\\partial J}{ \\partial \\hat{r}} = -w_i^T + \\frac{1}{\\sum_{j=1}^{|V|} \\exp(w_j^T \\hat{r})} \\sum_{k=1}^{|v|} \\exp(w_k^T \\hat{r}) w_k  = -w_i^T + \\sum_{k=1}^{|V|} P(\\textrm{word}_k | \\hat{r}, w) w_k$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}