a2_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [a2](http://web.stanford.edu/class/cs224n/assignments/a2.pdf) solutions\n",
    "\n",
    "(a). note $y$ one-hot, thus $-\\sum_w y_w\\log \\hat{y}_w= - \\sum_w 1_{w=o}\\log\\hat{y}_w = -\\log \\hat{y}_o$.\n",
    "\n",
    "(b). Rewrite $J=-\\log \\hat{y}_o$ and $\\hat{y}_o = \\frac{\\exp(y_o^T U^T v_c)}{\\sum_w \\exp(y_w^T U^T v_c)}$, that\n",
    "\\begin{align}\n",
    "       \\frac{\\partial J}{\\partial \\hat{y}_o} = - \\frac{1}{\\hat{y}_o},\n",
    "       \\frac{\\partial \\hat{y}_o}{\\partial v_c} = \\frac{(.)\\exp(y_o^T U^T v_c) Uy_o - \\exp(y_o^T U^T v_c) \\sum_w \\exp(y_w^T U^T v_c) Uy_w}{(.)^2} = \\hat{y}_o Uy_o - \\hat{y}_o \\sum_w \\hat{y}_w U y_w\n",
    "\\end{align}\n",
    "thus\n",
    "\\begin{align}\n",
    "       \\frac{\\partial J}{\\partial v_c} = \\sum_w \\hat{y}_w U y_w - U y_o\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "(c). when $w\\neq o$, $\\frac{\\partial J}{\\partial u_w} =  \\hat{y}_w v_c$; \n",
    "when $w=o$, $\\frac{\\partial J}{\\partial u_w} = (\\hat{y}_w-1) v_c$. (当=1, 即分布match了，即到极点了)\n",
    "\n",
    "(d). diag($\\sigma(1-\\sigma)$)\n",
    "\n",
    "(e). \\begin{align}\n",
    "        \\frac{\\partial J}{\\partial v_c} &= -(1-\\sigma(u_o^Tv_c))u_o +\\sum_k (1-\\sigma(-u_k^T v_c))u_k \\\\\n",
    "        \\frac{\\partial J}{\\partial u_o} &= -(1-\\sigma(u_o^Tv_c))v_c \\\\\n",
    "        \\frac{\\partial J}{\\partial u_k} &= (1-\\sigma(-u_k^Tv_c))v_c \\\\\n",
    "\\end{align}\n",
    "\n",
    "Since without care the sparse $U$.\n",
    "\n",
    "(f).\n",
    "  - $\\sum_{j} \\frac{\\partial J_j}{\\partial U}$\n",
    "  - $\\sum_{j} \\frac{\\partial J_j}{\\partial v_c}$\n",
    "  - 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}