Jacobian-Matrix-in-Tensorflow2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jacobian Matrix of a function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Suppose $f : ℝ^{n} → ℝ^{m}$ is a function which takes as input the vector $x ∈ ℝ^{n}$ and produces as output the vector $f(x) ∈ ℝ^{m}$. Then the Jacobian matrix $J$ of $f$ is an $m×n$ matrix, usually defined and arranged as follows:\n",
    "\n",
    "$$\\mathbf J = \\begin{bmatrix}\n",
    "    \\dfrac{\\partial \\mathbf{f}}{\\partial x_1} & \\cdots & \\dfrac{\\partial \\mathbf{f}}{\\partial x_n} \\end{bmatrix}\n",
    "= \\begin{bmatrix}\n",
    "    \\dfrac{\\partial f_1}{\\partial x_1} & \\cdots & \\dfrac{\\partial f_1}{\\partial x_n}\\\\\n",
    "    \\vdots & \\ddots & \\vdots\\\\\n",
    "    \\dfrac{\\partial f_m}{\\partial x_1} & \\cdots & \\dfrac{\\partial f_m}{\\partial x_n} \\end{bmatrix}$$\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "Source: https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example #1, Calculating Jacobian Matrix of a given function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$f(x,y) = ((x+y),(x×y))$\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "${f}_{i,j} =\n",
    "\\begin{bmatrix}\n",
    "  \\frac{\\partial f_1}{\\partial x_1} & \n",
    "    \\frac{\\partial f_1}{\\partial x_2} \\\\[1ex] % <-- 1ex more space between rows of matrix\n",
    "  \\frac{\\partial f_2}{\\partial x_1} & \n",
    "    \\frac{\\partial f_2}{\\partial x_2} \n",
    "\\end{bmatrix}\n",
    "$\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "Jaccobian Matrix of the preceding function = $\\begin{bmatrix}1 & 1\\\\y & x\\end{bmatrix}$\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "If $x=2$ and $y=3$ then Jacobian Matrix will be:$\\begin{bmatrix}1 & 1\\\\3 & 2\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0 1.0\n",
      "3.0 2.0\n"
     ]
    }
   ],
   "source": [
    "def f(x,y):\n",
    "    return x+y, x*y\n",
    "\n",
    "x = tf.Variable(2.0)\n",
    "y = tf.Variable(3.0)\n",
    "\n",
    "with tf.GradientTape(persistent=True) as t:\n",
    "    t.watch([x,y])\n",
    "    f1,f2 = f(x,y)\n",
    "\n",
    "df1_dx = t.gradient(f1, x).numpy()\n",
    "df1_dy = t.gradient(f1, y).numpy()\n",
    "df2_dx = t.gradient(f2, x).numpy()\n",
    "df2_dy = t.gradient(f2, y).numpy()\n",
    "\n",
    "del t\n",
    "print(df1_dx,df1_dy)\n",
    "print(df2_dx,df2_dy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example #2, Calculating Jacobian Matrix of a Neural Network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten (Flatten)            (None, 2)                 0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 3)                 9         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 8         \n",
      "=================================================================\n",
      "Total params: 17\n",
      "Trainable params: 17\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# First Creating a neural network:\n",
    "#following neural net inputs two floating numbers and outouts two floating numbers:\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=(2,1)),\n",
    "    tf.keras.layers.Dense(3),\n",
    "    tf.keras.layers.Dense(2)\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.41490406]] [[0.6787679]]\n",
      "[[-0.38703954]] [[0.47246915]]\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable([[2.0]])\n",
    "y = tf.Variable([[3.0]])\n",
    "#Persistent parameter of the GradientTape is set to True so that after first call of \n",
    "#t.gradient, t is not destroyed:\n",
    "with tf.GradientTape(persistent=True) as t:\n",
    "    t.watch([x,y])\n",
    "    z = tf.concat([x,y],1)\n",
    "    f1 = model(z)[0][0]\n",
    "    f2 = model(z)[0][1]\n",
    "    \n",
    " \n",
    "df1_dx = t.gradient(f1, x).numpy()\n",
    "df1_dy = t.gradient(f1, y).numpy()\n",
    "df2_dx = t.gradient(f2, x).numpy()\n",
    "df2_dy = t.gradient(f2, y).numpy()\n",
    "\n",
    "del t\n",
    "print(df1_dx,df1_dy)\n",
    "print(df2_dx,df2_dy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}