Add IPython notebooks, update README

.gitignore

@@ -0,0 +1,10 @@
+*/.ipynb_checkpoints/*
+*.gz
+*.aux
+*.log
+*.out
+.ipynb*
+*.pyc
+tags
+.idea/*
+*.swp

README.md

@@ -1,8 +1,38 @@
-# mathematical-methods-in-deep-learning
+# Mathematical Methods in Deep Learning
+
 Python code (in the form of Jupyter ipython notebooks) to support the book
 "Mathematical Methods in Deep Learning" (Krishnendu Chaudhury, Ananya Ashok Honnedevasthana,
 Sujay Narumanchi, Devashish Shankar).
 This repository contains the example code - mostly in Numpy and PyTorch - corresponding to
 the theoretical topics introduced in the book. The code listings are organized in chapters
 that correspond to the main book.
-(TBD: content)
+
+## Table of Contents
+
+* Chapter 2:
+  * [2.2 Intro to Vectors](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.2-vector-numpy-pytorch-intro.ipynb)
+  * [2.4 Intro to Matrices, Tensors and Images](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.4-matrix-numpy-pytorch-intro.ipynb)
+  * [2.7 Basic Vector and Matrix operations](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.7-transpose-dot-matmul-numpy-pytorch.ipynb)
+  * [2.12.5 Solving an overdetermined system using pseudo inverse](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.12.5-overdet-numpy.ipynb)
+  * [2.13 Eigenvalues and Eigenvectors](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.13-eig-numpy.ipynb)
+  * [2.14.1 Rotation Matrices](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.14.1-rotation-numpy.ipynb)
+  * [2.15.1 Matrix Diagonalization](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.15.1-mat-diagonalization-numpy.ipynb)
+  * [2.15.2 Spectral Decomposition of a Symmetric Matrix](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.15.2-spectral-decomp-numpy.ipynb)
+  * [2.15.5 Finding the axes of a hyper-ellipse](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch2/2.15.5-hyper-ellipse-numpy.ipynb)
+
+
+* Chapter 3
+  * [3.4 Common code for chapter 3](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch3/3.4.1-gradients-numpy-pytorch.ipynb)
+  * [3.4.1 Gradient Descent](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch3/3.4-common.ipynb)
+  * [3.4.2 Non-linear Models](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch3/3.4.2-gradients-nonlinear-numpy-pytorch.ipynb)
+  * [3.4.3 A Linear Model for the cat-brain](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch3/3.4.3-gradients-catbrain-numpy-pytorch.ipynb)
+
+* Chapter 4
+  * [4.3.2 Common code for chapter 4](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.3.2-common.ipynb)
+  * [4.3.2 PCA on synthetic correlated data](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.3.2-pca-numpy.ipynb)
+  * [4.3.2 PCA on synthetic uncorrelated data](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.3.2-pca-uncorrelated-numpy.ipynb)
+  * [4.3.3 PCA on synthetic correlated non-linear data](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.3.3-pca-nonlinear-numpy.ipynb)
+  * [4.4.4 Linear system solving via SVD](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.4.4-svd-linear-system-numpy.ipynb)
+  * [4.4.5 PCA computation via SVD](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.4.5-svd-pca-numpy.ipynb)
+  * [4.5.3 LSA/SVD on a 500 × 3 dataset](https://nbviewer.jupyter.org/github/krishnonwork/mathematical-methods-in-deep-learning-ipython/blob/master/python/ch4/4.5.3-svd-lsa-numpy.ipynb)
+

python/ch2/2.12.5-overdet-numpy.ipynb

@@ -0,0 +1,137 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Solving an overdetermined system using  pseudo inverse\n",
+    "\n",
+    "Consider the overdetermined system corresponding to cat-brain from Chapter 2.\n",
+    "\n",
+    "There are 15 training examples, each with input and desired outputs specified.\n",
+    "\n",
+    "Our goal is to determine 3 unkwnowns (w0, w1, b).\n",
+    "\n",
+    "This can be cast as an over-determined system of equations\n",
+    "$$\n",
+    "A\\vec{w} = \\vec{y}\n",
+    "$$\n",
+    "where\n",
+    "$$ \n",
+    "A =\n",
+    "\\begin{bmatrix}\n",
+    "        0.11 & 0.09 & 1.00 \\\\\n",
+    "        0.01 & 0.02 & 1.00 \\\\\n",
+    "        0.98 & 0.91 & 1.00 \\\\\n",
+    "        0.12 & 0.21 & 1.00 \\\\\n",
+    "        0.98 & 0.99 & 1.00 \\\\\n",
+    "        0.85 & 0.87 & 1.00 \\\\\n",
+    "        0.03 & 0.14 & 1.00 \\\\\n",
+    "        0.55 & 0.45 & 1.00 \\\\\n",
+    "        0.49 & 0.51 & 1.00 \\\\\n",
+    "        0.99 & 0.01 & 1.00 \\\\\n",
+    "        0.02 & 0.89 & 1.00 \\\\\n",
+    "        0.31 & 0.47 & 1.00 \\\\\n",
+    "        0.55 & 0.29 & 1.00 \\\\\n",
+    "        0.87 & 0.76 & 1.00 \\\\\n",
+    "        0.63 & 0.24 & 1.00\n",
+    "\\end{bmatrix}\n",
+    "\\;\\;\\;\\;\\;\\;\\;\n",
+    "\\vec{y} = \n",
+    "\\begin{bmatrix}\n",
+    "        -0.8 \\\\\n",
+    "        -0.97 \\\\\n",
+    "        0.89 \\\\ \n",
+    "        -0.67 \\\\ \n",
+    "        0.97 \\\\ \n",
+    "        0.72 \\\\ \n",
+    "        -0.83 \\\\ \n",
+    "        0.00 \\\\\n",
+    "        0.00 \\\\\n",
+    "        0.00 \\\\\n",
+    "        -0.09 \\\\\n",
+    "        -0.22 \\\\ \n",
+    "        -0.16 \\\\\n",
+    "        0.63 \\\\\n",
+    "        0.37\n",
+    "\\end{bmatrix}\n",
+    "\\;\\;\\;\\;\\;\\;\\;\n",
+    "\\vec{w} = \\begin{bmatrix} w_{0}\\\\w_{1}\\\\b\\end{bmatrix}\n",
+    "$$\n",
+    "\n",
+    "We solve for $\\vec{w}$ using the pseudo inverse formula $\\space\\space\\large{\\vec{w} = (A^TA)^{-1}A^Ty}$"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The solution is [ 1.07661761  0.89761672 -0.95816936]\n",
+      "Note that this is almost equal to [1.0, 1.0, -1.0])\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "# Let us revisit our cat brain data set\n",
+    "# Notice that there are 15 training examples, with 3\n",
+    "# unkwnowns (w0, w1, b).\n",
+    "# This is an over determined system.\n",
+    "# It can be easily seen that the solution is roughly\n",
+    "# $w_{0} = 1, w_{1} = 1, b = -1$.\n",
+    "# It has been deliberately chosen as such.\n",
+    "# But the equations are not fully consistent (i.e., there is\n",
+    "# no solution that satisfies all the equations).\n",
+    "# We want to find the best values such that it minimizes Aw - b.\n",
+    "# This is what the pseudo-inverse does.\n",
+    "\n",
+    "def pseudo_inverse(A):\n",
+    "    return np.matmul(np.linalg.inv(np.matmul(A.T, A)), A.T)\n",
+    "\n",
+    "# The usual cat-brain input dataset\n",
+    "X = np.array([[0.11, 0.09], [0.01, 0.02], [0.98, 0.91], [0.12, 0.21],\n",
+    "              [0.98, 0.99], [0.85, 0.87], [0.03, 0.14], [0.55, 0.45],\n",
+    "              [0.49, 0.51], [0.99, 0.01], [0.02, 0.89], [0.31, 0.47],\n",
+    "              [0.55, 0.29], [0.87, 0.76], [0.63, 0.24]])\n",
+    "\n",
+    "# Output threat score modeled as a vector\n",
+    "y = np.array([-0.8, -0.97, 0.89, -0.67, 0.97, 0.72, -0.83, 0.00, 0.00,\n",
+    "              0.00, -0.09, -0.22, -0.16, 0.63, 0.37])\n",
+    "A = np.column_stack((X, np.ones(15)))\n",
+    "\n",
+    "# Column stack will add an additional column of 1s to the training\n",
+    "# dataset to represent the coefficient of the bias\n",
+    "w = np.matmul(pseudo_inverse(A), y)\n",
+    "\n",
+    "print(\"The solution is {}\\n\"\n",
+    "      \"Note that this is almost equal to [1.0, 1.0, -1.0])\".format(w))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

python/ch2/2.13-eig-numpy.ipynb

@@ -0,0 +1,89 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eigenvalues and Eigenvectors\n",
+    "\n",
+    "Multiplication by a matrix is a linear tansformation that maps vectors (points) to different vectors(points).\n",
+    "\n",
+    "But in special cases,  vectors will not change (except for scaling) when transformed.\n",
+    "\n",
+    "An eigen vector $\\vec{e}$ for a linear transformation represented by matrix $A$ will satisfy $A\\vec{e} = \\lambda \\vec{e}$.\n",
+    "\n",
+    "In other words, $A$ maps its eigenvector $\\vec{e}$ to (a scaled version) of itself."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Eigen values are\n",
+      "[0.7071+0.7071j 0.7071-0.7071j 1.    +0.j    ]\n",
+      "\n",
+      "Eigen vectors are\n",
+      "[[0.70710678+0.j         0.        +0.70710678j 0.        +0.j        ]\n",
+      " [0.70710678-0.j         0.        -0.70710678j 0.        -0.j        ]\n",
+      " [0.        +0.j         0.        +0.j         1.        +0.j        ]]\n",
+      "\n",
+      "Note that 1 + 0j i.e 1 is an Eigen value\n",
+      "and [0, 0, 1+0j] i.e Z Axis is an Eigen vector.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "from numpy import linalg as LA\n",
+    "\n",
+    "# A = [cos(45)  sin(45)  0]\n",
+    "#     [-sin(45) cos(45)  0]\n",
+    "#     [0              0  1]\n",
+    "# This matrix performs a rotation about the Z axis [0  0  1].\n",
+    "# All rotation matrices have 1 as an eigen-value. The corresponding\n",
+    "# eigenvector will be the axis of rotation.\n",
+    "# This is consistent with the physical definition of rotation axis -\n",
+    "# that points on the axis stay where they were post the rotation.\n",
+    "A = np.array([[0.7071, 0.7071, 0], [-0.7071, 0.7071, 0], [0, 0, 1]])\n",
+    "\n",
+    "# LA.eig returns eigen values in the array l and the corresponding\n",
+    "# eigen vectors as columns of e.\n",
+    "# Note that eigen values can be complex numbers (with a real and\n",
+    "# imaginary part). The imaginary part will be expressed via\n",
+    "# j which denotes square root of -1 (imaginary number).\n",
+    "l, e = LA.eig(A)\n",
+    "\n",
+    "print(\"Eigen values are\\n{}\\n\".format(l)) \n",
+    "print(\"Eigen vectors are\\n{}\\n\\n\"\n",
+    "      \"Note that 1 + 0j i.e 1 is an Eigen value\\n\"\n",
+    "      \"and [0, 0, 1+0j] i.e Z Axis is an Eigen vector.\".format(e.T))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}