ChangeOfVariables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \n",
    "\n",
    "Suppose we have two cartesian coordinate systems (unprimed and primed), where coordinate vectors are denoted as $\\mathbf{i}$ and $\\mathbf{j}$ and $\\mathbf{i}'$ and $\\mathbf{j}'$ respectively.\n",
    "\n",
    "$$\n",
    "x = 2x'\n",
    "$$\n",
    "\n",
    "$$\n",
    "y = 2y'\n",
    "$$\n",
    "\n",
    "Suppose\n",
    "$$\n",
    "F(x,y) = x^2 e^y\n",
    "$$\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial F}{\\partial x} & = & 2 x e^y \\\\\n",
    "\\frac{\\partial F}{\\partial y} & = & x^2 e^y \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "If we substitude primed coordinates\n",
    "\n",
    "$$\n",
    "F'(x',y') = 4 (x')^2 e^{2 y'}\n",
    "$$\n",
    "\n",
    "The partial derivative in the primed coordinate system is \n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial F'}{\\partial x'} & = & 8 x' e^{2 y' } \\\\\n",
    "\\frac{\\partial F'}{\\partial y'} & = & 8 (x')^2 e^{2 y'} \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "We conclude\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial F'}{\\partial x'} & = & 4 x e^{y } = 2 \\frac{\\partial F}{\\partial x} \\\\\n",
    "\\frac{\\partial F'}{\\partial y'} & = & 2 x^2 e^{y} = 2 \\frac{\\partial F}{\\partial y}\\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "But if we want the gradient be a vector, a geometrical object entirely independent of the particular choice of coordinates.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coordinate Systems\n",
    "\n",
    "Consider the plane $E^2$ (the two dimensional Euclidian space) and a general point $\\omega \\in E^2$. We will first resist the temptation of denoting points in this space\n",
    "by a pair of numbers as $\\omega = (x_1, x_2)$; rather we will merely view $\\omega$ as a geometrical object. In fact,\n",
    "the geometrical notions distance (metric) and angle are sufficient to describe Euclidian spaces.\n",
    "\n",
    "Wikipedia: \"One way to think of the Euclidean plane is as a set of points satisfying certain relationships, expressible in terms of distance and angle.\"\n",
    "\n",
    "We will define two functions $x_1 : E^2 \\rightarrow \\mathcal{R}$ and $x_2 : E^2 \\rightarrow \\mathcal{R}$ such that \n",
    "for every point $\\omega$ in the plane, we will have a so called coordinate representation\n",
    "\\begin{eqnarray}\n",
    "\\left(x_1(\\omega), x_2(\\omega) \\right)\n",
    "\\end{eqnarray}\n",
    "These functions will associate with each point $\\omega$ a pair of numbers. The familiar Cartesian coordinate system is the first example. The idea of a coordinate system is so natural that $\\omega$ is taken equivalent to $(x_1, x_2)$. \n",
    "\n",
    "We could have defined another coordinate system, such as polar coordinates,\n",
    "\\begin{eqnarray}\n",
    "\\left(\\xi_1(\\omega), \\xi_2(\\omega) \\right)\n",
    "\\end{eqnarray}\n",
    "\n",
    "We can think of a collection of points "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAFdCAYAAACet25NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAHCVJREFUeJztnU2Infd5R89rD6kwgiqCKi20URYipDB00S5iQbqRBfVq\nLEhqkEOhNUhCILWurYhko3hTHGSqikqiir0YEeiitFrYKyHy4UICCZQq1IsEEoogAXUhummRg1Xl\n7eK9ihVb+uv536/n/TgHzvKX+8xcxTpzdWemadsWERERmS5PZB8gIiIiuRgDIiIiE8cYEBERmTjG\ngIiIyMQxBkRERCaOMSAiIjJxjAEREZGJYwyIiIhMHGNARERk4hgDIiIiE8cYEBERmTjGgIiIyMQx\nBkRERCaOMSAiIjJxjAEREZGJYwyIiIhMHGNARERk4mxkH7AMmqZ5CrgA/BNwu3L+NPBp4BuVuw3g\nldnuVuX2D4D9wNcrd08Afw1cBW5Wbj8NPEv3eWortyeAbwI/rtztBf4U+DvgXuX2CPBvwI3K3W8D\nfw78LXC3cvtF4D+B7bZt71RuRUQGS9O2tX8v9I+maT4F/ADYk3uJjIQTbdteyj5CRGRdjCIGAJqm\n2QS+BdwBjlH3CsEx4ChwEdiu2O0B3gSepPtKtuYVghfoXlm4QvfVepRddK8ofJzu5psV2y3gDN0r\nC18j/grBTuAS8EngOHWvEDwDvAZcB75K/BWCHcB5YBM4Sd0rBPuBc8D3gdPEXyHYAF4HPgscatv2\nWsVjiogMl7ZtRyOwD/g53V9Wv1O5fZXuL8evVO5+F/gp3V/Kn6rcvjR7zK8xC7Pg7reAd4H/An6/\n8jFfBH4J/APwRMXuN+lefflv4A8rH/MLwP8B/whsVOyeAr4N/C/wx5WP+SzwC+Bt4Dcqdh+bbX4B\nPLuqP6uqqn0y/YClf0BdEPxsziD46gJB8JM5g+CvFgiC/5gzCP7CICjuDAJVnZTpB6zkgzIIIluD\noLwzCFR1MqYfsLIPzCCIbA2C8s4gUNVJmH7ASj84gyCyXSQIvp8QBN+aMwj+ZIEgeMsgUNUxm37A\nyj9AgyCyzQiCzxsEqqr9MP2AtXyQBkFkaxCUdwaBqo7W9APW9oEaBJGtQVDeGQSqOkrTD1jrB2sQ\nRLa+qbC8802Fqjo60w9Y+wdsEES2BkF5ZxCo6qhMPyDlgzYIIluDoLwzCFR1NKYfkPaBGwSRrUFQ\n3hkEqjoK0w9I/eANgsjWICjvDAJVHbzpB2RrEIS2fpdBeed3GajqoE0/oA8aBKGtQVDeGQSqOljT\nD+iLBkFoaxCUdwaBqg7S9AP65CwIfp4QBD81CIpbg0BVdYWmH9A3FwyCVxOC4CWDoLgzCFRVH2P6\nAX3UIAhtDYLyziBQ1cGYfkBfNQhCW4OgvDMIVHUQph/QZw2C0NYgKO8MAlXtvekH9F0++C6DH80R\nBPffVPjlyt0i32VwPwheMwgeujMIVFU/ZPoBQ/CBVwjmCYL7rxDMEwSLvkJgEDx8ZxCoqj5g+gFD\n0SAIbQ2C8s4gUNVemn7AkDQIQluDoLwzCFS1d6YfMDQNgtDWICjvDAJV7ZVN27ZIHU3T7APeofsL\n5C+B2xXzY8BR4AJwpWK3B3gTeBI4Atyq2L4AvAJsAxcrdruAy8BuuptvVmy3gDPAVbrvboj+QdtJ\nd+Ne4Djdd3JEOTB7rOt0b968F9ztAM4Dm8BJ4EbFY+4HztFFzGngbnC3AZyd7V8Gttu2vVPxuCIi\nS8MYmJNZEHyP7i9pkUU50bbtpewjRGSaGAML0DTNJt1LzHfovuL3FYKP4isEZe6/QvA0cKht22sV\njykishyy/51i6LKcH0yU8R4CfzDRw3e+h0BVJ2f6AWOQxd5UuMgPJvK3HZa3BoGqasD0A8bigkGQ\n8QqBQVDeGQSqOhnTDxiTBkFoaxCUdwaBqq7d9APGpkEQ2hoE5Z1BoKprNf2AMWoQhLYGQXlnEKjq\n2kw/YKwaBKGtQVDeGQSquhbTDxizBkFoaxCUdwaBqq7c9APGrkEQ2hoE5Z1BoKorNf2AKchyfjDR\nVyp3BsHjtwaBqmprDKzvE71YENz/wUQGwcN3BoGq6gKmHzAlfYUgtDUIyjuDQFWXbvoBU9MgCG0N\ngvLOIFDVpZp+wBQ1CEJbg6C8MwhUdWmmHzBVDYLQ1iAo7wwCVV2K6QdMWYMgtDUIyjuDQFUXNv2A\nqWsQhLYGQXlnEKjqQqYfoAZBcGsQlHcGgarObfoBOnsiDILI1iAo7wwCVZ3L9AP0gSfDH0wU2RoE\n5Z1BoKrVph+gH3pCcl4h+D2D4LFbg0BVR2v6AfqQJ8V/MohsDYLyziBQ1bDpB+gjnpjhBcFLCwTB\nu3MGwYsLBMEP5gyCLywQBN+eMwieXSAI3jYIVPVxph+ghSfHIIhsDYLyziBQ1ceafoA+5gnqguBn\ncwbBIm8q/ElCEAzpnwwMAlUdjekHaOBJygsC30NQ3hoEqjoKm7Ztkf7TNM0+4DvAHeAkcLtifhQ4\nBlwEtit2e4A3gSeBI8Ctiu1h4BRwBbhQsdsFXAZ20919s2K7BZwBrtKFSPQP9066z81e4DhddEV5\nBngNuD577F8GdzuA88AmcAL4YcVj7gfO0UXMl+iCJMIG8DrwNPAysN227Z2KxxWRkWIMDIhZEHyP\n7i9pkUU50bbtpewjRCQfY2BgNE2zSfdtanfovtqveYXgGN1X275C8HCm9grBZ4FDbdteq3hMERkj\n2f9OofUyvO8ymOJ7CJ6s2D34HoLPVT7mg+8h+FjFzvcQqOqvTD9A53ziDILI1iAo7wwCVaVtjYFB\naxCEtgZBeWcQqKoxMHQNgtA2Owj8tkNV7bXpB+gSnsTh/WAig6C883cZqOpaTT9Al/RELhYE918h\n+HLlbkpBMJUfXWwQqE7Q9AN0iU+mQRDZZgfBEN5DYBCoTsz0A3TJT+hy3kNgEDx8ZxCo6ihNP0BX\n8KQaBJGtQVDeGQSqEzL9AF3RE2sQRLYGQXlnEKhOxPQDdIVPbt63HRoE5a1vKlTVXpl+gK74CTYI\nIluDoLwzCFRHbvoBuoYn2SCIbA2C8s4gUB2x6Qfomp7o5XzboT+p8OE7g0BVB236AbrGJ3t4QfDS\nAkHw7pxB8KJBUNwZBKojNP0AXfMTbhBEtgZBeWcQqI7M9AM04Uk3CCJbg6C8MwhUR2T6AZr0xBsE\nka1BUN4ZBKojMf0ATXzyDYLI1iAo7wwC1RGYfoAm/wHw1x9Hthm//vjzCwSBv/5YVatMP0DzNQhC\nW4OgvDMIVAds+gHaDw2C0NYgKO8MAtWBmn6A9kcW+0mFBkF5ZxCoam9NP0D7pUEQ2hoE5Z1BoDow\n0w/Q/mkQhLZTDAJ//bHqSE0/QPvpgkHw6gJB4LcdlrfZ33ZYGwRvGwSq/bdp2xaRh9E0zT7gHbq/\nQP4SuF0xPwYcBS4C2xW7PcCbwJPAEeBWxfYF4BXgCnChYrcL+Drwcbqbb1Zst4AzwFW6EIn+H2on\ncAn4JHCcLrqiPAO8BlyneyXmXnC3AzgPbAIngRsVj7kfOEf3qsaX6IIkwgbwOvA08DKw3bbtnYrH\nFZE1YAxIkVkQfI/uL2mRRTnRtu2l7CNE5NcxBuSxNE2zSfdvznfovuKf5xWCC3RfsUdZxisE23Sv\nTETJeoXgIrCX+lcIDswea5FXCE4AP6x4zEVfIfgscKht22sVjykiqyb73yl0GPLBewh+xPzvIfhy\n5W4Z7yF4jfnfQ/CZyscc2psK77+H4HOVj+l7CFRHZvoBOhwNgtDWICjvDALVHpp+gA5LgyC0Hep3\nGRgEqhM1/QAdnvhth5HtUINgnb/t0CBQ7YnpB+gwxd9lENkOLQj8SYWqEzX9AB2uBkFoaxCUdwaB\nag9MP0CHrUEQ2hoE5Z1BoJps+gE6fA2C0DbjuwwMAlUNmX6AjkODILQd2rcdGgSqEzH9AB2PBkFo\naxCUdwaBaoLpB+i4NAhCW4OgvDMIVNds+gE6Pg2C0Db7PQRPVuwMAtWRm36AjlODILQdWhD4kwpV\nR2r6ATpeDYLQ1iAo7wwC1TWYfoCOW4MgtDUIyjuDQHXFph+g49cgCG0NgvLOIFBdoekH6DRksV9u\ntEgQzPvLjQyC8s4gUB2R6QfodFwwCF5NCIIp/rZDg0B1gqYfoNPSIAhts4PAX3+sOjHTD9DpaRCE\nthlB4A8mUp2o6QfoNDUIQluDoLwzCFSXZPoBOl0NgtDWICjvDALVJZh+gE5b/LbDyNbfZVDeGQSq\nC5p+gKpBENoaBOWdQaC6gOkHqLatQRDcGgTlnUGgOqfpB6je1yAIbQ2C8s4gUJ3Dpm1bRPpC0zT7\ngO8Ad4CTwO2K+VHgGHAR2K7Y7QHeADaAI8Ctiu1h4BRwBbhQsdsFXAZ20919s2K7BZwBrtKFSPT/\nxDvpPjd7geN00RXlwOyxrtOF173gbgdwHtikez5vVDzmfuAcXcScBu4GdxvA2dn+ZWC7bds7FY8r\nMjmMAekdsyD4LvCJ7FtkFJxo2/ZS9hEifcYYkF7SNM0m8E3gPbqv9n2F4KP4CkGZ+68QPA0catv2\nWsVjikyL7H+nUH2U+B6CyNb3EJR3vodANWD6AaolDYLQ1iAo7wwC1ceYfoDq4zQIQluDoLwzCFQL\nph+gGtEgCG0NgvLOIFB9hOkHqEY1CEJbg6C8MwhUH2L6Aao1GgShrUFQ3hkEqh8y/QDVWg2C0NYg\nKO8MAtUHTD9AdR4NgtDWICjvDALVmekHqM6rQRDaGgTlnUGg2hoDOnANgtDWICjvDAKdvOkHqC6q\nQRDaGgTlnUGgkzb9ANVlaBCEtgZBeWcQ6GRNP0B1WRoEoa1BUN4ZBDpJ0w9QXaYGQWhrEJR3BoFO\nzvQDVJetQRDaGgTlnUGgkzL9ANVVaBCEtgZBeWcQ6GRMP0B1VRoEoa1BUN4ZBDoJ0w9QXaUGQWhr\nEJR3BoGO3vQDVFetQRDaGgTlnUGgozb9ANV1aBCEtgZBeWcQ6GhNP0B1XRoEoa1BUN4ZBDpK0w9Q\nXacGQWhrEJR3BoGOzvQDVNetQRDaGgTlnUGgozL9ANUMDYLQ1iAo7wwCHY3pB6hmaRCEtgZBeWcQ\n6ChMP0A1U4MgtDUIyjuDQAdv07YtIlOmaZp9wHeAO8BJ4HbF/ChwDLgIbFfs9gBvABvAEeBWxfYw\ncAq4Alyo2O0CLgO76e6+WbHdAs4AV+lCJPofjp10n5u9wHG66IpyYPZY1+nC615wtwM4D2zSPZ83\nKh5zP3COLmJOA3eDuw3g7Gz/MrDdtu2discVScUYEOFXQfBd4BPZt8goONG27aXsI0SiGAMiM5qm\n2QS+CbxH99W+rxB8FF8hKHP/FYKngUNt216reEyRPLL/nUK1T+J7CCJb30NQ3vkeAh2c6Qeo9k2D\nILQ1CMo7g0AHZfoBqn3UIAhtDYLyziDQwZh+gGpfNQhCW4OgvDMIdBCmH6DaZw2C0NYgKO8MAu29\n6Qeo9l2DILQ1CMo7g0B7bfoBqkPQIAhtDYLyziDQ3pp+gOpQNAhCW4OgvDMItJemH6A6JA2C0NYg\nKO8MAu2d6QeoDk2DILQ1CMo7g0B7ZfoBqkPUIAhtDYLyziDQ3ph+gOpQNQhCW4OgvDMItBemH6A6\nZA2C0NYgKO8MAk03/QDVoWsQhLYGQXlnEGiq6QeojkGDILQ1CMo7g0DTTD9AdSwaBKGtQVDeGQSa\nYvoBqmPSIAhtDYLyziDQtZt+gOrYNAhCW4OgvDMIdK2mH6A6Rg2C0NYgKO8MAl2b6QeojlWDILQ1\nCMo7g0DXYvoBqmPWIAhtDYLyziDQlZt+gOrYNQhCW4OgvDMIdKWmH6A6BQ2C0NYgKO8MAl2Z6Qeo\nTkWDILQ1CMo7g0BXYvoBqlPSIAhtDYLyziDQpZt+gOrUNAhCW4OgvDMIdKmmH6A6RQ2C0NYgKO8M\nAl2aTdu2iMj6aZpmH/AOcBc4BtyumB+dbS4C2xW7PcAbwAZwBLhVsT0MnAKuABcqdruAy8Buurtv\nVmy3gDPAVboQif4Hayfd52YvcJwuuqIcmD3Wdbrwuhfc7QDOA5vASeBGxWPuB87RRcxpuj8TETaA\ns7P9823bvlXxmCK/whgQSaRpmi26r+5EFuV94Lm2ba9lHyLDwxgQSaRpmqeAg3Rfrb+HrxA8Cl8h\nKLNB9/k5CBwyCKSa7H+nUFXfQxDc+h6C8s73EOjcph+gqp0GQWhrEJR3BoHOZfoBqvqBBkFoaxCU\ndwaBVpt+gKr+ugZBaGsQlHcGgVaZfoCqflSDILQ1CMo7g0DDph+gqg/XIAhtDYLyziDQkOkHqOqj\nNQhCW4OgvDMI9LGmH6CqZQ2C0NYgKO8MAi2afoCqPl6DILQ1CMo7g0AfafoBqhrTIAhtDYLyziDQ\nh5p+gKrGNQhCW4OgvDMI9COmH6CqdRoEoa1BUN4ZBPrrfyayD1DVeg2C0NYgKO8MAv3gz0P2Aao6\nnwZBaGsQlHcGgXZ/FrIPUNX5NQhCW4OgvDMI1BhQHboGQWhrEJR3BsHETT9AVRfXIAhtDYLyziCY\nsOkHqOpyNAhCW4OgvDMIJmr6Aaq6PA2C0NYgKO8MggmafoCqLleDILQ1CMo7g2Biph+gqsvXIAht\nDYLyziCYkOkHqOpqNAhCW4OgvDMIJmL6Aaq6Og2C0NYgKO8MggmYfoCqrlaDILQ1CMo7g2Dkph+g\nqqvXIAhtDYLyziAYsekHqOp6NAhCW4OgvDMIRmoze4JFZAI0TbMPeAe4CxwDblfMj842F4Htit0e\n4A1gAzgC3KrYHgZOAVeACxW7XcBlYDfd3TcrtlvAGeAqXYhE/yO5k+5zsxc4ThddUQ7MHus6XXjd\nC+52AOeBTeAkcKPiMfcD5+gi5jTdn4kIG8DZ2f75tm3fqnhM6SnGgMjEaJpmi+6rO5FFeR94rm3b\na9mHyGIYAyITo2map4CDdF+tv4evEDwKXyEos0H3+TkIHDIIBk72v1Ooao74HoLI1vcQlHe+h2Ak\nph+gqnkaBKGtQVDeGQQjMP0AVc3VIAhtDYLyziAYuOkHqGq+BkFoaxCUdwbBgE0/QFX7oUEQ2hoE\n5Z1BMFDTD1DV/mgQhLYGQXlnEAzQ9ANUtV8aBKGtQVDeGQQDM/0AVe2fBkFoaxCUdwbBgEw/QFX7\nqUEQ2hoE5Z1BMBDTD1DV/moQhLYGQXlnEAzA9ANUtd8aBKGtQVDeGQQ9N/0AVe2/BkFoaxCUdwZB\nj00/QFWHoUEQ2hoE5Z1B0FPTD1DV4WgQhLYGQXlnEPTQ9ANUdVgaBKGtQVDeGQQ9M/0AVR2eBkFo\naxCUdwZBj0w/QFWHqUEQ2hoE5Z1B0BPTD1DV4WoQhLYGQXlnEPTA9ANUddgaBKGtQVDeGQTJph+g\nqsPXIAhtDYLyziBINP0AVR2HBkFoaxCUdwZBkukHqOp4NAhCW4OgvDMIEkw/QFXHpUEQ2hoE5Z1B\nsGbTD1DV8WkQhLYGQXlnEKzR9ANUdZwaBKGtQVDeGQRrMv0AVR2vBkFoaxCUdwbBGmxmn2wRkZXQ\nNM0+4B3gLnAMuF0xPzrbXAS2K3Z7gDeADeAIcKtiexg4BVwBLlTsdgGXgd10d9+s2G4BZ4CrdCES\n/Q/zTrrPzV7gOF10RTkwe6zrdOF1L7jbAZwHNoGTwI2Kx9wPnKOLmNN0fyYibABnZ/vn27Z9q+Ix\nJYAxICIrp2maLbqv7kQW5X3gubZtr2UfMiaMARFZOU3TPAUcpPtq/T18heBR+ApBmQ26z89B4JBB\nsESy/51CVacjvocgsvU9BOWd7yFYgekHqOq0NAhCW4OgvDMIlmz6Aao6PQ2C0NYgKO8MgiWafoCq\nTlODILQ1CMo7g2BJph+gqtPVIAhtDYLyziBYgukHqOq0NQhCW4OgvDMIFjT9AFVVgyC0NQjKO4Ng\nAdMPUFVtW4MguDUIyjuDYE7TD1BVva9BENoaBOWdQTCH6Qeoqj6oQRDaGgTlnUFQafoBqqof1iAI\nbQ2C8s4gqPk8Zx+gqvowDYLQ1iAo7wyC6Ocq+wBV1UdpEIS2BkF5ZxBEPk/ZB6iqljQIQluDoLwz\nCB73Oco+QFX1cRoEoa1BUN4ZBKXPT/YBqqoRDYLQ1iAo7wyCR31usg9QVY1qEIS2BkF5ZxA87POS\nfYCqao0GQWhrEJR3BsGHPyfZB6iq1moQhLYGQXlnEDz4+cg+QFV1Hg2C0NYgKO8Mgvufi+wDVFXn\n1SAIbQ2C8s4gaI0BVR24BkFoaxCUd5MPgvQDVFUX1SAIbQ2C8m7SQZB+gKrqMjQIQluDoLybbBCk\nH6CquiwNgtDWICjvJhkE6Qeoqi5TgyC0NQjKu8kFQTP7wEVERkPTNPuAd4C7wDHgdsX86GxzEdiu\n2O0B3gA2gCPArYrtYeAUcAW4ULHbBVwGdtPdfbNiuwWcAa7ShUj0L4OddJ+bvcBxuuiKcmD2WNfp\nwutecLcDOA9sAieBGxWPuR84Rxcxp+n+TETYAM7O9s+3bftWxWMODmNAREZJ0zRbdF/diSzK+8Bz\nbdteyz5kVRgDIjJKmqZ5CjhI99X6e/gKwaPwFYIyG3Sfn4PAodEGQfa/U6iqrlJ8D0Fk63sIyrvR\nv4cg/QBV1VVrEIS2BkF5N+ogSD9AVXUdGgShrUFQ3o02CNIPUFVdlwZBaGsQlHejDIL0A1RV16lB\nENoaBOXd6IIg/QBV1XVrEIS2BkF5N6ogSD9AVTVDgyC0NQjKu9EEQfoBqqpZGgShrUFQ3o0iCNIP\nUFXN1CAIbQ2C8m7wQZB+gKpqtgZBaGsQlHeDDoL0A1RV+6BBENoaBOXdYIMg/QBV1b5oEIS2BkF5\nN8ggSD9AVbVPGgShrUFQ3g0uCNIPUFXtmwZBaGsQlHeDCoL0A1RV+6hBENoaBOXdYIIg/QBV1b5q\nEIS2BkF5N4ggSD9AVbXPGgShrUFQ3vU+CNIPUFXtuwZBaGsQlHe9DoL0A1RVh6BBENoaBOVdb4Mg\n/QBV1aFoEIS2BkF518sgSD9AVXVIGgShrUFQ3vUuCNIPUFUdmgZBaGsQlHe9CoLUB1dVHaoGQWhr\nEJR3vQmCtAdWVR26BkFoaxCUd70IgpQHVVUdi7Mg+HlCEPzUIChuDYKa2zMeVFV1TC4YBK8mBMFL\nCwTBu3MGwYsLBMEP5gyCLywQBN+eMwieXSAI3s4KgmZ2hIiILEDTNPuAfwXeB75I9x/1KMeAo8Br\nwL9U7PYAbwJPAoeB/6nYvgC8AlwEtit2u4CvAx+f/W/crthuAWeAbwB/X7HbCVwCPgn8GV14RXmG\n7vP6FvA3FbsdwHlgk+6VjZ9UbPcD54B3gK9U7DaA14Gn6f48/HPbtncq9nNjDIiILImmabbo/tIR\nWQZ/1Lbtv6/jgYwBEZEl0TTNU8Bnsu+Q0fBjXxkQERGRtfBE9gEiIiKSizEgIiIycYwBERGRiWMM\niIiITBxjQEREZOIYAyIiIhPHGBAREZk4xoCIiMjEMQZEREQmjjEgIiIycYwBERGRiWMMiIiITBxj\nQEREZOIYAyIiIhPHGBAREZk4xoCIiMjEMQZEREQmjjEgIiIycYwBERGRiWMMiIiITBxjQEREZOIY\nAyIiIhPHGBAREZk4xoCIiMjEMQZEREQmjjEgIiIycYwBERGRiWMMiIiITBxjQEREZOIYAyIiIhPH\nGBAREZk4xoCIiMjEMQZEREQmjjEgIiIycYwBERGRiWMMiIiITJz/B9ypb812Kp2EAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1062721d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "N = 10\n",
    "M = 10\n",
    "\n",
    "n0 = -N/2.\n",
    "m0 = -M/2.\n",
    "#n0 = 1\n",
    "#m0 = 1\n",
    "\n",
    "dn = 0.1\n",
    "dm = 0.1\n",
    "\n",
    "xx0 = np.arange(n0, n0+N+dn, dn)\n",
    "xx1 = np.arange(m0, m0+M+dm, dm)\n",
    "\n",
    "P = []\n",
    "\n",
    "for i in range(M+1):\n",
    "    pt = np.c_[xx0, (i+m0)*np.ones_like(xx0)].T\n",
    "    P.append(pt)\n",
    "\n",
    "for j in range(N+1):\n",
    "    pt = np.c_[(j+n0)*np.ones_like(xx1), xx1].T\n",
    "    P.append(pt)\n",
    "\n",
    "#for p in P:\n",
    "#    plt.plot(p[0,:],p[1,:],'k-')\n",
    "\n",
    "A = np.array([[1,0.5],[-1,0]])\n",
    "for p in P:\n",
    "    z = A.dot(p)\n",
    "    plt.plot(z[0,:],z[1,:],'k-')\n",
    "\n",
    "#for p in P:\n",
    "#    plt.plot(np.sqrt(p[0,:]**2 + p[1,:]**2), np.arctan2(p[1,:], p[0,:]) ,'k-')\n",
    "    \n",
    "#for p in P:\n",
    "#    plt.plot(p[0,:]*np.cos(2*np.pi*p[1,:]/(N)), p[0,:]*np.sin(2*np.pi*p[1,:]/(N)),'k-')\n",
    "\n",
    "#for p in P:\n",
    "#    plt.plot(np.exp(p[0,:]/M), np.exp(p[1,:]/N),'k-')\n",
    "\n",
    "#for p in P:\n",
    "#    plt.plot(1/p[0,:], 1/p[1,:],'k-')\n",
    "    \n",
    "ax = plt.gca()\n",
    "ax.axis('equal')\n",
    "ax.axis('off')\n",
    "#ax.set_xlim([n0-1,n0+N+1])\n",
    "#ax.set_ylim([m0-1, m0+M+1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the key points in using tensor calculus is the fact that vectors should be considered as geometrical objects rather than a collection of numbers. This is especially difficult for persons from a computer science background as the term vector is used as a synonym for an array of numbers. The actual numbers that are needed to represent the vector are always with respect to an ambient basis; when this basis is changed, the numbers must accordingly change but the vector should be the same vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The directional derivative\n",
    "\n",
    "$F(x)$ where $x = x_0 + s v$. Here $s$ is a scalar and $v$ is a vector pointing into a direction.\n",
    "\n",
    "The directional derivative is\n",
    "$$\n",
    "\\frac{d F(x_0 + s v)}{d s} = \\nabla_x F \\cdot v \n",
    "$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor calculus achieves invariance by establishing rules for forming expressions\n",
    "that evaluate to the same value in all coordinate systems.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chain Rule, Partial Derivatives and Einstein notation\n",
    "\n",
    "Take $F(a^1, a^2, a^3)$\n",
    "\n",
    "$$\n",
    "f(\\mu_1, \\mu_2) =  F(A^1(\\mu^1, \\mu^2), A^2(\\mu^1, \\mu^2), A^3(\\mu^1, \\mu^2))\n",
    "$$\n",
    "\n",
    "Is equivalently expressed as\n",
    "\n",
    "$$\n",
    "f(\\mu) =  F(A(\\mu))\n",
    "$$\n",
    "\n",
    "The function as a graph is shown below:\n",
    "<img src=\"../feed_forward1.png\" width=\"200\">\n",
    "\n",
    "Here, we apply the convention that rounded squares are variable nodes and diamonds are functions of several variables. One key property of our graphical notation is that we distinguish between functions and their values. In many calculus texts, this distinction is not explicitely stated and is a source of confusion. \n",
    "\n",
    "The partial derivatives\n",
    "$$\n",
    "\\frac{\\partial}{\\partial \\mu^1} F(A^1(\\mu^1, \\mu^2), A^2(\\mu^1, \\mu^2), A^3(\\mu^1, \\mu^2)) = \n",
    "\\frac{\\partial{F}}{\\partial{a^1}}\\frac{\\partial{A^1}}{\\partial{\\mu^1}} + \\frac{\\partial{F}}{\\partial{a^2}}\\frac{\\partial{A^2}}{\\partial{\\mu^1}} + \\frac{\\partial{F}}{\\partial{a^3}}\\frac{\\partial{A^3}}{\\partial{\\mu^1}}\n",
    "$$\n",
    "\n",
    "These are the products of all the derivatives over all the paths that connect $\\mu^1$ with $F$\n",
    "\n",
    "\n",
    "This is compactly denoted by \n",
    "\n",
    "$$\n",
    "\\frac{\\partial{F}}{\\partial \\mu^\\alpha} = \\frac{\\partial{F}}{\\partial a^i} \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \n",
    "$$\n",
    "\n",
    "where repeated index means summation.\n",
    "\n",
    "The second derivative is\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2{F}}{\\partial \\mu^\\alpha \\partial \\mu^\\beta} & = & \n",
    "\\frac{\\partial}{\\partial \\mu^\\beta}\\left(\\frac{\\partial{F}}{\\partial a^i} \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \\right) \\\\\n",
    "& = & \n",
    "(\\frac{\\partial}{\\partial \\mu^\\beta} \\frac{\\partial{F}}{\\partial a^i} ) \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} +  \\frac{\\partial{F}}{\\partial a^i} (\\frac{\\partial}{\\partial \\mu^\\beta} \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha}) \\\\\n",
    "& = & \n",
    "\\frac{\\partial^2{F}}{\\partial a^i \\partial a^j}  \\frac{\\partial A^j}{\\partial \\mu^\\beta}  \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} +  \\frac{\\partial{F}}{\\partial a^i} \\frac{\\partial^2{A^i}}{\\partial \\mu^\\beta \\partial \\mu^\\alpha} \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "The third derivative is\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^3{F}}{\\partial \\mu^\\alpha \\partial \\mu^\\beta \\partial \\mu^\\gamma} & = & \\frac{\\partial}{\\partial \\mu^\\gamma}\\left(\\frac{\\partial^2{F}}{\\partial a^i \\partial a^j}  \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \\frac{\\partial A^j}{\\partial \\mu^\\beta}  +  \\frac{\\partial{F}}{\\partial a^i}  \\frac{\\partial^2{A^i}}{\\partial \\mu^\\beta \\partial \\mu^\\alpha}  \\right) \\\\\n",
    "& = & \\frac{\\partial^3{F}}{\\partial a^i \\partial a^j \\partial a^k}  \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \\frac{\\partial A^j}{\\partial \\mu^\\beta}\\frac{\\partial A^k}{\\partial \\mu^\\gamma} +\n",
    "\\frac{\\partial^2{F}}{\\partial a^i \\partial a^j}  \\frac{\\partial^2{A^i}}{\\partial \\mu^\\alpha \\partial \\mu^\\gamma} \\frac{\\partial A^j}{\\partial \\mu^\\beta} +\n",
    "\\frac{\\partial^2{F}}{\\partial a^i \\partial a^j}  \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \\frac{\\partial^2 A^j}{\\partial \\mu^\\beta \\partial \\mu^\\gamma} + \\\\\n",
    "& & \\frac{\\partial^2{F}}{\\partial a^i \\partial a^k} \\frac{\\partial A^k}{\\partial \\mu^\\gamma} \\frac{\\partial^2{A^i}}{\\partial \\mu^\\beta \\partial \\mu^\\alpha} +\n",
    "\\frac{\\partial{F}}{\\partial a^i} \\frac{\\partial^3{A^i}}{\\partial \\mu^\\alpha \\partial \\mu^\\beta  \\partial \\mu^\\gamma} \\\\\n",
    "& = & \\frac{\\partial^3{F}}{\\partial a^i \\partial a^j \\partial a^k}  \\frac{\\partial{A^i}}{\\partial \\mu^\\alpha} \\frac{\\partial A^j}{\\partial \\mu^\\beta}\\frac{\\partial A^k}{\\partial \\mu^\\gamma} +\n",
    "\\frac{\\partial^2{F}}{\\partial a^i \\partial a^j} \\left( \\frac{\\partial^2{A^i}}{\\partial \\mu^\\alpha \\partial \\mu^\\gamma} \\frac{\\partial A^j}{\\partial \\mu^\\beta} +\n",
    "\\frac{\\partial^2 A^i}{\\partial \\mu^\\beta \\partial \\mu^\\gamma}  \\frac{\\partial{A^j}}{\\partial \\mu^\\alpha}  +    \\frac{\\partial^2{A^i}}{\\partial \\mu^\\beta \\partial \\mu^\\alpha} \\frac{\\partial A^j}{\\partial \\mu^\\gamma} \\right) +\n",
    "\\frac{\\partial{F}}{\\partial a^i} \\frac{\\partial^3{A^i}}{\\partial \\mu^\\alpha \\partial \\mu^\\beta  \\partial \\mu^\\gamma} \n",
    "\\end{eqnarray}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inverse Function \n",
    "\n",
    "\n",
    "$\\phi$ and $f$ are inverse functions.\n",
    "\n",
    "This is written as \n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\xi & = & \\phi(x) \\\\\n",
    "x & = & f(\\xi) \n",
    "\\end{eqnarray}\n",
    "\n",
    "$$f(\\phi(x)) = (f \\circ \\phi)(x) = x$$\n",
    "\n",
    "<img src=\"../feed_forward2.png\" width=\"250\">\n",
    "\n",
    "The chain rule implies\n",
    "$$ \\frac{d\\phi}{dx} \\frac{d f}{d\\xi} = 1$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "However, it is difficult to invent a new name for each function, so we reuse the name of the variables $x$ instead of $f$ and $\\xi$ instead of $\\phi$.\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\xi & = & \\xi(x) \\\\\n",
    "x & = & x(\\xi) \n",
    "\\end{eqnarray}\n",
    "\n",
    "Now, we can write: $$x(\\xi(x)) = x$$\n",
    "\n",
    "<img src=\"../feed_forward2b.png\" width=\"250\">\n",
    "\n",
    "The chain rule implies\n",
    "$$ \\frac{d\\xi}{dx} \\frac{d x}{d\\xi} = 1$$\n",
    "\n",
    "This overloaded notation is simpler but we have to remind ourselves always the distinction between the variable $x$ and the function $x()$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "x(\\xi(x)) &=& x \\\\\n",
    "x'(\\xi(x)) \\xi'(x) & = & 1 \\\\\n",
    "x''(\\xi(x)) (\\xi'(x))^2 + x'(\\xi(x)) \\xi''(x) & = & 0 \\\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "F(f(x,y), g(x, y)) &=& x \\\\\n",
    "G(f(x,y), g(x, y)) &=& y \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "Let $X = f(x,y)$ and $Y = g(x,y)$\n",
    "\n",
    "<img src=\"../feed_forward3.png\" width=\"250\">\n",
    "\n",
    "\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial F}{\\partial X} \\frac{\\partial f}{\\partial x} + \\frac{\\partial F}{\\partial Y} \\frac{\\partial g}{\\partial x} & = & 1 \\\\\n",
    "\\frac{\\partial F}{\\partial X} \\frac{\\partial f}{\\partial y} + \\frac{\\partial F}{\\partial Y} \\frac{\\partial g}{\\partial y} & = & 0 \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial G}{\\partial X} \\frac{\\partial f}{\\partial x} + \\frac{\\partial G}{\\partial Y} \\frac{\\partial g}{\\partial x} & = & 0 \\\\\n",
    "\\frac{\\partial G}{\\partial X} \\frac{\\partial f}{\\partial y} + \\frac{\\partial G}{\\partial Y} \\frac{\\partial g}{\\partial y} & = & 1 \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\left( \\begin{array}{cc} \n",
    "\\frac{\\partial F}{\\partial X} & \\frac{\\partial F}{\\partial Y} \\\\\n",
    "\\frac{\\partial G}{\\partial X} & \\frac{\\partial G}{\\partial Y}\n",
    "\\end{array} \\right) \n",
    "\\left( \\begin{array}{cc} \n",
    "\\frac{\\partial f}{\\partial x} & \\frac{\\partial f}{\\partial y} \\\\\n",
    "\\frac{\\partial g}{\\partial x} & \\frac{\\partial g}{\\partial y}\n",
    "\\end{array} \\right) & = & \\left( \\begin{array}{cc} \n",
    "1 & 0 \\\\ 0 & 1\n",
    "\\end{array} \\right)\n",
    "\\end{eqnarray}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial x(\\xi(x))}{\\partial x_i} $$\n",
    "\n",
    "$$  \\frac{\\partial x^j}{\\partial \\xi^k} \\frac{\\partial \\xi^k}{\\partial x^i} = \\delta^j_{i}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with coordinates $Z$.\n",
    "\n",
    "The $i$'th (contravariant) component of the coordinates $Z$ is $Z^i$.\n",
    "\n",
    "Define a position vector $\\mathbf{R}$. \n",
    "\n",
    "Overload the notation: $\\mathbf{R}(Z)$ is a function that gives the position vector at coordinates $Z$. In $N$ dimensional space, we have\n",
    "$$\n",
    "\\mathbf{R}(Z^1, Z^2, \\dots, Z^i, \\dots, Z^N)\n",
    "$$\n",
    "\n",
    "The derivative vector $\\mathbf{R}'(h)$ of the position is an entirely geometrical concept. Take a scalar parameter $h$ and let $\\mathbf{R}(h)$ be a curve.\n",
    "\n",
    "Vectors can be (i) added, (ii) multiplied by scalars, (iii) can be said to approach a limit.\n",
    "$$\n",
    "\\lim_{h\\rightarrow 0} |\\mathbf{A}(h) - \\mathbf{B}| = 0\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{R}'(\\alpha) = \\lim_{h\\rightarrow 0 }\\frac{\\mathbf{R}(\\alpha + h) - \\mathbf{R}(\\alpha)}{h}\n",
    "$$\n",
    "\n",
    "Think of a circle in 2D, the position vector is $\\mathbf{A}(\\alpha)$ at $\\alpha$ angles. The derivative is tangential and points to the direction of positive $h$. We can show that $\\mathbf{R}'(\\alpha)$ and $\\mathbf{R}(\\alpha)$ are orthogonal in purely geometric terms\n",
    "\n",
    "$$\n",
    "\\mathbf{R}(\\alpha) \\cdot \\mathbf{R}(\\alpha) = 1\n",
    "$$\n",
    "Take derivatives on both sides\n",
    "$$\n",
    "\\mathbf{R}(\\alpha) \\cdot \\mathbf{R}'(\\alpha) +  \\mathbf{R}'(\\alpha) \\cdot  \\mathbf{R}(\\alpha)= 0\n",
    "$$\n",
    "Hence\n",
    "$$\n",
    "\\mathbf{R}(\\alpha) \\cdot \\mathbf{R}'(\\alpha) = 0\n",
    "$$\n",
    "\n",
    "For the second derivative, we differentiate again\n",
    "$$\n",
    "\\mathbf{R}'(\\alpha) \\cdot \\mathbf{R}'(\\alpha) + \\mathbf{R}(\\alpha) \\cdot \\mathbf{R}''(\\alpha) = 0\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{R}(\\alpha) \\cdot \\mathbf{R}''(\\alpha) = -1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Definition: Covariant basis\n",
    "\n",
    "$$\\mathbf{Z}_i = \\frac{\\partial \\mathbf{R}(Z)}{\\partial Z^i}$$\n",
    "\n",
    "The coordinate vector $\\mathbf{Z}_i$ at position $\\mathbf{R}$ in space is tangential to the $i$'th coordinate curve.\n",
    "\n",
    "An invariant vector in space can be expanded in terms of the covariant basis:\n",
    "$$\n",
    "\\mathbf{V} = V^i \\mathbf{Z}_i\n",
    "$$\n",
    "where $V^i$ are the contravariant components.\n",
    "\n",
    "Definition: Covariant metric tensor is $Z_{i,j} = \\mathbf{Z}_i \\cdot \\mathbf{Z}_j$.\n",
    "\n",
    "Dot product of any two vectors at a position is $\\mathbf{V}\\cdot \\mathbf{U} = U^i V^j Z_{i,j}$\n",
    "\n",
    "Definition: Contravariant metric tensor $Z^{i,j}$ is given by\n",
    "\n",
    "$Z^{i,j} Z_{j,k} = \\delta^{i}_k$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contravariant basis\n",
    "$$\\mathbf{Z}^i = Z^{i,j} \\mathbf{Z}_j $$\n",
    "\n",
    "\n",
    "\\begin{eqnarray}\n",
    "Z_{k,i} \\mathbf{Z}^i &=&  \\mathbf{Z}_k \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "$$\\mathbf{Z}^i \\cdot \\mathbf{Z}_k  = Z^{i,j} \\mathbf{Z}_j \\cdot \\mathbf{Z}_k = Z^{i,j} Z_{j,k} =\\delta^i_k$$\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\cos \\alpha = \\frac{\\mathbf{Z}^1 \\cdot \\mathbf{Z}_1}{|\\mathbf{Z}^1||\\mathbf{Z}_1|} = \\frac{1}{|\\mathbf{Z}^1||\\mathbf{Z}_1|} \\geq 0 \n",
    "\\end{eqnarray}\n",
    "\n",
    "\n",
    "The length of a curve\n",
    "$$\n",
    "L = \\int_a^b \\sqrt{Z_{i,j} \\frac{dZ^i}{dt} \\frac{dZ^j}{dt}} dt\n",
    "$$\n",
    "\n",
    "$\\mathbf{Z}^i \\cdot \\mathbf{Z}^j \\mathbf{Z}_j \\cdot \\mathbf{Z}_k = \\delta^{i}_k$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition: Christoffel Symbol\n",
    "\n",
    "Christoffel Symbol measures the variation of the covariant basis from point to point\n",
    "$$\n",
    "\\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} = \\Gamma^{k}_{ij} \\mathbf{Z}_k \n",
    "$$\n",
    "\n",
    "Each basis vector $\\mathbf{Z}_i$ changes with respect to each coordinate $\\partial Z^j$ to provide $N$ vectors (with $N^2$ numbers). Each vector $\\mathbf{Z}_i$ is to be decomposed on the covariant basis. The Christoffel symbol gives the components of these vectors.\n",
    "\n",
    "As \n",
    "$$\n",
    "\\mathbf{Z}_i  = \\frac{\\partial \\mathbf{R}(Z) }{\\partial Z^i} \n",
    "$$\n",
    "We have \n",
    "$$\n",
    "\\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} = \\frac{\\partial \\mathbf{R} }{\\partial Z^i \\partial Z^j} =  \\frac{\\partial \\mathbf{Z}_j }{\\partial Z^i}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\Gamma^{k}_{ij} =\n",
    "\\Gamma^{k}_{ji}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Christoffel for the polar coordinates $(r, \\theta) \\equiv (Z^1, Z^2)$.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1}  =  \\lim_{h\\rightarrow 0} \\frac{ \\mathbf{R}(Z^1 + h, Z^2) - \\mathbf{R}(Z^1, Z^2)}{h}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^2}  =  \\lim_{h\\rightarrow 0} \\frac{ \\mathbf{R}(Z^1, Z^2+h) - \\mathbf{R}(Z^1, Z^2)}{h}\n",
    "$$\n",
    "\n",
    "Take the two unit vectors $\\mathbf{e}_1$ and $\\mathbf{e}_2$\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\mathbf{R}(Z^1, Z^2) & = & Z^1 \\cos(Z^2) \\mathbf{e}_1 + Z^1 \\sin(Z^2) \\mathbf{e}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "The covariant coordinate basis elements $\\mathbf{Z}_k$ are\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1} & = & \\cos(Z^2) \\mathbf{e}_1 + \\sin(Z^2) \\mathbf{e}_2 = \\mathbf{Z}^1\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^2} & = & -Z^1 \\sin(Z^2) \\mathbf{e}_1 + Z^1 \\cos(Z^2) \\mathbf{e}_2 = \\mathbf{Z}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1 \\partial Z^1} & = & 0\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1 \\partial Z^2} & = & - \\sin(Z^2) \\mathbf{e}_1 + \\cos(Z^2) \\mathbf{e}_2 \\\\\n",
    "& = & \\frac{1}{Z^1} \\mathbf{Z}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^2 \\partial Z^2} & = & -Z^1 \\cos(Z_2) \\mathbf{e}_1 - Z_1 \\sin(Z_2) \\mathbf{e}_2 \\\\\n",
    "& = & - Z^1 \\mathbf{Z}_1\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\left(\\begin{array}{cc} \\Gamma^{1}_{11} & \\Gamma^{1}_{12} \\\\ \\Gamma^{1}_{21} & \\Gamma^{1}_{22} \\end{array}\\right)  & = & \\left(\\begin{array}{cc} 0 & 0 \\\\ 0 & - Z^1 \\end{array}\\right) \n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\left(\\begin{array}{cc} \\Gamma^{2}_{11} & \\Gamma^{2}_{12} \\\\ \\Gamma^{2}_{21} & \\Gamma^{2}_{22} \\end{array}\\right)  & = & \\left(\\begin{array}{cc} 0 & {1}/{Z^1} \\\\ {1}/{Z^1} & 0 \\end{array}\\right) \n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Christoffel for the scaled polar coordinates $(r, u) \\equiv (Z^1, Z^2)$.\n",
    "\n",
    "\n",
    "Take the two unit vectors $\\mathbf{e}_1$ and $\\mathbf{e}_2$\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\mathbf{R}(Z^1, Z^2) & = & Z^1 \\cos(2\\pi Z^2 /N) \\mathbf{e}_1 + Z^1 \\sin(2\\pi Z^2 /N) \\mathbf{e}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "The covariant coordinate basis elements $\\mathbf{Z}_k$ are\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1} & = & \\cos(2\\pi Z^2 /N) \\mathbf{e}_1 + \\sin(2\\pi Z^2 /N) \\mathbf{e}_2 = \\mathbf{Z}^1\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{R}(Z^1, Z^2)}{\\partial Z^2} & = & - \\frac{2\\pi}{N} Z^1 \\sin(2\\pi Z^2 /N) \\mathbf{e}_1 + \\frac{2\\pi}{N} Z^1 \\cos(2\\pi Z^2 /N) \\mathbf{e}_2 = \\mathbf{Z}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1 \\partial Z^1} & = & 0\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^1 \\partial Z^2} & = & - \\frac{2\\pi}{N}\\sin(2\\pi Z^2 /N) \\mathbf{e}_1 + \\frac{2\\pi}{N}\\cos(2\\pi Z^2 /N) \\mathbf{e}_2 \\\\\n",
    "& = & \\frac{1}{Z^1} \\mathbf{Z}_2\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 \\mathbf{R}(Z^1, Z^2)}{\\partial Z^2 \\partial Z^2} & = & -\\left(\\frac{2\\pi}{N}\\right)^2 Z^1 \\cos(2\\pi Z^2 /N) \\mathbf{e}_1 - \\left(\\frac{2\\pi}{N}\\right)^2 Z_1 \\sin(2\\pi Z^2 /N) \\mathbf{e}_2 \\\\\n",
    "& = & - \\left(\\frac{2\\pi}{N}\\right)^2 Z^1 \\mathbf{Z}_1\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\left(\\begin{array}{cc} \\Gamma^{1}_{11} & \\Gamma^{1}_{12} \\\\ \\Gamma^{1}_{21} & \\Gamma^{1}_{22} \\end{array}\\right)  & = & \\left(\\begin{array}{cc} 0 & 0 \\\\ 0 & - (2\\pi/N)^2 Z^1 \\end{array}\\right) \n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\left(\\begin{array}{cc} \\Gamma^{2}_{11} & \\Gamma^{2}_{12} \\\\ \\Gamma^{2}_{21} & \\Gamma^{2}_{22} \\end{array}\\right)  & = & \\left(\\begin{array}{cc} 0 & {1}/{Z^1} \\\\ {1}/{Z^1} & 0 \\end{array}\\right) \n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Properties of the Christoffel symbol\n",
    "\n",
    "The Christoffel symbol $\\Gamma^k_{ij}$ is the $k$'th contravariant component of the derivative of the $i$'th covariant coordinate vector with respect to its $j$'th component, as expressed in the covariant basis $\\mathbf{Z}_k$.\n",
    "Mathematically, this is:\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} &=& \\Gamma^{k}_{ij} \\mathbf{Z}_k \\\\\n",
    "\\end{eqnarray}\n",
    "Taking the dot product with the dual basis $\\mathbf{Z}^l$, we obtain\n",
    "\\begin{eqnarray}\n",
    "\\mathbf{Z}^l \\cdot \\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} &=& \\Gamma^{k}_{ij} \\mathbf{Z}_k \\cdot \\mathbf{Z}^l = \\Gamma^{k}_{ij} \\delta_k^l\\\\\n",
    "\\mathbf{Z}^l \\cdot \\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} &=& \\Gamma^{l}_{ij}  \\\\\n",
    "\\mathbf{Z}^k \\cdot \\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} &=& \\Gamma^{k}_{ij}  \\\\\n",
    "\\end{eqnarray}\n",
    "The last step merely replaces index $k$ with $l$.\n",
    "\n",
    "A natural question is how is the contravariant basis $\\mathbf{Z}^k$ is changing with respect to its components, i.e.\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial\\mathbf{Z}^k}{\\partial Z^j} = ?\n",
    "\\end{eqnarray}\n",
    "\n",
    "The result is provided by the chain rule: We observe that\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial (\\mathbf{Z}^k \\cdot \\mathbf{Z}_i)}{\\partial Z^j} &=& \\frac{\\partial\\mathbf{Z}^k}{\\partial Z^j} \\cdot \\mathbf{Z}_i + \\mathbf{Z}^k \\cdot \\frac{\\partial \\mathbf{Z}_i}{\\partial Z^j} \n",
    "\\end{eqnarray}\n",
    "with $\\mathbf{Z}^k \\cdot \\mathbf{Z}_i) = \\delta^k_i$. As the Kronecker delta is not changing with respect to $Z^j$, we have ${\\partial \\delta^k_i }/{\\partial Z^j} = 0$ and \n",
    "\\begin{eqnarray}\n",
    "0 & = & \\frac{\\partial\\mathbf{Z}^k}{\\partial Z^j} \\cdot \\mathbf{Z}_i + \\Gamma^{k}_{ij} \n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial\\mathbf{Z}^k}{\\partial Z^j}  = - \\Gamma^{k}_{ij} \\mathbf{Z}^i\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "\\frac{\\partial Z_{i,j}}{\\partial Z^k} & = & \\frac{\\partial (\\mathbf{Z}_i \\cdot \\mathbf{Z}_j) }{\\partial Z^k} = \\frac{\\partial \\mathbf{Z}_i  }{\\partial Z^k} \\cdot \\mathbf{Z}_j + \\frac{\\partial \\mathbf{Z}_j  }{\\partial Z^k} \\cdot \\mathbf{Z}_i \\\\\n",
    "& = & \\frac{\\partial \\mathbf{Z}_i  }{\\partial Z^k} \\cdot Z_{l,j} \\mathbf{Z}^l + \\frac{\\partial \\mathbf{Z}_j  }{\\partial Z^k} \\cdot Z_{l,i} \\mathbf{Z}^l \\\\\n",
    "& = & \\Gamma^{l}_{ik} Z_{l,j} + \\Gamma^{l}_{jk} Z_{l,i}\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "\\frac{\\partial Z_{i,m}}{\\partial Z^j} & = & \\Gamma^{l}_{ij} Z_{l,m} + \\Gamma^{l}_{mj} Z_{l,i} \\\\\n",
    "\\frac{\\partial Z_{j,m}}{\\partial Z^i} & = & \\Gamma^{l}_{ji} Z_{l,m} + \\Gamma^{l}_{mi} Z_{l,j} \\\\\n",
    "\\frac{\\partial Z_{j,i}}{\\partial Z^m} & = & \\Gamma^{l}_{jm} Z_{l,i} + \\Gamma^{l}_{im} Z_{l,j} \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "Z^{k,m}(\\frac{\\partial Z_{i,m}}{\\partial Z^j} + \\frac{\\partial Z_{j,m}}{\\partial Z^i} - \\frac{\\partial Z_{j,i}}{\\partial Z^m})\n",
    " &=& \\Gamma^{l}_{ij} Z_{l,m} Z^{k,m} + \\Gamma^{l}_{mj} Z_{l,i}Z^{k,m} + \\Gamma^{l}_{ji} Z_{l,m}Z^{k,m} + \\Gamma^{l}_{mi} Z_{l,j}Z^{k,m} - \\Gamma^{l}_{jm} Z_{l,i}Z^{k,m} - \\Gamma^{l}_{im} Z_{l,j} Z^{k,m} \\\\\n",
    " &=& \\Gamma^{l}_{ij} \\delta_{l}^k  + \\Gamma^{l}_{mj} Z_{l,i}Z^{k,m} + \\Gamma^{l}_{ji} \\delta_{l}^k  + \\Gamma^{l}_{mi} Z_{l,j}Z^{k,m} - \\Gamma^{l}_{jm} Z_{l,i}Z^{k,m} - \\Gamma^{l}_{im} Z_{l,j} Z^{k,m} \\\\\n",
    " &=& 2 \\Gamma^{k}_{ij}   + \\Gamma^{l}_{mj} Z_{l,i}Z^{k,m}  + \\Gamma^{l}_{mi} Z_{l,j}Z^{k,m} - \\Gamma^{l}_{jm} Z_{l,i}Z^{k,m} - \\Gamma^{l}_{im} Z_{l,j} Z^{k,m} \\\\\n",
    " & = &  2 \\Gamma^{k}_{ij} \n",
    "\\end{eqnarray}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jacobians\n",
    "\n",
    "<img src=\"../feed_forward.png\" width=\"300\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Covariant tensor of order one\n",
    "\n",
    "$T_i = T_{i'} J_i^{i'}$\n",
    "\n",
    "Contravariant tensor of order one\n",
    "\n",
    "$T^i = T^{i'} J^i_{i'}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "J^{i}_{i',j'} = \\frac{\\partial^2 Z^{i} }{\\partial Z^{i'} \\partial Z^{j'}} = \\frac{\\partial }{\\partial Z^{i'}} \\left( \\frac{\\partial Z^i}{\\partial Z^{j'}} \\right) \n",
    "\\end{eqnarray}\n",
    "\n",
    "Note that the derivative of the Jacobian is always between the primed and unprimed indices.\n",
    "\n",
    "\\begin{eqnarray}\n",
    " \\delta^{i}_{j} & = &  \\frac{\\partial Z^{i} }{\\partial Z^{i'}} \\frac{\\partial Z^{i'} }{\\partial Z^{j}} \\\\\n",
    "\\frac{\\partial }{\\partial Z^{k}} \\delta^{i}_{j} & = &  \\frac{\\partial }{\\partial Z^{k}}(\\frac{\\partial Z^{i} }{\\partial Z^{i'}} \\frac{\\partial Z^{i'} }{\\partial Z^{j}}) \\\\\n",
    "\\frac{\\partial}{\\partial Z^{k}} & = & \\frac{\\partial Z^{j'}}{\\partial Z^{k}} \\frac{\\partial}{\\partial Z^{j'}} \\\\\n",
    "0 & = & (\\frac{\\partial}{\\partial Z^{k}}  \\frac{\\partial Z^{i} }{\\partial Z^{i'} }) \\frac{\\partial Z^{i'} }{\\partial Z^{j}} + \\frac{\\partial Z^{i} }{\\partial Z^{i'}} \\frac{\\partial }{\\partial Z^{k}}\\frac{\\partial Z^{i'} }{\\partial Z^{j}} \\\\\n",
    "0 & = & (\\frac{\\partial Z^{j'}}{\\partial Z^{k}} \\frac{\\partial}{\\partial Z^{j'}}  \\frac{\\partial Z^{i} }{\\partial Z^{i'} }) \\frac{\\partial Z^{i'} }{\\partial Z^{j}} + \\frac{\\partial Z^{i} }{\\partial Z^{i'}} \\frac{\\partial }{\\partial Z^{k}}\\frac{\\partial Z^{i'} }{\\partial Z^{j}} \\\\\n",
    "0 & = & J^{j'}_k J^{i}_{i',j'} J^{i'}_{j} + J^{i}_{i'} J^{i'}_{k,j}\n",
    "\\end{eqnarray}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Tensor or not a tensor?\n",
    "\n",
    "The covariant basis $\\mathbf{Z}_i$ is a tensor\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{R}(Z')}{\\partial Z^{i'}} &=& \\frac{\\partial \\mathbf{R}(Z(Z'))}{\\partial Z^{i'}} = \\frac{\\partial \\mathbf{R}}{\\partial Z^i} \\frac{\\partial Z^i}{\\partial Z^{i'}} \\\\\n",
    "\\mathbf{Z}_{i'} &=& \\mathbf{Z}_{i} J^i_{i'}\n",
    "\\end{eqnarray}\n",
    "\n",
    "- The covariant metric tensor is indeed a tensor.\n",
    "\n",
    "The partial derivative of an invariant (scalar quantity) is a tensor:\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial F}{\\partial Z^{i'}} = \\frac{\\partial F}{\\partial Z^{i}} \\frac{\\partial Z^i}{\\partial Z^{i'}} =  \\frac{\\partial F}{\\partial Z^{i}}J^{i}_{i'}\n",
    "\\end{eqnarray}\n",
    "\n",
    "So, ${\\partial F}/{\\partial Z^{i}}$ is a tensor. \n",
    "\n",
    "The second derivative is not a tensor\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial^2 F}{\\partial Z^{i'} \\partial Z^{j'}} = \\frac{\\partial^2 F}{\\partial Z^{i} \\partial Z^{j}} \\frac{\\partial Z^i}{\\partial Z^{i'}} + \\frac{\\partial F}{\\partial Z^{i}} \\frac{\\partial^2 Z^i}{\\partial Z^{i'} \\partial Z^{j'}}  \n",
    "\\end{eqnarray}\n",
    "So, ${\\partial^2 F}/{\\partial Z^{i} \\partial Z^{j}}$ is not a tensor. \n",
    "\n",
    "The derivative of an order one covariant tensor is not a tensor \n",
    "\n",
    "\\begin{eqnarray}\n",
    "T_{i'} & = & T_i \\frac{\\partial Z^{i}}{\\partial Z^{i'}} \\\\ \n",
    "\\frac{\\partial T_{i'}}{\\partial Z^{j'}} & = & \\frac{\\partial T_{i}}{\\partial Z^{j}} \\frac{\\partial Z^{j}}{\\partial Z^{j'}} \\frac{\\partial Z^{i}}{\\partial Z^{i'}} + T_i \\frac{\\partial^2 Z^{i}}{\\partial Z^{j'} \\partial Z^{i'}} \\\\\n",
    "& = & \\frac{\\partial T_{i}}{\\partial Z^{j}} J^{j}_{j'} J^{i}_{i'} + T_i J^{i}_{i',j'} \n",
    "\\end{eqnarray}\n",
    "\n",
    "The skew-symmetric part is a tensor\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial T_{i}}{\\partial Z^{j}} - \\frac{\\partial T_{j}}{\\partial Z^{i}} & = & \\frac{\\partial T_{i'}}{\\partial Z^{j'}} J^{j'}_{j} J^{i'}_{i} + T_{i'} J^{i'}_{i,j} - \\frac{\\partial T_{j'}}{\\partial Z^{i'}} J^{i'}_{i} J^{j'}_{j} - T_{i'} J^{i'}_{j,i} \\\\\n",
    "& = & (\\frac{\\partial T_{i'}}{\\partial Z^{j'}} - \\frac{\\partial T_{j'}}{\\partial Z^{i'}}) J^{j'}_{j} J^{i'}_{i} \n",
    "\\end{eqnarray}\n",
    "\n",
    "The derivative of an order one contravariant tensor is not a tensor \n",
    "\n",
    "\\begin{eqnarray}\n",
    "T^{i} & = & T^{i'} \\frac{\\partial Z^{i}}{\\partial Z^{i'}} \\\\ \n",
    "\\frac{\\partial T^{i}}{\\partial Z^{j}} & = & \\frac{\\partial T^{i'}}{\\partial Z^{j'}} \\frac{\\partial Z^{j'}}{\\partial Z^{j}}  \\frac{\\partial Z^{i}}{\\partial Z^{i'}} + T^{i'} \\frac{\\partial Z^{j'}}{\\partial Z^{j}} \\frac{\\partial^2 Z^{i}}{\\partial Z^{i'} \\partial Z^{j'}} \\\\\n",
    "& = & \\frac{\\partial T^{i'}}{\\partial Z^{j'}} J^{j'}_j J^{i}_{i'} + T^{i'} J^{j'}_j  J^{i}_{i',j'} \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "The partial derivative of the Christoffel symbol wrt contravariant index\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\Gamma^{k}_{i,j} & = & \\mathbf{Z}^k \\cdot \\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} \n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\Gamma^{k}_{i,j}}{\\partial Z^{k'}} & = & \\frac{\\partial {Z}^{u}}{\\partial Z^{k'}} \\frac{\\partial \\mathbf{Z}^k}{\\partial Z^{u}}  \\cdot \\frac{\\partial \\mathbf{Z}_i }{\\partial Z^j} + \\mathbf{Z}^k \\cdot \\frac{\\partial {Z}^{j'} }{\\partial Z^{j}}  \\frac{\\partial^2 \\mathbf{Z}_i }{\\partial Z^{k'} \\partial Z^{j'}} \\\\\n",
    "& = & J^{u}_{k'} (- \\Gamma^{k}_{u,r} \\mathbf{Z}^r   ) \\Gamma^{s}_{i,j} \\mathbf{Z}_s + \\mathbf{Z}^k \\cdot  J^{j'}_j \\frac{\\partial^2 \\mathbf{Z}_i }{\\partial Z^{k'} \\partial Z^{j'}} \\\\\n",
    "& = & - J^{u}_{k'} \\Gamma^{k}_{u,r} \\mathbf{Z}^r \\cdot \\mathbf{Z}_s  \\Gamma^{s}_{i,j}  + \\mathbf{Z}^k \\cdot  J^{j'}_j \\frac{\\partial^2 \\mathbf{Z}_i }{\\partial Z^{k'} \\partial Z^{j'}} \\\\\n",
    "& = & - J^{u}_{k'} \\Gamma^{k}_{u,r} \\delta^r_s \\Gamma^{s}_{i,j}  + \\mathbf{Z}^k \\cdot  J^{j'}_j \\frac{\\partial^2 \\mathbf{Z}_i }{\\partial Z^{k'} \\partial Z^{j'}} \\\\\n",
    "& = & - J^{u}_{k'} \\Gamma^{k}_{u,s} \\Gamma^{s}_{i,j}  + \\mathbf{Z}^k \\cdot  J^{j'}_j \\frac{\\partial^2 \\mathbf{Z}_i }{\\partial Z^{k'} \\partial Z^{j'}} \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "The transform rule for the Christoffel symbol\n",
    "\\begin{eqnarray}\n",
    "\\Gamma^{k'}_{i',j'} & = & \\mathbf{Z}^{k'} \\cdot \\frac{\\partial \\mathbf{Z}_{i'} }{\\partial Z^{j'}}\n",
    "\\end{eqnarray}\n",
    "\n",
    "$\\mathbf{Z}_{i'}$ is a covariant tensor, we found the derivatives of tensors as\n",
    "$$\n",
    "\\frac{\\partial \\mathbf{Z}_{i'}}{\\partial Z^{j'}} = \\frac{\\partial \\mathbf{Z}_{i}}{\\partial Z^{j}} J^{j}_{j'} J^{i}_{i'} + \\mathbf{Z}_i J^{i}_{i',j'} \n",
    "$$\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\Gamma^{k'}_{i',j'} & = & (\\mathbf{Z}^{k} J^{k'}_k ) \\cdot (\\frac{\\partial \\mathbf{Z}_{i}}{\\partial Z^{j}} J^{j}_{j'} J^{i}_{i'} + \\mathbf{Z}_i J^{i}_{i',j'} ) \\\\\n",
    "& = & \\mathbf{Z}^{k}  \\cdot \\frac{\\partial \\mathbf{Z}_{i}}{\\partial Z^{j}} J^{k'}_k J^{j}_{j'} J^{i}_{i'} + \\mathbf{Z}^k \\mathbf{Z}_i J^{k'}_k J^{i}_{i',j'} \\\\\n",
    "& = & \\Gamma^{k}_{i,j} J^{k'}_k J^{j}_{j'} J^{i}_{i'} +  J^{k'}_k J^{k}_{i',j'} \n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variant: An object that can be obtained through applying the same rule in a coordinate system. Examples are the covariant basis ($\\mathbf{Z}_i$ or the covariant metric tensor $Z_{i,j}$, the Christoffel symbol).\n",
    "The jacobian $J^{i}_{i'}$ and its derivative $J^{i}_{i',j'}$ are not variants as one needs two coordinate systems for their construction. \n",
    "\n",
    "A tensor $T_i$ is a variant that transforms according to the rule \n",
    "$$\n",
    "T_{i'} = T_i J^{i}_{i'}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Covariant Derivative\n",
    "\n",
    "Take a vector (an invariant) and calculate the partial derivative with respect to the $k$'th contravariant component of the covariant basis \n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{V}}{\\partial Z^{k}} & = & \\frac{\\partial (V^i\\mathbf{Z}_i) }{\\partial Z^{k}}\n",
    "= \\frac{\\partial V^i}{\\partial Z^{k}} \\mathbf{Z}_i + V^i \\frac{\\partial \\mathbf{Z}_i}{\\partial Z^{k}}\n",
    "=  \\frac{\\partial V^i}{\\partial Z^{k}} \\mathbf{Z}_i + V^i \\Gamma_{ik}^j \\mathbf{Z}_j\n",
    "\\end{eqnarray}\n",
    "Changing names of the dummy indices that are contracted over, we find\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{V}}{\\partial Z^{k}} & = & \n",
    "  \\frac{\\partial V^i}{\\partial Z^{k}} \\mathbf{Z}_i + V^i \\Gamma_{ik}^j \\mathbf{Z}_j\n",
    "  = \\frac{\\partial V^i}{\\partial Z^{k}} \\mathbf{Z}_i + V^j \\Gamma_{jk}^i \\mathbf{Z}_i\n",
    "  = \\left( \\frac{\\partial V^i}{\\partial Z^{k}} +  \\Gamma_{jk}^i V^j \\right) \\mathbf{Z}_i\n",
    "\\end{eqnarray}\n",
    "\n",
    "The covariant derivative is \n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\nabla_k T^i \\equiv \\frac{\\partial T^i}{\\partial Z^{k}} +  \\Gamma_{rk}^i T^r \n",
    "\\end{eqnarray}\n",
    "\n",
    "Do the same derivation using the contravariant basis\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{V}}{\\partial Z^{k}} & = & \\frac{\\partial (V_i \\mathbf{Z}^i) }{\\partial Z^{k}}\n",
    "= \\frac{\\partial V_i}{\\partial Z^{k}} \\mathbf{Z}^i + V_i \\frac{\\partial \\mathbf{Z}^i}{\\partial Z^{k}} \n",
    "\\end{eqnarray}\n",
    "\n",
    "From the properties of the Christoffel symbol:\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{Z}^i}{\\partial Z^{k}} = -\\Gamma^{i}_{jk} \\mathbf{Z}^{j}\n",
    "\\end{eqnarray}\n",
    "and exchange dummy indices\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial \\mathbf{V}}{\\partial Z^{k}} & = & \n",
    "\\frac{\\partial V_i}{\\partial Z^{k}} \\mathbf{Z}^i - V_j \\Gamma^{j}_{ik} \\mathbf{Z}^{i} = \\left(\\frac{\\partial V_i}{\\partial Z^{k}} - V_j \\Gamma^{j}_{ik} \\right) \\mathbf{Z}^{i}\n",
    "\\end{eqnarray}\n",
    "\n",
    "Similarly we define \n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\nabla_k T_i \\equiv \\frac{\\partial T_i}{\\partial Z^{k}} -  \\Gamma_{ik}^m T_m \n",
    "\\end{eqnarray}\n",
    "\n",
    "This implies that \n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\nabla_k T^i_j \\equiv \\frac{\\partial T^i_j}{\\partial Z^{k}} +  \\Gamma_{rk}^i T^r_j - \\Gamma_{kj}^m T^i_m\n",
    "\\end{eqnarray}\n",
    "\n",
    "Metrinillic property\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\nabla_k \\mathbf{Z}_i = \\frac{\\partial \\mathbf{Z}_i}{\\partial Z^{k}} - \\Gamma_{ik}^m \\mathbf{Z}_m = \\Gamma_{ik}^m \\mathbf{Z}_m - \\Gamma_{ik}^m \\mathbf{Z}_m = 0\n",
    "\\end{eqnarray}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\cos(\\arccos(x)) = x$\n",
    "\n",
    "$-\\sin(\\arccos(x))\\arccos'(x) = 1$\n",
    "\n",
    "$\\sin^2(\\arccos(x)) + \\cos^2(\\arccos(x)) = 1$\n",
    "\n",
    "$\\sin^2(\\arccos(x)) = 1 - x^2$\n",
    "\n",
    "$\n",
    "\\arccos'(x) = \\pm\\frac{1}{\\sqrt{1-x^2}}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diffeomorphism\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amari"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\newcommand{\\E}[1]{\\left\\langle{#1}\\right\\rangle}$\n",
    "\n",
    "\n",
    "* All the $p(x; \\theta)$ have the same support\n",
    "* $\\ell(x; \\theta) = \\log p(x; \\theta)$\n",
    "* For $i=1\\dots n$, the derivatives as functions in $x$ are linearly independent\n",
    "$$\n",
    "\\frac{\\partial \\ell(x; \\theta)}{\\partial \\theta^i}\n",
    "$$\n",
    "* The moments of ${\\partial \\ell(x; \\theta)}/{\\partial \\theta^i}$ exist.\n",
    "\n",
    "The tangent space\n",
    "$T_\\theta$\n",
    "\n",
    "## Coordinate curves\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\theta_1(t) & = & (\\theta^1_0+t, \\theta^2_0, \\dots, \\theta^n_0)\n",
    "\\end{eqnarray}\n",
    "\n",
    "Tangent $C_1$\n",
    "\\begin{eqnarray}\n",
    "C_1 f & = & \\frac{d}{dt} f(\\theta_1(t))  = \\frac{\\partial f(\\theta_1(t))}{\\partial \\theta^i} \\frac{\\partial \\theta^i}{\\partial t}\n",
    "\\end{eqnarray}\n",
    "is denoted as $\\partial_1$\n",
    "\n",
    "The covariant basis is denoted by $\\partial_i$, so any vector in the tangent space is\n",
    "\\begin{eqnarray}\n",
    "A & = & A^i \\partial_i\n",
    "\\end{eqnarray}\n",
    "As random variables\n",
    "\n",
    "\\begin{eqnarray}\n",
    "T_\\theta^{(1)} = \\left\\{ A(x) : A(x)  =  A^i \\partial_i \\ell(x; \\theta) \\right\\}\n",
    "\\end{eqnarray}\n",
    "\n",
    "\n",
    "\n",
    "\\begin{eqnarray}\n",
    "1 &=& \\int p(x;\\theta) dP(x) \\\\\n",
    "0 & = & \\frac{\\partial}{\\partial \\theta_i} \\int p(x;\\theta) dP(x) = \\int \\frac{\\partial}{\\partial \\theta_i} p(x;\\theta) dP(x) \\\\\n",
    "& = &  \\int p(x;\\theta) \\partial_i \\ell(x; \\theta)  dP(x) = \\E{\\partial_i \\ell(x; \\theta)}\n",
    "\\end{eqnarray}\n",
    "\n",
    "\n",
    "Jacobians\n",
    "\n",
    "\\begin{eqnarray}\n",
    "J^\\alpha_i & = & \\frac{\\partial \\xi^\\alpha}{\\partial \\theta^i}\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\bar{J}^i_\\alpha & = & \\frac{\\partial \\theta^i}{\\partial \\xi^\\alpha}\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\partial_\\alpha & = & \\bar{J}^i_\\alpha \\partial_i\n",
    "\\end{eqnarray}\n",
    "\n",
    "Metric tensor\n",
    "\n",
    "\\begin{eqnarray}\n",
    "g_{i,j} & = & \\E{ \\partial_i \\cdot \\partial_j}\n",
    "\\end{eqnarray}\n",
    "\n",
    "\n",
    "\n",
    "## Standard Normal\n",
    "\\begin{eqnarray}\n",
    "\\theta_1 & = & \\mu \\\\\n",
    "\\theta_2 & = & \\sigma\n",
    "\\end{eqnarray}\n",
    "\n",
    "\\begin{eqnarray}\n",
    "\\xi_1 & = & \\mu \\\\\n",
    "\\xi_2 & = & \\mu^2 + \\sigma^2\n",
    "\\end{eqnarray}\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"../feed_forward2.png\" width=\"250\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"350\"\n",
       "            height=\"100\"\n",
       "            src=\"../latex_figures/feed_forward2.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x10284fc88>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML, display\n",
    "from IPython.display import IFrame\n",
    "\n",
    "display(HTML('<img src=\"../feed_forward2.png\" width=\"250\">'))\n",
    "\n",
    "IFrame(\"../latex_figures/feed_forward2.pdf\", width=350, height=100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "-0.605727292083\n",
      "-0.539235254827\n",
      "[ 2.          0.24682407  0.20232278]\n",
      "[-0.         -0.35868752 -0.36001073]\n",
      "2.0 -0.0\n",
      "0.246824066159 -0.358687519304\n",
      "0.202322781119 -0.360010730582\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "x_1 = np.array([0,1.2,5])\n",
    "x_2 = np.array([0,-3,5])\n",
    "\n",
    "for x,y in zip(x_1,x_2):\n",
    "    print(np.sin(2*x)*np.cos(x*y))\n",
    "\n",
    "    \n",
    "dfdx1 = -np.sin(2*x_1)*np.sin(x_1*x_2)*x_2 + 2*np.cos(x_1)*np.cos(x_1*x_2)\n",
    "dfdx2 = -np.sin(2*x_1)*np.sin(x_1*x_2)*x_1\n",
    "\n",
    "print(dfdx1)\n",
    "\n",
    "print(dfdx2)\n",
    "\n",
    "for i in range(3):\n",
    "    print(dfdx1[i], dfdx2[i])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (py36)",
   "language": "python",
   "name": "py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}