03. Basic Derivative Rules for Combining Functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Derivative Rules for Combining Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen the various functions and how to compute their derivatives. However, when using derivatives in modeling and applications it is rarely that we will achieve anything with a single simple function such is $f(x)=sin(x)$. Usually, we combinine different functions together. The basic ways to combine functions by using algebra involve:\n",
    "- summing (and subtracting) functions\n",
    "- multiplying (and dividing) functions\n",
    "- composing functions (output of one function becomes the input of another)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to compute derivatives of such complex functions, we need to see how these 3 elementary operations translate into the world of calculus. Thus, we will investigate how to differentiate sums (and differences), products (and divisions), and functional composition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differentiating sums (and differences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first consider what summing and subtracting funcions entails. We start with two functions, $f(x)$ and $g(x)$ that are different from each other. We are interested in computing their sum $f(x)+g(x)$. Let's introduce a geometric intuition of this problem. On the animation below, we see a small black triangle representing the variable $x$, sliding across the horizontal axis, oscillating between zero and some positive value. We also see three curves:\n",
    "- the green curve is a graph of the function $f(x)$\n",
    "- the blue curve is a graph of the function $g(x)$\n",
    "- the red curve is a graph of the sum $f(x)+g(x)$\n",
    "\n",
    "What can these curves tell us about their respective functions?  The horizontal axis represents the input space of the function, while the vertical its output space. Each function joins a single value on the horizontal axis with a single value on the vertical axis. Repeating this for each value of input, is equivalent to tracing a curve on the two-dimensional plane. All three functions are dependent on the variable $x$, which means that for every $x$, each function has its own individual output. For some inputs, two functions might have the same output. In our example, when $x$ is equal to zero, the output of both $f(x)$ and $g(x)$ has the same value—zero. You can recognise this situation by looking at the graphs of two functions and recognising points where they intersect. Above the curves there are three vertical lines, whose lengths indicate the output value of each function as $x$ varies. What we can notice is that when the green line and the blue line point in the same direction (both function's output is either over or under the horizontal axis), their individual lengths __add__ into a lenght of the red curve. When the vertical lines are pointing in the opposite directions (one function outputs over the horizontal axis, the other under it), their lenghts cancel each other. We obtain the lenght of the red line by __subtracting__ one input length from another. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/derivatives-31.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have an analoguous situation with adding (and subtracting) numbers. If numbers are of the same sign, (both positive or both negative) their magnitude will increase in eigher positive or in the negative direction. If they have opposite signs, the magnitude in one direction will be subtracted by the magnitude in the oposite direction. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the question is what is the derivative of the sum of the two functions $f(x)$ and $g(x)$? The derivative of this sum can be written as following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} (g(x)+h(x))\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find an answer we need to introduce a slight change to the input of the functions $f(x)$ and $g(x)$, and evaluate how do their outputs change in terms of the input, for a smaller and smaller amount of change. In the animation below, the value $x$ on the horizontal axis is increased by the small amount $dx$ (the increase is made much larger on the drawing). For each function, its input values $x$ and $x+dx$, will have unique output values on the vertical axis. For example, the function $f(x)$ will yield the values $f(x)$ and $f(x+dx)$. We can use these values to construct a green triangle on the animation below. The points will mark its diagonals, and its height corresponds to tge change in the output $df$. The function $g(x)$ will yield the points $g(x)$ and $g(x+dx)$. We can use these two values to construct the blue triangle. Its height corresponds to the change in the output $dg$. The green and the blue triangle both have the same width ($dx$), therefore only their height varies. This allows us to make a parallel to our previous example. When the green triangle and the blue triangle point in the same direction (their changes in output are either both positive or both negative), their individual heights will add into the height of the red triangle, whose change in output will be of the same sign. When the heights of the triangles are pointing in the opposite directions (one change in output is positive, and the other one is negative or vice versa) their heights will cancel each other. We will obtain the height of the red triangle, by subtracting the height of one of the input triangles from another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/derivatives-32.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sum Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The behaviour of derivatives in terms of addition and subtraction seems to be analogous to adding and subtracting functions, thus adding and subtracting numbers. We can thus conclude that the derivative of the sum of two functions is equal to the sum of their derivatives. This is known as the sum rule in calculus. We can express this mathematically as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} (f(x)+g(x))=\\frac {df}{dx} + \\frac {dg}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do an example. We define the function $f$ as $f(x)=x^2$, and the function $g$ as $g(x)=3x$. We wish to compute $\\frac {d}{dx} (x^2+3x)$. The sum rule tells us that we need to add derivatives of both of these functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} (x^2+3x)&=\\frac {d}{dx}x^2 + \\frac {d}{dx}3x\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the $\\frac{d}{dx}$ acts as an distributive oprator on this sum, as if it was a simple variable. Now, we already know how to compute the derivatives of the functions $x^2$ and $3x$ so we can conclude:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} (x^2+3x)&=2x + 3\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differentiating products (and divisions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand what the derivative of a product is about, it is much easier if we represent the multiplication as an area. This time, we are multiplying two different functions $f(x)=sin(x)$ whose graph is drawn in green, and $g(x)=x^2$ whose graph is drawn in red. Their product is another function that we will name $h(x)$. Both of these functions depend on the same variable $x$ that can take different values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function $h(x)=f(x)\\cdot g(x)=x^2\\cdot sin(x)$ is represented as the blue rectangle on the left. As the $x$ oscillates between $0$ and $\\pi$, lengths of the sides of the rectangle change. This is because their lenght is the same as the output value of their respective functions for the input value $x$. In other words, the red dotted line equals the blue rectangle's height, and the green dotted line equals the blue rectangle's width. Since $sin(x)$ is a periodic function whose output oscillates between $-1$ and $1$, so will the width of the blue rectangle. Its height, on the other side will increase indefinitely with the increase of $x$. The question that we introduce with derivatives is how does the output $dh$ change as we increase the input value $x$ by a small amount $dx$. Since the graphs of $f(x)$ and $g(x)$ are two curves of different shapes, adding the amount $dx$ will produce the change in area that is unique to the functions we are multiplying."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/derivatives-33.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's further investigate the change of the area of the function $h(x)=x^2sin(x)$ due to the introducion of  $dx$. First let's consider the change to the function $g(x)=x^2$ We can think of the function $x^2$ as the area of the square, like we did before. By increasing the $dx$, the change we obtained the change in area equal to $dg$ or $d(x^2)$. This change in area now becomes the amount by which we increase the height of our rectangle! Let's now consider the function $f(x)=sin(x)$. By increasing the value of $x$ for $dx$ the output of the function might decrease or increase, depending where on the function we are! If the slope of the function is positive at that point, the output will increase, and if it is negative, the output will decrease. We can denote the change of the output of the function $f(x)=sin(x)$ as $df$ or $d(sin(x)$.This now becomes the amount by which we increase (or decrease) the width of our rectangle. As we now know how much we increase (or decrease) the rectangle's width and height, we can calculate the new areas marked in purple:\n",
    "- the area of the top rectangle is:  $sin(x)\\cdot d(x^2)$, which is equal to $f(x) \\cdot dg$\n",
    "- the area of the side rectangle is: $x^2 \\cdot d(sin(x))$, which is equal to $g(x) \\cdot df$\n",
    "- the area of the top-right rectangle is: $d(x^2) \\cdot d(sin(x))$, which is equal to $dg \\cdot df$\n",
    "\n",
    "Since the top-right rectangle is the only element in which the infinitesimal is raised to a power, we can ignore it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/derivatives-34.png\" alt=\"drawing\" width=\"450\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Product Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we can conclude is that the derivative of the product of two functions is equal to sum of the first function multiplying the derivative of the second, and the second function multiplying the derivative of the first. This is known as the __product rule__ in calculus and can be expressed algebraically as: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} (f(x)\\cdot g(x))=f(x)\\frac {dg}{dx} +g(x)\\frac {df}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do an example. We use the same as before function $f$ as $f(x)=sin(x)$, and the function $g$ as $g(x)=x^2$. We wish to compute $\\frac {d}{dx} (sin(x) \\cdot x^2)$. The product rule tells us that we need to add the product of the first function with the derivative of the second to the product of the second function and the derivative of the first:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align*} \\ \n",
    "\\frac {d}{dx} (sin(x) \\cdot x^2)&=sin(x) \\cdot \\frac{d}{dx} x^2 + x^2 \\cdot \\frac {d}{dx} sin(x) \\\\\\\\\n",
    "\\frac {d}{dx} (sin(x) \\cdot x^2)&=2xsin(x)+ x^2cos(x)\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differentiating functional composition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final remaining operation to cover is functional composition. Simply put, we are interested in the result of an interplay of functions where the output of one function becomes the input of another. The functional composition is very often used in computer programming. Let's say that we have one function `func_one` that doubles its input $t$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def func_one(x):\n",
    "    return 2*x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we set the input to be $3$, we should get the result equal to $6$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "func_one(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define another function `func_two` that squares its input $x$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def func_two(x):\n",
    "    return x*x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we set the input to $5$, the result should be $25$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "func_two(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With functional composition we can start with a single input $x$, then apply a first function, and to its output apply the second function, all in one go. If we start with $x=5$, apply the first function and then second, we should get  $5$ multiplied by $2$ ($10$) and then squared, ending with 100. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "func_two(func_one(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we first apply the second function, and then first, we first square the $5$ ($25$) and then double it, ending with $50$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "func_one(func_two(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine how this works visually. We start with two independent functions, $f$ and $g$ that are not yet composed:\n",
    "- $f(x)=x^2$\n",
    "- $g(x)=sin(x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we change the value of $x$, the value of $f(x)$ and $g(x)$ vary with it tracing their own well known graphs. The question is, what happens when we compose $f$ and $g$ such that the output of $f$ becomes the input of $g$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align*} \\ \n",
    "h(x)=g(f(x))\n",
    "\\end{align*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The animation below illustrates this. First, let's concentrate on the top right graph below. As $x$ increases, $f(x)$ increases such that for each horizontal coordinate $x$ we get a vertical coordinate at $x^2$. This is also reflected in the graph on the left. You can see that as $x$ grows steadily, $x^2$ accelerates! Now let's examine what happens with the bottom right graph. Its input on the horizontal axis is no longer $x$ but $x^2$. Due to chaining $g$ on top of $f$, the vertical coordinate of the top-right drawing becomes the horizontal coordinate on the bottom right drawing! As the point $x$ moves from $0$ to $3$, the coordinate $x^2$ will move from $0$ to $9$. The vertical coordinate on the bottom-right graph oscillates from $-1$ to $1$ according to the function $sin(x)$, but the speed of oscillation is controlled by the function $x^2$. This can be observed on the left graph. When $x$ is small, the blue dot marked with $sin(x^2)$ oscillates slowly, and the larger the $x$ becomes, the faster the $sin(x^2)$ oscillates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/derivatives-35.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like in the previous examples, the main question is what happens to the output to input ratio as we change the input for a very small amount. Here, the output is a composition of functions $g(f(x))$, the input is $x$, and the change in input is $dx$. Let's concentrate on the top-right graph first. We have added $dx$ to $x$, and we are sliding the value of $x$ from zero to some positive value. This produces two new values in the vertical axis indicating a diagonal of a triangle. $dx$ is a change of input indicated with a purple horizontal line segment, the the $df$ or $d(x^2$) is the corresponding change of the function $f(x)=x^2$ indicated with a purple vertical line segment. If we let the ratio $\\frac {df}{dx}$ approach some value while $dx$ approaches zero, we will end up with the derivative of the function $f$ in terms of the input $x$. However, we have chained another function on top of the output of the function $f$. Since its output becomes the input for the function $g$, the vertical coordinate of $x$ on the top-right graph becomes the horizontal coordinate on the bottom-right graph. Moreover, the change in output of $f$, $df$ or $d(x^2)$ becomes the new change in input on the graph below! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/derivatives-36.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chain Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative asks how does the output of some function change in terms of its input, if we increase the input for a very small amount. In the case of function composition, the output is actually the output of the whole chain, corresponding to the purple horizontal line at the bottom whose length is $d(sin(x^2)$. The small change in input that caused is still $dx$. This can be written as following: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} g(h(x))=\\frac {dg}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our particular example, this equals:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=\\frac {d(sin(x^2)}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, this is not the end. Chain rule has another important aspect to it which becomes essential in machine learning. In fact, we wouldn't be able to make neural networks learn without this particular aspect. We have already introduced the word chaining, but we used it to express the composition, or how the functions are chained together. However, there is also an aspect of chanining when computing the derivatives of composed functions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In praxis, it can happen that the derivative $\\frac{dg}{dx}$ is to complex to compute. What chain rule allows us to do is to decompose the computation in multiple simpler to compute stages. Instead of computing the derivative $\\frac{dg}{dx}$ directly, we can compute the derivative of the output $dg$ in terms of the intermediate step $dh$. Then we can compute the derivative of this intermediate step in terms of the original input $dx$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} g(h(x))=\\frac {dg}{dx}=\\frac{dg}{dh} \\frac{dh}{dx}\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we know from algebra, we could cancel the $dh$'s from the equation below, and still end up with $\\frac{dg}{dx}$. But the derivatives $\\frac{dg}{dh}$ and $\\frac{dh}{dx}$ actually allow us to effectively compute the value of $\\frac{dg}{dx}$. Instead of computing one giant derivative expression, we are decomposing it into layers, computing derivatives at each stage and multiplying them together. Thus the name __chain rule__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish of with computing the derivative of our example. Chain rule allow us to represent the expression:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=\\frac {d(sin(x^2))}{dx}\\\n",
    "\\end{eqnarray}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=\\frac {d(sin(x^2))}{d(x^2)}\\ \\frac {d(x^2)}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The right side of the equation now might look a bit intimidating, but in fact it is not. Let's just for a moment replace the value $x^2$ with for example $p$ ($x^2=p$). Now we have:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=\\frac {d(sin(p))}{dp}\\ \\frac {d(x^2)}{dx}\\\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression $\\frac{d(sin(p))}{dp}$ is simply asking what is the derivative of the $sin(x)$ function. We already know this is a $cos(x)$ function. The expression $\\frac{d(x^2)}{dx}$ is simply a derivative of the $x^2$ function and it is equal to $2x$. Therefore we can conclude:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=cos(p)2x\n",
    "\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can again replace $p$ with $x^2$ and we get:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \\ \n",
    "\\frac {d}{dx} sin(x^2)=cos(x^2)2x\n",
    "\\end{eqnarray}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}