Chapter 2 - Optics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "# Chapter 2 - Optics\n",
    "\n",
    "In this article, we are going to find out how an image is formed on an image plane. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "The overall principle is quite basic: An object reflects light that falls on an Image sensor, which captures the lights intensity and\n",
    "therefore, forms an image. To ensure that every part of the scenery only falls onto the optical sensor at one spot only, we can introduce\n",
    "an optical barrier with a hole in it which ensures that - for each point in our scene - only light rays with a particular \n",
    "angle fall onto the image plane.\n",
    "We can therefore create an upside-down copy of the scenery on our optical sensor. The smaller the barriers hole, the more angle-selective\n",
    "our camera becomes, the sharper the image appears. The hole is also known as aperture, or pinhole. \n",
    "\n",
    "![Pinhole Model, p. 7](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/1_pinhole_camera.png)\n",
    "*Figure 1: Concept of a Pinhole camera. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "An ideal pinhole camera has - mathematically - an infinitely small pinhole to have an infinitely sharp image. In practice, this comes in\n",
    "hand with two problems: The smaller the pinhole, the less light we can capture on our sensor. Further, a pinhole smaller than 0.3mm will\n",
    "cause light wave interferences, making the image blurry again due to diffraction effects. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "To combat these issues, lenses are used. They have the property that they bundle light rays coming from the same point in our scenery\n",
    "into a (preferably) single spot on the optical sensor. Lenses must fulfill two characteristics to be fitting camera lenses:\n",
    "- 1. Light rays that travel through the optical center of the lens will not be deviated\n",
    "- 2. Light rays that fall parallel to the optical axis into the lens are focused in a so called \"focal point\" *f* behind the lens. \n",
    "\n",
    "![Lens, p. 16](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/lense.png)\n",
    "*Figure 2: Lens. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "With combining these two properties, we can derive the **thin lens equation**: From a single object in our scene at distance *z</t>* and height *A</t>*\n",
    "in front of our lens, we can construct two light rays: One passing through the optical center, the other entering the lens parallel\n",
    "to the optical axis. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "To make the object appear sharp, we have to ensure that both light rays fall onto the same point on our optical sensor. Since the\n",
    "focal point is given by the lens properties, we cannot vary the focal length *f</t>* (distance between focal point and lens). What we \n",
    "can change is the position of the optical sensor: We can either bring it closer or farther to lens. We call the distance to the\n",
    "optical sensor *e*. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "With looking back at similarity triangles principles, we can see that The objects elevation *A</t>* in respect to the position of the sensor crossing point *B</t>* must be the same as the distance to the object *z</t>* in respect to the distance from the lens to the image sensor *e</t>*. We therefore conduct that *B/A = e/z</t>*. \n",
    "As a second equation, we can conduct that *B/A</t>* must also be equal to the ratio *e-f/f</t>*. To simplify, we can also write\n",
    "*B/A = e/f - 1*.\n",
    "\n",
    "![Thin Lens equation, p. 17](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/2_thin_lens_equation.png)\n",
    "*Figure 2: Thin lens equation model visualization. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "With combining these two equations, we get that *e/f - 1 = e/z</t>* and finally **1/f = 1/z + 1/e**. Therefore, for a given\n",
    "distance *z</t>*, the object only appears sharp if the optical sensor is distance *e</t>* away from the lens.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "It becomes clear that if we move an object further away (increasing *z</t>*), the distance *e</t>* must be changed to make the object appear\n",
    "sharp again. \n",
    "When the thin lens equation is not satisfied, the light rays do not intersect at the optical sensor, creating a blur circle which\n",
    "is perceived as \"unsharp\". Only a blur circle with radius less than 1 Pixel gives a sharp image. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "The distance between the focal plane (where the light rays at this distance actually meet) and the image plane (optical sensor)\n",
    "is called δ, the diameter of a pinhole is referred to as *L*. For simple pinhole cameras, this gives us a blur circle \n",
    "radius of:\n",
    "\n",
    "\\begin{align*}\n",
    "\\dfrac{L/2}{R}=\\dfrac{e}{\\delta} \\implies R = \\dfrac{L \\cdot \\delta}{2 \\cdot e} \n",
    "\\end{align*}\n",
    "\n",
    "![Blur circle, p. 19](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/3_blur_circle.png)\n",
    "*Figure 3: Blur circle. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "Why is this relevant? Well, for large distances to our object *z* we can approximate our lens-based camera model with a pinhole camera,\n",
    "since the distance to any world object is much much larger than the focal length or the lens size. Typically, smartphone cameras\n",
    "have focal lengths of 1.7mm and lens sizes of < 1 cm. We can therefore focus at objects at infinity. This implies that the \n",
    "focal plane for objects that are infinitely far away moves to the focal point. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "We can therefore safely approximate that the focal length *f</t>* is equal to the optical sensor distance *e : e ≈ f</t>*. This makes the \n",
    "relation between our objects elevation *A</t>* and the point on the image plane *B</t>* even simpler: We don't have to consider two\n",
    "lightrays but just one falling straight throught the pinhole. This also leaves us with a simpler equation to\n",
    "find the point *B</t>* where a object at distce *z</t>* and elevation *A</t>* would fall on the image sensor: *B/A = f/z</t>*, or *B = f/z ∙ A</t>*. \n",
    "Therefore, objects twice as far away appear half as large in on the optical sensor. \n",
    "\n",
    "![Pinhole approximation, p. 22](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/4_pinhole_approximation.png)\n",
    "*Figure 4: Pinhole approximation. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "The distance range in which an object appears sharp (R < 1 pixel) is called the Depth of Field (DoF). The smaller the apperture, \n",
    "the larger the Depth of Field, but the less light we have left for the image sensor. \n",
    "Further, the lens size at a fixed focal length defines the biggest angle the camera can perceive, the Field of View (FoV). \n",
    "We can also increase/decrease the FoV by changing the focal length: Larger focal lengths intuitively result in a more narrow\n",
    "viewing angle. The ratio between the focal length *f</t>*, lenswidth *W</t>* and the FoV angle *θ</t>* can be simply expressed via a tangential relation:\n",
    "*tan(θ/2) = W/(2∙f)</t>*\n",
    "\n",
    "![FoV-Focal length ratio, p. 22](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/5_ration_fow_focal_length.png)\n",
    "*Figure 5: Ratio between Field of View and Focal length. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "An interesting consequence of perspective projection is that parallel lines in the 3D world are no longer perceived as parallel on the 2D image\n",
    "plane. Neither are angles preserved. Further, it seems that parallel lines in an image will cross at some point, the so called\n",
    "vanishing point. With tracking all these vanishing points, we can fit a vanishing line through them: A line on which all vanishing\n",
    "points land. We observe two vanishing lines: one for horizontal and one for vertical parallel lines. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "![Vanishing Lines and Points](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/6_vanishing_points_and_lines.png)\n",
    "*Figure 6: Vanishing Lines and Points. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*\n",
    "\n",
    "We can mathematically proof that parallel vanishing lines will intersect at a vanishing point (x<sub>VP</sub>, y<sub>VP</sub>) in the camera frame. Note that this vanishing point must NOT be on the visible part of the image frame. \n",
    "Let's consider the perspective equations in camera metric coordinates. We know that x = f * (X/Z) and y = f * (Y/Z). This is given by the previously seen similarity triangle principles. \n",
    "\n",
    "We can now define two parallel lines in parametric equations. A line is described via a point, a direction and a direction-scalar that can be changed to reach any point in the system. To make two parallel lines, we choose different base points but the same directions. \n",
    "\n",
    "![Two parallel lines](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/parallel_lines.png)\n",
    "*Figure 7: Two parallel lines. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*\n",
    "\n",
    "To proof that at infinity, the lines will cross at a specific point (x<sub>VP</sub>, y<sub>VP</sub>), we look at both x and y dimensions seperately and take the limes from s to infinity. \n",
    "First of all, the X/Z part becomes irrelevantly small, so it can be neglected. We are left with the term f * sl/sn, from which we can cancel out the *s*. We finally get a point x<sub>VP</sub> that is only dependent on the direction of the line. The same holds true for y<sub>VP</sub>. We have  found the image coordinates of the vanishing point. \n",
    "\n",
    "![Taking the limes of the vanishing line](https://github.com/joelbarmettlerUZH/PyVisualOdometry/raw/master/img/chapter_2/limes_of_vanishing_lines.png)\n",
    "*Figure 8: Taking the limes of the vanishing line. [source](http://rpg.ifi.uzh.ch/docs/teaching/2019/01_introduction.pdf)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "stem_cell": {
   "cell_type": "raw",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}