002_Python_Generators.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<small><small><i>\n",
    "All the IPython Notebooks in **Python Advanced Topics** lecture series by Dr. Milaan Parmar are available @ **[GitHub](https://github.com/milaan9/07_Python_Advanced_Topics)**\n",
    "</i></small></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Python Generators\n",
    "\n",
    "In this class, you'll learn how to create iterations easily using Python generators, how it is different from iterators and normal functions, and why you should use it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators in Python\n",
    "\n",
    "There is a lot of work in building an **[iterator](https://github.com/milaan9/07_Python_Advanced_Topics/blob/main/001_Python_Iterators.ipynb)** in Python. We have to implement a class with **`__iter__()`** and **`__next__()`** method, keep track of internal states, and raise **`StopIteration`** when there are no values to be returned.\n",
    "\n",
    "This is both lengthy and counterintuitive. Generator comes to the rescue in such situations.\n",
    "\n",
    "Python generators are a simple way of creating iterators. All the work we mentioned above are automatically handled by generators in Python.\n",
    "\n",
    "Simply speaking, a generator is a function that returns an object (iterator) which we can iterate over (one value at a time)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Generators in Python\n",
    "\n",
    "It is fairly simple to create a generator in Python. It is as easy as defining a normal function, but with a **`yield`** statement instead of a **`return`** statement.\n",
    "\n",
    "If a function contains at least one **`yield`** statement (it may contain other **`yield`** or **`return`** statements), it becomes a generator function. Both **`yield`** and **`return`** will return some value from a function.\n",
    "\n",
    "The difference is that while a **`return`** statement terminates a function entirely, **`yield`** statement pauses the function saving all its states and later continues from there on successive calls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differences between Generator function and Normal function\n",
    "\n",
    "Here is how a generator function differs from a normal **[function](https://github.com/milaan9/04_Python_Functions/blob/main/001_Python_Functions.ipynb)**.\n",
    "\n",
    "1. Generator function contains one or more **`yield`** statements.\n",
    "2. When called, it returns an object (iterator) but does not start execution immediately.\n",
    "3. Methods like **`__iter__()`** and **`__next__()`** are implemented automatically. So we can iterate through the items using **`next()`**.\n",
    "4. Once the function yields, the function is paused and the control is transferred to the caller.\n",
    "5. Local variables and their states are remembered between successive calls.\n",
    "6. Finally, when the function terminates, **`StopIteration`** is raised automatically on further calls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example to illustrate all of the points stated above. We have a generator function named **`my_gen()`** with several **`yield`** statements.\n",
    "\n",
    "```python\n",
    ">>> # A simple generator function\n",
    ">>> def my_gen():\n",
    ">>>     n = 1\n",
    ">>>     print('This is printed first')\n",
    ">>> # Generator function contains yield statements\n",
    ">>>     yield n\n",
    "\n",
    ">>>     n += 1\n",
    ">>>     print('This is printed second')\n",
    ">>>     yield n\n",
    "\n",
    ">>>     n += 1\n",
    ">>>     print('This is printed at last')\n",
    ">>>     yield n\n",
    "```\n",
    "\n",
    "An interactive run in the interpreter is given below. Run these in the Python shell to see the output.\n",
    "\n",
    "```python\n",
    ">>> # It returns an object but does not start execution immediately.\n",
    ">>> a = my_gen()\n",
    "\n",
    ">>> # We can iterate through the items using next().\n",
    ">>> next(a)\n",
    "This is printed first\n",
    "1\n",
    ">>> # Once the function yields, the function is paused and the control is transferred to the caller.\n",
    "\n",
    ">>> # Local variables and theirs states are remembered between successive calls.\n",
    ">>> next(a)\n",
    "This is printed second\n",
    "2\n",
    "\n",
    ">>> next(a)\n",
    "This is printed at last\n",
    "3\n",
    "\n",
    ">>> # Finally, when the function terminates, StopIteration is raised automatically on further calls.\n",
    ">>> next(a)\n",
    "Traceback (most recent call last):\n",
    "...\n",
    "StopIteration\n",
    ">>> next(a)\n",
    "Traceback (most recent call last):\n",
    "...\n",
    "StopIteration\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One interesting thing to note in the above example is that the value of variable **`n`** is remembered between each call.\n",
    "\n",
    "Unlike normal functions, the local variables are not destroyed when the function yields. Furthermore, the generator object can be iterated only once.\n",
    "\n",
    "To restart the process we need to create another generator object using something like **`a = my_gen()`**.\n",
    "\n",
    "One final thing to note is that we can use generators with **[for loop](https://github.com/milaan9/03_Python_Flow_Control/blob/main/005_Python_for_Loop.ipynb)** directly.\n",
    "\n",
    "This is because a **`for`** loop takes an iterator and iterates over it using **`next()`** function. It automatically ends when **`StopIteration`** is raised. Check here to **[know how a for loop is actually implemented in Python](https://github.com/milaan9/07_Python_Advanced_Topics/blob/main/001_Python_Iterators.ipynb)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:08:41.287154Z",
     "start_time": "2021-06-22T06:08:41.275439Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is printed first\n",
      "1\n",
      "This is printed second\n",
      "2\n",
      "This is printed at last\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# A simple generator function\n",
    "\n",
    "def my_gen():\n",
    "    n = 1\n",
    "    print('This is printed first')\n",
    "    # Generator function contains yield statements\n",
    "    yield n\n",
    "\n",
    "    n += 1\n",
    "    print('This is printed second')\n",
    "    yield n\n",
    "\n",
    "    n += 1\n",
    "    print('This is printed at last')\n",
    "    yield n\n",
    "\n",
    "\n",
    "# Using for loop\n",
    "for item in my_gen():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more elegant way of automatically iterating is by using the for loop. Using this, we can iterate over any object that can return an iterator, for example list, string, file etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Generators with a Loop\n",
    "\n",
    "The above example is of less use and we studied it just to get an idea of what was happening in the background.\n",
    "\n",
    "Normally, generator functions are implemented with a loop having a suitable terminating condition.\n",
    "\n",
    "Let's take an example of a generator that reverses a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:08:44.728642Z",
     "start_time": "2021-06-22T06:08:44.710090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "o\n",
      "l\n",
      "l\n",
      "e\n",
      "h\n"
     ]
    }
   ],
   "source": [
    "def rev_str(my_str):\n",
    "    length = len(my_str)\n",
    "    for i in range(length - 1, -1, -1):\n",
    "        yield my_str[i]\n",
    "\n",
    "\n",
    "# For loop to reverse the string\n",
    "for char in rev_str(\"hello\"):\n",
    "    print(char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we have used the **`range()`** function to get the index in reverse order using the for loop.\n",
    "\n",
    ">**Note**: This generator function not only works with strings, but also with other kinds of iterables like **[list](https://github.com/milaan9/02_Python_Datatypes/blob/main/003_Python_List.ipynb)**, **[tuple](https://github.com/milaan9/02_Python_Datatypes/blob/main/004_Python_Tuple.ipynb)**, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Generator Expression\n",
    "\n",
    "Simple generators can be easily created on the fly using generator expressions. It makes building generators easy.\n",
    "\n",
    "Similar to the lambda functions which create **[anonymous functions](https://github.com/milaan9/04_Python_Functions/blob/main/006_Python_Function_Anonymous.ipynb)**, generator expressions create **anonymous generator functions**.\n",
    "\n",
    "The syntax for generator expression is similar to that of a **[list comprehension in Python](https://github.com/milaan9/02_Python_Datatypes/blob/main/003_Python_List.ipynb)**. But the square brackets are replaced with round parentheses.\n",
    "\n",
    "The major difference between a list comprehension and a generator expression is that a list comprehension produces the entire list while the generator expression produces one item at a time.\n",
    "\n",
    "They have lazy execution ( producing items only when asked for ). For this reason, a generator expression is much more memory efficient than an equivalent list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:08:51.544724Z",
     "start_time": "2021-06-22T06:08:51.532032Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 9, 36, 100]\n",
      "<generator object <genexpr> at 0x000001D284B3B430>\n"
     ]
    }
   ],
   "source": [
    "# Initialize the list\n",
    "my_list = [1, 3, 6, 10]\n",
    "\n",
    "# square each term using list comprehension\n",
    "list_ = [x**2 for x in my_list]\n",
    "\n",
    "# same thing can be done using a generator expression\n",
    "# generator expressions are surrounded by parenthesis ()\n",
    "generator = (x**2 for x in my_list)\n",
    "\n",
    "print(list_)\n",
    "print(generator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see above that the generator expression did not produce the required result immediately. Instead, it returned a generator object, which produces items only on demand.\n",
    "\n",
    "Here is how we can start getting items from the generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:08:54.397338Z",
     "start_time": "2021-06-22T06:08:54.076039Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "9\n",
      "36\n",
      "100\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-f7b055f5764d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Initialize the list\n",
    "my_list = [1, 3, 6, 10]\n",
    "\n",
    "a = (x**2 for x in my_list)\n",
    "print(next(a))\n",
    "\n",
    "print(next(a))\n",
    "\n",
    "print(next(a))\n",
    "\n",
    "print(next(a))\n",
    "\n",
    "next(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generator expressions can be used as function arguments. When used in such a way, the round parentheses can be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:09:00.644066Z",
     "start_time": "2021-06-22T06:08:56.219645Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "146"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(x**2 for x in my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:09:00.763212Z",
     "start_time": "2021-06-22T06:09:00.730988Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(x**2 for x in my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use of Python Generators\n",
    "\n",
    "There are several reasons that make generators a powerful implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Easy to Implement\n",
    "\n",
    "Generators can be implemented in a clear and concise way as compared to their iterator class counterpart. Following is an example to implement a sequence of power of 2 using an iterator class.\n",
    "\n",
    "```python\n",
    ">>> class PowTwo:\n",
    ">>>     def __init__(self, max=0):\n",
    ">>>         self.n = 0\n",
    ">>>         self.max = max\n",
    "\n",
    ">>>     def __iter__(self):\n",
    ">>>         return self\n",
    "\n",
    ">>>     def __next__(self):\n",
    ">>>         if self.n > self.max:\n",
    ">>>             raise StopIteration\n",
    "\n",
    ">>>         result = 2 ** self.n\n",
    ">>>         self.n += 1\n",
    ">>>         return result\n",
    "```\n",
    "\n",
    "The above program was lengthy and confusing. Now, let's do the same using a generator function.\n",
    "\n",
    "```python\n",
    ">>> def PowTwoGen(max=0):\n",
    ">>>     n = 0\n",
    ">>>     while n < max:\n",
    ">>>         yield 2 ** n\n",
    ">>>         n += 1\n",
    "```\n",
    "\n",
    "Since generators keep track of details automatically, the implementation was concise and much cleaner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Memory Efficient\n",
    "\n",
    "A normal function to return a sequence will create the entire sequence in memory before returning the result. This is an overkill, if the number of items in the sequence is very large.\n",
    "\n",
    "Generator implementation of such sequences is memory friendly and is preferred since it only produces one item at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Represent Infinite Stream\n",
    "\n",
    "Generators are excellent mediums to represent an infinite stream of data. Infinite streams cannot be stored in memory, and since generators produce only one item at a time, they can represent an infinite stream of data.\n",
    "\n",
    "The following generator function can generate all the even numbers (at least in theory).\n",
    "\n",
    "```python\n",
    ">>> def all_even():\n",
    ">>>     n = 0\n",
    ">>>     while True:\n",
    ">>>         yield n\n",
    ">>>         n += 2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Pipelining Generators\n",
    "\n",
    "Multiple generators can be used to pipeline a series of operations. This is best illustrated using an example.\n",
    "\n",
    "Suppose we have a generator that produces the numbers in the Fibonacci series. And we have another generator for squaring numbers.\n",
    "\n",
    "If we want to find out the sum of squares of numbers in the Fibonacci series, we can do it in the following way by pipelining the output of generator functions together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-22T06:09:09.356685Z",
     "start_time": "2021-06-22T06:09:09.348874Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4895\n"
     ]
    }
   ],
   "source": [
    "def fibonacci_numbers(nums):\n",
    "    x, y = 0, 1\n",
    "    for _ in range(nums):\n",
    "        x, y = y, x+y\n",
    "        yield x\n",
    "\n",
    "def square(nums):\n",
    "    for num in nums:\n",
    "        yield num**2\n",
    "\n",
    "print(sum(square(fibonacci_numbers(10))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pipelining is efficient and easy to read (and yes, a lot cooler!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}