Asnycio for Dummies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Introduction to Asynchronous Programming using Python's asyncio\n",
    "\n",
    "Asynchronous programming solves a VERY specific problem: if you have a program that sits there waiting for some other routine to complete, but (crucially) could be doing other things while waiting for the aforementioned routine to complete then you might want to use asnychronous programming.\n",
    "\n",
    "What might a routine look like that waits around?  Perhaps an i/o routine!?  Yes! That's where the python package name comes from: asyncio provides a way to call i/o routines and then wait for them to finish and start doing something else whilst you wait for the io operation to finish.  Another big use of asynchronous programming is for user interfaces where you are waiting for input most of the time but then need to quickly process the input and go back to waiting for more input.\n",
    "\n",
    "![A Diagram of the difference between snychronous and asynchronous programming](https://i.imgur.com/fdFfGRq.jpg)\n",
    "Figure 1: Proof why it's a good thing I'm not a front-end developer and a diagram of why asynchronous programming can be faster.\n",
    "\n",
    "You might rightfully be asking how/why this is different from using <a href=\"https://en.wikipedia.org/wiki/Thread_(computing)\">threads</a>.  Well it's similar but different.  You could do everything here by spawning multiple threads, but threads aren't free as far as resources go: you may be in an instance where you need to only use one thread.  More importantly, multi-threading is actually going to have to run like the synchronous example in Figure 1 above!  If you have to wait for something, the thread has to actively wait for the something to complete, unless you're using a asynchronous programming.\n",
    "\n",
    "Enough talking, let's get started with some code!  The syntax for asyncio got much cleaner and easier to follow/read as of version 3.6 so I need to make sure we're using version 3.6 or later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "assert sys.version_info.major >= 3\n",
    "assert sys.version_info.minor >= 6\n",
    "\n",
    "import asyncio\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key object here which allows asynchronous programming to work is called the event loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loop = asyncio.get_event_loop()\n",
    "# loop.set_debug(True) #this provides MUCH more helpful error messages; always do this when debugging/developing\n",
    "# Not set here to keep the notebook output looking cleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You place objects called coroutines into the loop.  A coroutine is an object which knows how to cede control back to the event loop and then let the event loop know when it is ready to resume control.  The simplest coroutine is one that just sleeps by cedining control to the event loop and then prints something:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "async def simple_sleeping_coroutine(label):\n",
    "    waiting_time_start = time.time()\n",
    "    await asyncio.sleep(1.0)\n",
    "    print(f\"routine {label} waited {time.time() - waiting_time_start} s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a couple of new things here for a lot of programmers.  First off, the keyword `async` tells the interpreter that we are writing a coroutine and thus that it is an object which can be put into the event loop.\n",
    "\n",
    "Then the keyword `await (expression)` is where the magic happens!  It tells the interpreter: \"OK!  I'm done until the expression is done computing, then I need to get control back eventually\".  Crucially, that expression must iteself be a coroutine and thus can talk to the interpreter.  True to the spirit of python syntax which came before it, `await` is just like reading plain language: await this result to be finished.  We now have a coroutine; let's put this guy into the event loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "routine a waited 1.0013272762298584 s\n"
     ]
    }
   ],
   "source": [
    "loop.run_until_complete(simple_sleeping_coroutine(\"a\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a rather simple example: we put only one coroutine on the event loop and we ran it until it completed.  The real power comes from placing multiple coroutines on the event loop which we can do using `asyncio.gather`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "routine b waited 1.0013608932495117 s\n",
      "routine a waited 1.0015013217926025 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loop.run_until_complete(asyncio.gather(simple_sleeping_coroutine(\"a\"), simple_sleeping_coroutine(\"b\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will note that the above routine outputed `[None, None]` and you can indeed use return statements inside of coroutines, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def less_simple_sleeping_coroutine(label):\n",
    "    waiting_time_start = time.time()\n",
    "    await asyncio.sleep(1.0)\n",
    "    sleeping_time_s = time.time() - waiting_time_start\n",
    "    print(f\"routine {label} waited {sleeping_time_s} s\")\n",
    "    return sleeping_time_s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use a generator to make multiple coroutines: it's just an object and doesn't get called until the event loop does, so no reason you can't."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "routine 4 waited 1.0014550685882568 s\n",
      "routine 2 waited 1.0015904903411865 s\n",
      "routine 0 waited 1.002209186553955 s\n",
      "routine 1 waited 1.0022671222686768 s\n",
      "routine 3 waited 1.0026037693023682 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.002209186553955,\n",
       " 1.0022671222686768,\n",
       " 1.0015904903411865,\n",
       " 1.0026037693023682,\n",
       " 1.0014550685882568]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loop.run_until_complete(asyncio.gather(*[less_simple_sleeping_coroutine(i) for i in range(5)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also worth noting that because only one function is ever running at the same time, you can use a simple collection variable like a list instead of the necessary thread safe containers if you were using a threading or multiprocessing solution to your problem.  Let me reiterate since this is really, really cool and does a lot to make asynchronous programming so attractive as a paradigm: **you can use any data structure you want when programming asynchronously**.  That may be hard to follow so here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "routine 0 waited 1.0014450550079346 s\n",
      "routine 1 waited 1.0015833377838135 s\n",
      "routine 2 waited 1.002211570739746 s\n",
      "routine 3 waited 1.0024409294128418 s\n",
      "routine 4 waited 1.0026600360870361 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.0014450550079346,\n",
       " 1.0015833377838135,\n",
       " 1.002211570739746,\n",
       " 1.0024409294128418,\n",
       " 1.0026600360870361]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collection_list = []\n",
    "async def collection_example_sleeping_coroutine(label):\n",
    "    waiting_time_start = time.time()\n",
    "    await asyncio.sleep(1.0)\n",
    "    sleeping_time_s = time.time() - waiting_time_start\n",
    "    print(f\"routine {label} waited {sleeping_time_s} s\")\n",
    "    collection_list.append(sleeping_time_s)\n",
    "loop.run_until_complete(asyncio.gather(*[collection_example_sleeping_coroutine(i) for i in range(5)]))\n",
    "collection_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be wondering why the routine's output when they did.  The short answer is that basically the event loop does what it will do and it tries its best to schedule efficiently but you can't be guaranteed--unless you program it to do so--that one routine will come before another.\n",
    "\n",
    "## The End of Contrived Examples: a webcrawler\n",
    "OK well that's all been well and cool but still completely contrived examples, since all they did was sleep; no i/o was done... until now!  So let's examine something that this would be very useful for: web-crawling.  Say you have a list of websites you want to grab data from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_test_list = [\n",
    "    'http://ziprecruiter.com',\n",
    "    'http://google.com',\n",
    "    'http://reddit.com',\n",
    "    'http://news.ycombinator.com/',\n",
    "    'http://httpbin.org/',\n",
    "    'http://scouting.org',\n",
    "    'https://en.wikipedia.org/wiki/Main_Page',\n",
    "    'https://www.amazon.com/',\n",
    "    'http://github.com',\n",
    "    'http://fakey.fakefake'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These websites will all take a different amount of time to load based on a bunch of factors: how much data they're serving, what their code that delivers that data is and many other things.  That alone isn't necessarily enough to justify using asynchronous programming, but say we want to do something with the data once we've fetched it: perhaps count all instances of `<` and `>` in the html of the returned website.  That is indeed something we can do while waiting for another website to load so this is a PERFECT instance of how asyncio should be used!  \n",
    "\n",
    "So with that in mind, let's write our fetcher coroutine.  We're going to need a connection library, however, that is a coroutine.  Asyncio has network connection primitives but in the interest of conciseness, I'm going to use [aiohttp](http://aiohttp.readthedocs.io/en/stable/) which provides everything we need for this example in a simple package.  You can also easily write servers with that package but we're just focusing on a client for now.  OK!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot connect to http://fakey.fakefake\n",
      "fetching http://fakey.fakefake took 0.01189732551574707 s\n",
      "fetching https://en.wikipedia.org/wiki/Main_Page took 0.044196367263793945 s\n",
      "fetching http://news.ycombinator.com/ took 0.1682143211364746 s\n",
      "fetching http://httpbin.org/ took 0.16768479347229004 s\n",
      "fetching http://scouting.org took 0.1816091537475586 s\n",
      "fetching http://github.com took 0.28900766372680664 s\n",
      "fetching http://ziprecruiter.com took 0.35688257217407227 s\n",
      "fetching http://google.com took 0.40779542922973633 s\n",
      "fetching https://www.amazon.com/ took 0.6077656745910645 s\n",
      "fetching http://reddit.com took 1.211148977279663 s\n",
      "CPU times: user 240 ms, sys: 24 ms, total: 264 ms\n",
      "Wall time: 1.25 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('http://ziprecruiter.com', (1133, 1133)),\n",
       " ('http://google.com', (161, 160)),\n",
       " ('http://reddit.com', (4087, 4084)),\n",
       " ('http://news.ycombinator.com/', (1488, 1488)),\n",
       " ('http://httpbin.org/', (454, 459)),\n",
       " ('http://scouting.org', (15, 15)),\n",
       " ('https://en.wikipedia.org/wiki/Main_Page', (1906, 1906)),\n",
       " ('https://www.amazon.com/', (4133, 4078)),\n",
       " ('http://github.com', (909, 909)),\n",
       " ('http://fakey.fakefake', (0, 0))]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import aiohttp\n",
    "import async_timeout\n",
    "\n",
    "# this is basically just taken from the example on their website\n",
    "async def async_fetch_text(url):\n",
    "    try:\n",
    "        # These are coroutine context managers: pretty cool!  \n",
    "        async with aiohttp.ClientSession() as session:  #wait to get the session, cede control to the event loop\n",
    "            async with async_timeout.timeout(10):\n",
    "                async with session.get(url) as response:\n",
    "                    return await response.text()\n",
    "    except aiohttp.ClientConnectorError:\n",
    "        print(f\"cannot connect to {url}\")\n",
    "        return \"\"\n",
    "                        \n",
    "\n",
    "# this doesn't really need to be async because it needs the entire thread's attention to calculate\n",
    "def count_angle_brakets(text_input):\n",
    "    left_count = 0\n",
    "    right_count = 0\n",
    "    for char in text_input:\n",
    "        if char == '<':\n",
    "            left_count += 1\n",
    "        if char == '>':\n",
    "            right_count += 1\n",
    "    return left_count, right_count\n",
    "\n",
    "async def async_worker(url):\n",
    "    start_time = time.time()\n",
    "    url_text = await async_fetch_text(url)\n",
    "    print(f\"fetching {url} took {time.time() - start_time} s\")\n",
    "    count =  count_angle_brakets(url_text)\n",
    "    return url, count\n",
    "\n",
    "%time loop.run_until_complete(asyncio.gather(*[async_worker(url) for url in url_test_list]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to convince you that this is indeed faster than doing this synchronously, let's quickly do it the old fashioned synchronous way!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fetching http://ziprecruiter.com took 0.6322219371795654 s\n",
      "fetching http://google.com took 0.38666296005249023 s\n",
      "fetching http://reddit.com took 0.30480074882507324 s\n",
      "fetching http://news.ycombinator.com/ took 0.14882588386535645 s\n",
      "fetching http://httpbin.org/ took 0.20450830459594727 s\n",
      "fetching http://scouting.org took 0.16111159324645996 s\n",
      "fetching https://en.wikipedia.org/wiki/Main_Page took 0.046398162841796875 s\n",
      "fetching https://www.amazon.com/ took 0.11020135879516602 s\n",
      "fetching http://github.com took 0.29704856872558594 s\n",
      "Cannot connect to http://fakey.fakefake\n",
      "fetching http://fakey.fakefake took 0.003855466842651367 s\n",
      "CPU times: user 260 ms, sys: 4 ms, total: 264 ms\n",
      "Wall time: 2.33 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('http://ziprecruiter.com', (1118, 1118)),\n",
       " ('http://google.com', (161, 160)),\n",
       " ('http://reddit.com', (23, 23)),\n",
       " ('http://news.ycombinator.com/', (1488, 1488)),\n",
       " ('http://httpbin.org/', (454, 459)),\n",
       " ('http://scouting.org', (15, 15)),\n",
       " ('https://en.wikipedia.org/wiki/Main_Page', (1906, 1906)),\n",
       " ('https://www.amazon.com/', (119, 118)),\n",
       " ('http://github.com', (909, 909)),\n",
       " ('http://fakey.fakefake', (0, 0))]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "def sync_fetch_text(url):\n",
    "    try:\n",
    "        r = requests.get(url)\n",
    "        return r.text\n",
    "    except requests.ConnectionError:\n",
    "        print(f\"Cannot connect to {url}\")\n",
    "        return \"\"\n",
    "\n",
    "def sync_worker(url):\n",
    "    start_time = time.time()\n",
    "    url_text = sync_fetch_text(url)\n",
    "    print(f\"fetching {url} took {time.time() - start_time} s\")\n",
    "    count =  count_angle_brakets(url_text)\n",
    "    return url, count\n",
    "\n",
    "%time [sync_worker(url) for url in url_test_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "beautiful!  The asynchronous solution is much quicker!\n",
    "\n",
    "## Various Tips\n",
    "There are a few more things worth covering in a basic introduction:\n",
    "### Another way to start the event loop running.\n",
    "\n",
    "The event loop can start in another way.  We used `gather` and `run_until_complete`, but it also has a `run_forever` function, but to get there you have to register tasks with the loop first, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Task pending coro=<less_simple_sleeping_coroutine() running at <ipython-input-6-396f216a1440>:1>>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loop.create_task(less_simple_sleeping_coroutine(\"a\"))\n",
    "loop.create_task(less_simple_sleeping_coroutine(\"b\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we run the loop forever, we can't run anything more in the python interpreter!  So I find it useful to define a function which stops the loop after a while to avoid running forever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "routine a waited 1.0014173984527588 s\n",
      "routine b waited 1.0015285015106201 s\n",
      "CPU times: user 0 ns, sys: 4 ms, total: 4 ms\n",
      "Wall time: 3 s\n"
     ]
    }
   ],
   "source": [
    "async def time_limiter(s_to_wait):\n",
    "    await asyncio.sleep(s_to_wait)\n",
    "    loop.stop()\n",
    "loop.create_task(time_limiter(3))\n",
    "%time loop.run_forever()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pattern would also make it possible to use a coroutine with a non-terminating loop to keep sending requests off to somewhere for some reason or another: very powerful!\n",
    "\n",
    "### Firing off a New Thread\n",
    "\n",
    "If you have a routine you want to fire off in its own thread to run in its own process in parallel, you can do that using `loop.run_in_executor`.  You might do this because you simply need to run a legacy function merged with your fancy modern asynchronous code.  Or perhaps you have a heavy task you want to send off to its own process.\n",
    "\n",
    "`loop.run_in_executor` needs an Executor object to tell it how to run, a function handle and then the arguments to that function.  It is super useful anything that needs its own thread or process and for wrapping legacy code that you don't have the time to re-write; it provides a nice bridge between asynchronous programming and parallel programming.  But how do you use it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fib_20=10946 calculated in 0.5288066864013672\n",
      "Fib_21=17711 calculated in 0.5657856464385986\n",
      "Fib_22=28657 calculated in 0.5089752674102783\n",
      "Fib_25=121393 calculated in 0.6511735916137695\n",
      "Fib_23=46368 calculated in 0.5038025379180908\n",
      "Fib_24=75025 calculated in 0.7797455787658691\n",
      "Fib_26=196418 calculated in 1.229043960571289\n",
      "Fib_27=317811 calculated in 1.7452843189239502\n",
      "Fib_28=514229 calculated in 2.2597830295562744\n",
      "Fib_29=832040 calculated in 3.533963918685913\n",
      "Fib_30=1346269 calculated in 4.4870829582214355\n",
      "Fib_31=2178309 calculated in 5.071220874786377\n",
      "Fib_32=3524578 calculated in 7.060687780380249\n",
      "Fib_33=5702887 calculated in 7.627435922622681\n",
      "CPU times: user 7.9 s, sys: 4 ms, total: 7.9 s\n",
      "Wall time: 7.71 s\n",
      "[10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887]\n"
     ]
    }
   ],
   "source": [
    "#Thread execution:\n",
    "def fibonacci(n):\n",
    "    if n <= 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return fibonacci(n - 1) + fibonacci(n - 2)\n",
    "\n",
    "async def async_fib_worker_thread(i):\n",
    "    start_time = time.time()\n",
    "    result = await loop.run_in_executor(None, fibonacci, i)\n",
    "    print(f\"Fib_{i}={result} calculated in {time.time() - start_time}\")\n",
    "    return result\n",
    "%time fib_numbers = loop.run_until_complete(asyncio.gather(*[async_fib_worker_thread(i) for i in range(20,34)]))\n",
    "print(fib_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in case you weren't certain this would happen before, that the output of run_until_complete is in order, despite that fact that it did not calculate everything in order.  I wouldn't ever rely on this being the case though.\n",
    "\n",
    "Now let's spin up new processes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fib_20=10946 calculated in 0.02011728286743164\n",
      "Fib_21=17711 calculated in 0.017861366271972656\n",
      "Fib_22=28657 calculated in 0.02513408660888672\n",
      "Fib_23=46368 calculated in 0.03589582443237305\n",
      "Fib_24=75025 calculated in 0.04911947250366211\n",
      "Fib_26=196418 calculated in 0.3784801959991455\n",
      "Fib_25=121393 calculated in 0.13418269157409668\n",
      "Fib_27=317811 calculated in 0.1956641674041748\n",
      "Fib_28=514229 calculated in 0.21662235260009766\n",
      "Fib_29=832040 calculated in 0.31270551681518555\n",
      "Fib_30=1346269 calculated in 0.5519745349884033\n",
      "Fib_31=2178309 calculated in 0.7180044651031494\n",
      "Fib_32=3524578 calculated in 1.2151618003845215\n",
      "Fib_33=5702887 calculated in 1.8036613464355469\n",
      "CPU times: user 76 ms, sys: 252 ms, total: 328 ms\n",
      "Wall time: 2.08 s\n",
      "[10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887]\n"
     ]
    }
   ],
   "source": [
    "# Process Execution\n",
    "import concurrent.futures\n",
    "process_executor = concurrent.futures.ProcessPoolExecutor()\n",
    "async def async_fib_worker_proc(i):\n",
    "    start_time = time.time()\n",
    "    result = await loop.run_in_executor(process_executor, fibonacci, i)\n",
    "    print(f\"Fib_{i}={result} calculated in {time.time() - start_time}\")\n",
    "    return result\n",
    "%time fib_numbers = loop.run_until_complete(asyncio.gather(*[async_fib_worker_proc(i) for i in range(20,34)]))\n",
    "print(fib_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process-based task should have been much quicker at the expense of using more system resources, because each coroutine got its own process, so it could run in parallel to the master event loop thread, whereas the one before it had to share time with all the other processes.\n",
    "\n",
    "### A Note on the Event Loop\n",
    "\n",
    "In order to save resources, we really ought to close the event loop when we're done with it.  Don't worry!  You can get a new one back if you need it, but the old one is gone forever:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "loop.close()\n",
    "# to get it back:\n",
    "loop = asyncio.new_event_loop()\n",
    "asyncio.set_event_loop(loop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyone can develop their own event loop, though, and some people have gone and done that using the same event loop which powers node.js and call is [uvloop](https://github.com/MagicStack/uvloop) which claims to be 2-4x faster than the core python event loop for certain examples. \n",
    "\n",
    "(Before you ask, the python core (cpython) developers don't make it the default loop because it introduces extra dependencies because of how it's written, and the python developers try to keep the number of dependencies in the core python as low as possible.)\n",
    "\n",
    "Let's see if uvloop is indeed faster for our web crawling example!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot connect to http://fakey.fakefake\n",
      "fetching http://fakey.fakefake took 0.012985706329345703 s\n",
      "fetching https://en.wikipedia.org/wiki/Main_Page took 0.03833460807800293 s\n",
      "fetching http://news.ycombinator.com/ took 0.1383969783782959 s\n",
      "fetching http://httpbin.org/ took 0.15240740776062012 s\n",
      "fetching http://scouting.org took 0.18699407577514648 s\n",
      "fetching http://github.com took 0.30530810356140137 s\n",
      "fetching http://google.com took 0.3951401710510254 s\n",
      "fetching http://ziprecruiter.com took 0.4036407470703125 s\n",
      "fetching http://reddit.com took 0.49355626106262207 s\n",
      "fetching https://www.amazon.com/ took 0.59002685546875 s\n",
      "CPU times: user 200 ms, sys: 36 ms, total: 236 ms\n",
      "Wall time: 644 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('http://ziprecruiter.com', (1124, 1124)),\n",
       " ('http://google.com', (161, 160)),\n",
       " ('http://reddit.com', (4091, 4088)),\n",
       " ('http://news.ycombinator.com/', (1488, 1488)),\n",
       " ('http://httpbin.org/', (454, 459)),\n",
       " ('http://scouting.org', (15, 15)),\n",
       " ('https://en.wikipedia.org/wiki/Main_Page', (1906, 1906)),\n",
       " ('https://www.amazon.com/', (4229, 4174)),\n",
       " ('http://github.com', (909, 909)),\n",
       " ('http://fakey.fakefake', (0, 0))]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time loop.run_until_complete(asyncio.gather(*[async_worker(url) for url in url_test_list]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "loop.close() # since we're done with it!\n",
    "\n",
    "# from the uvloop docs\n",
    "import uvloop\n",
    "# Tell asyncio to use uvloop to create new event loops\n",
    "asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())\n",
    "loop = asyncio.new_event_loop()\n",
    "asyncio.set_event_loop(loop) # always register the new event loop with asyncio or it can get confused and give you odd errors when it tries to use an event loop other than the one you want it to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot connect to http://fakey.fakefake\n",
      "fetching http://fakey.fakefake took 0.006371974945068359 s\n",
      "fetching https://en.wikipedia.org/wiki/Main_Page took 0.025042295455932617 s\n",
      "fetching http://news.ycombinator.com/ took 0.14052820205688477 s\n",
      "fetching http://httpbin.org/ took 0.14242053031921387 s\n",
      "fetching http://scouting.org took 0.16412138938903809 s\n",
      "fetching http://github.com took 0.3040812015533447 s\n",
      "fetching http://ziprecruiter.com took 0.3699519634246826 s\n",
      "fetching http://google.com took 0.3802640438079834 s\n",
      "fetching http://reddit.com took 0.5155801773071289 s\n",
      "fetching https://www.amazon.com/ took 0.6419394016265869 s\n",
      "CPU times: user 196 ms, sys: 12 ms, total: 208 ms\n",
      "Wall time: 697 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('http://ziprecruiter.com', (1118, 1118)),\n",
       " ('http://google.com', (161, 160)),\n",
       " ('http://reddit.com', (4087, 4084)),\n",
       " ('http://news.ycombinator.com/', (1488, 1488)),\n",
       " ('http://httpbin.org/', (454, 459)),\n",
       " ('http://scouting.org', (15, 15)),\n",
       " ('https://en.wikipedia.org/wiki/Main_Page', (1906, 1906)),\n",
       " ('https://www.amazon.com/', (4221, 4166)),\n",
       " ('http://github.com', (909, 909)),\n",
       " ('http://fakey.fakefake', (0, 0))]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time loop.run_until_complete(asyncio.gather(*[async_worker(url) for url in url_test_list]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So indeed a bit faster!  Very cool!\n",
    "\n",
    "## Further Reading\n",
    "\n",
    "Even the core developers of python/asyncio will [admit the current documentation is pretty terrible](http://pyvideo.org/pycon-us-2017/asyncawait-and-asyncio-in-python-36-and-beyond.html).  But now if you followed everything I wrote, I hope the documentation will make a lot more sense.  What you can do with asnycio is very very rich: this guide only scratched the surface.  The video linked above is a great resouce, and here are some other good resources to learn more:\n",
    "* http://pyvideo.org/europython-2016/asyncawait-in-python-35-and-why-it-is-awesome.html\n",
    "* https://docs.python.org/3/library/asyncio.html#module-asyncio\n",
    "* https://hackernoon.com/asyncio-for-the-working-python-developer-5c468e6e2e8e\n",
    "* https://medium.com/python-pandemonium/asyncio-coroutine-patterns-beyond-await-a6121486656f\n",
    "* https://pymotw.com/3/asyncio/\n",
    "\n",
    "## Acknowledgements\n",
    "I want to thank Yury Selivanov because this document is essentially lecture notes on his [videos](http://pyvideo.org/speaker/yury-selivanov.html) that I wrote while trying to wrap my head around asynchronous programming.  Also want to thank my employer [ZipRectuiter](https://www.ziprecruiter.com/) for requiring that I understand asynchronous programming for a recent project and letting me write and post this.\n",
    "\n",
    "And thank you for reading!  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}