2_better_reduce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2. Build a better reduce.\n",
    "\n",
    "In this example, you will use `dask.delayed` and `dask.visualize` to investigate the task graph for Python's built-in `reduce` function. You will then make your own `reduce` function with logic that is (hopefully) more parallelizable. Lastly, you'll compare the parallel execution times of your reduce function with the built-in reduce to see if it worked. :)\n",
    "\n",
    "The reduce function, located in the built-in `functools` package, applies a function to a list to produce a single output. The function must take two inputs, and it is successively applied to pairs in the input list until the final output is reached. For example, if you want to apply an `add` function to a list of integers, reduce will first apply `add` to the first two elements of the list. It will then apply `add` to that result and the third element in the list. It will then apply `add` to that result and the fourth element in the list. And so on.\n",
    "\n",
    "## Step 1: Define function and inputs\n",
    "\n",
    "This is the function that we will call reduce upon. Just an add function with a bit of a computational delay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add(x, y):\n",
    "    n = 1e7\n",
    "    while n>0:\n",
    "        n -= 1  # make it kind of interesting ;)\n",
    "    return x + y\n",
    "inputs = range(8)  # let's add integers up to/including 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Reduce syntax without delay\n",
    "\n",
    "To help you get started, here is how reduce is called without any delayed function or dask magic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from functools import reduce\n",
    "\n",
    "z = reduce(add, inputs)\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Delay add, apply reduce\n",
    "\n",
    "In the subsequent cell, delay the `add` function and apply reduce on it (very similar to what we did for `map`).  \n",
    "* What is the type of the output object?\n",
    "* What does the task graph look like? Does it make sense? (**Hint**: use the keyword argument `rankdir='LR'` to show your task graph from left to right instead of top to bottom.)\n",
    "* How long does this workflow take to execute? (Use default scheduler; i.e., `z.compute()` is fine.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Build a better reduce\n",
    "\n",
    "Add logic to `2_problem.py` so that your `myreduce` function can be parallizeable. Load it in using the cell below, then check the task graphs and execution times. Did it work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load 2_problem.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Check your task graphs, execution\n",
    "\n",
    "Apply your `myreduce` function to the delayed `add` function. Look at the task graph. Does it look more parallelizable? Run the computation. Is it faster?\n",
    "\n",
    "**Hint!** If your code is not faster but your task graph looks good, be sure you've fed the correct scheduler option into `compute`. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Submit your work\n",
    "\n",
    "Please follow these instructions carefully! Otherwise I won't be able to load your solution into my notebook.\n",
    "\n",
    "In short, create a branch, add/commit **only** the text file you touched and then push the branch.\n",
    "\n",
    "```\n",
    "git checkout -b <unique branch name>\n",
    "git add 2_problem.py\n",
    "git commit -m \"uploading our solution\"\n",
    "git push origin <unique branch name>\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}