rq-godber/rq_godber.ipynb

{
 "metadata": {
  "celltoolbar": "Slideshow",
  "name": "",
  "signature": "sha256:d02baed5928efe80afc401b8bcb0b90ae945757bd3069df6048425c8990eec60"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Battle of the Queues\n",
      "\n",
      "RQ - Redis Queue (http://python-rq.org/docs/)\n",
      "\n",
      "Austin Godber (@godber)\n",
      "\n",
      "Feb 2015"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "## One Queue to Rule Them All"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Not really"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "RQ is Python specific, messages use `pickle`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Redis is a key value store, not a highly available message broker."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "RQ Workers fork new child processes for every task."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "But is quick and easy!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Redis Queue - Install\n",
      "\n",
      "* Dependencies\n",
      "  * Redis >= 2.6\n",
      "  * Python\n",
      "  * Thats it\n",
      "* Install\n",
      "  * `pip install rq`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Redis Queue - Usage"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Real Quick"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "Take any Python function\n",
      "\n",
      "```\n",
      "import requests\n",
      "\n",
      "def count_words_at_url(url):\n",
      "    resp = requests.get(url)\n",
      "    return len(resp.text.split())\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "Create a queue and enqueue the function reference and arguments\n",
      "\n",
      "```\n",
      "from redis import Redis\n",
      "from rq import Queue\n",
      "from count import count_words_at_url\n",
      "\n",
      "q = Queue(connection=Redis())\n",
      "q.enqueue(\n",
      "    count_words_at_url, 'http://uberhip.com'\n",
      ")\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "Run the worker on the command line\n",
      "\n",
      "```\n",
      "rqworker\n",
      "```\n",
      "\n",
      "Run the client\n",
      "\n",
      "```\n",
      "rq1.py\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "Worker logs show \n",
      "\n",
      "\n",
      "<img src='workerlog.png'>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "Remember, all we did was enqueue a job, specifying only the function and its arguments.\n",
      "\n",
      "```\n",
      "q = Queue(connection=Redis())\n",
      "q.enqueue(\n",
      "    count_words_at_url, 'http://uberhip.com'\n",
      ")\n",
      "```\n",
      "\n",
      "No queue names, no priorities, no handling of the return values."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Queues\n",
      "\n",
      "To enqueue something on a specific queue, instantiate Queue() with the queue name as an argument.\n",
      "\n",
      "```\n",
      "low_q = Queue('low', connection=Redis())\n",
      "low_q.enqueue(\n",
      "    count_words_at_url, 'http://uberhip.com'\n",
      ")\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "You can use any queue name."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Workers can be started on specific queues by providing queue names as command line arguments to `rqworker`.\n",
      "\n",
      "```\n",
      "rqworker low\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Other `Queue` keyword arguments include: `timeout`, `result_ttl`, `ttl`, `depends_on`, `job_id`, and `at_font`.  See http://python-rq.org/docs/ for details."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "The task function passed to `enqueue` can be EITHER a function reference OR a string.  So the Workers can be implemented in a separate code base from the `enqueueing` code."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "`Queue()` has the following methods and attributes:\n",
      "\n",
      "* `len(q)` - Number of jobs in queue\n",
      "* `q.job_ids` - List of enqueued job IDs\n",
      "* `q.jobs` - List of enqueued job instances\n",
      "* `job = q.fetch_job('my_id')` - Returns job having ID: my_id"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Workers"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Config file"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Burst Mode"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Lifecycle: `Boot` -> `Reg` -> `Listen` -> `Prep Exec` -> `Fork()` -> `Process` -> `Cleanup` -> `GOTO 3`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Custom Workers"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Read the docs, we don't have time for this!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Results\n",
      "\n",
      "Want results from your job?  `enqueue` returns a job reference, save it, ask it for results before the default `results_ttl` of 500s expires.\n",
      "\n",
      "```\n",
      "q = Queue(connection=Redis())\n",
      "job = q.enqueue(\n",
      "    count_words_at_url, 'http://uberhip.com'\n",
      ")\n",
      "time.sleep(2)\n",
      "print job.result\n",
      "```\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Jobs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "What, if we don't have time for Workers, we certainly don't have time for jobs!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Well, other than to emphasize there are two `times-to-live`\n",
      "\n",
      "* `result_ttl` - result time to live\n",
      "* `ttl` - job time to live"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "And ... there's a `meta` property to which you can attach arbitrary data using dictionary syntax."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Read the docs: http://python-rq.org/docs/jobs/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Monitoring\n",
      "\n",
      "* Command Line - `rqinfo`\n",
      "* Web - `rq-dashboard`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "DEMO!!!!!!!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Connections, Exceptions and Tests oh My!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Sure, you can connect to multiple Redis instances."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "(sounds a little fancy, maybe you want to try a different queueing system)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Exceptions end up in the `failed` queue, `rq-dashboard` is handy for reviewing this."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Tests!  Use the `SimpleWorker` otherwise you might encounter trouble with `fork()`."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Other Cool Things We Don't Have Time For\n",
      "\n",
      "* Job Dependencies: `depends_on`\n",
      "* `@job` decorator"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "# Tips\n",
      "\n",
      "* Be careful to weigh your `ttl`s and runtimes appropriately.  The default job `ttl` is 180s, if your jobs runtime is expected to exceed 3min you should be changing the `ttl`.\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}