4. Spark 1. rdd-basics.ipynb

{
    "nbformat": 4, 
    "nbformat_minor": 0, 
    "cells": [
        {
            "source": "# RDD basics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### [Introduction to Spark with Python, by Jose A. Dianes](https://github.com/jadianes/spark-py-notebooks)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook will introduce three basic but essential Spark operations. Two of them are the *transformations* `map` and `filter`. The other is the *action* `collect`. At the same time we will introduce the concept of *persistence* in Spark.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Getting the data and creating the RDD", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "As we did in our first notebook, we will use the reduced dataset (10 percent) provided for the KDD Cup 1999, containing nearly half million network interactions. The file is provided as a Gzip file that we will download locally.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "import urllib\nf = urllib.urlretrieve (\"http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz\", \"kddcup.data_10_percent.gz\")", 
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": []
        }, 
        {
            "source": "Now we can use this file to create our RDD.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "data_file = \"./kddcup.data_10_percent.gz\"\nraw_data = sc.textFile(data_file)", 
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": []
        }, 
        {
            "source": "## The `filter` transformation", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This transformation can be applied to RDDs in order to keep just elements that satisfy a certain condition. More concretely, a function is evaluated on every element in the original RDD. The new resulting RDD will contain just those elements that make the function return `True`.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "For example, imagine we want to count how many `normal.` interactions we have in our dataset. We can filter our `raw_data` RDD as follows.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "normal_raw_data = raw_data.filter(lambda x: 'normal.' in x)", 
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": []
        }, 
        {
            "source": "Now we can count how many elements we have in the new RDD.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "from time import time\nt0 = time()\nnormal_count = normal_raw_data.count()\ntt = time() - t0\nprint \"There are {} 'normal' interactions\".format(normal_count)\nprint \"Count completed in {} seconds\".format(round(tt,3))", 
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "There are 97278 'normal' interactions\nCount completed in 5.951 seconds\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "Remember from notebook 1 that we have a total of 494021 in our 10 percent dataset. Here we can see that 97278 contain the `normal.` tag word.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Notice that we have measured the elapsed time for counting the elements in the RDD. We have done this because we wanted to point out that actual (distributed) computations in Spark take place when we execute *actions* and not *transformations*. In this case `count` is the action we execute on the RDD. We can apply as many transformations as we want on a our RDD and no computation will take place until we call the first action that, in this case takes a few seconds to complete.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## The `map` transformation", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "By using the `map` transformation in Spark, we can apply a function to every element in our RDD. Python's lambdas are specially expressive for this particular.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this case we want to read our data file as a CSV formatted one. We can do this by applying a lambda function to each element in the RDD as follows.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "from pprint import pprint\ncsv_data = raw_data.map(lambda x: x.split(\",\"))\nt0 = time()\nhead_rows = csv_data.take(5)\ntt = time() - t0\nprint \"Parse completed in {} seconds\".format(round(tt,3))\npprint(head_rows[0])", 
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "Parse completed in 1.715 seconds\n[u'0',\n u'tcp',\n u'http',\n u'SF',\n u'181',\n u'5450',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'1',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'0',\n u'8',\n u'8',\n u'0.00',\n u'0.00',\n u'0.00',\n u'0.00',\n u'1.00',\n u'0.00',\n u'0.00',\n u'9',\n u'9',\n u'1.00',\n u'0.00',\n u'0.11',\n u'0.00',\n u'0.00',\n u'0.00',\n u'0.00',\n u'0.00',\n u'normal.']\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "Again, all action happens once we call the first Spark *action* (i.e. *take* in this case). What if we take a lot of elements instead of just the first few?  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "t0 = time()\nhead_rows = csv_data.take(100000)\ntt = time() - t0\nprint \"Parse completed in {} seconds\".format(round(tt,3))", 
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "Parse completed in 8.629 seconds\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "We can see that it takes longer. The `map` function is applied now in a  distributed way to a lot of elements on the RDD, hence the longer execution time.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Using `map` and predefined functions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Of course we can use predefined functions with `map`. Imagine we want to have each element in the RDD as a key-value pair where the key is the tag (e.g. *normal*) and the value is the whole list of elements that represents the row in the CSV formatted file. We could proceed as follows.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "def parse_interaction(line):\n    elems = line.split(\",\")\n    tag = elems[41]\n    return (tag, elems)\n\nkey_csv_data = raw_data.map(parse_interaction)\nhead_rows = key_csv_data.take(5)\npprint(head_rows[0])", 
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "(u'normal.',\n [u'0',\n  u'tcp',\n  u'http',\n  u'SF',\n  u'181',\n  u'5450',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'1',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'0',\n  u'8',\n  u'8',\n  u'0.00',\n  u'0.00',\n  u'0.00',\n  u'0.00',\n  u'1.00',\n  u'0.00',\n  u'0.00',\n  u'9',\n  u'9',\n  u'1.00',\n  u'0.00',\n  u'0.11',\n  u'0.00',\n  u'0.00',\n  u'0.00',\n  u'0.00',\n  u'0.00',\n  u'normal.'])\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "That was easy, wasn't it?", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In our notebook about working with key-value pairs we will use this type of RDDs to do data aggregations (e.g. count by key).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## The `collect` action", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "So far we have used the actions `count` and `take`. Another basic action we need to learn is `collect`. Basically it will get all the elements in the RDD into memory for us to work with them. For this reason it has to be used with care, specially when working with large RDDs.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "An example using our raw data.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "t0 = time()\nall_raw_data = raw_data.collect()\ntt = time() - t0\nprint \"Data collected in {} seconds\".format(round(tt,3))", 
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "Data collected in 17.927 seconds\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "That took longer as any other action we used before, of course. Every Spark worker node that has a fragment of the RDD has to be coordinated in order to retrieve its part, and then *reduce* everything together.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "As a last example combining all the previous, we want to collect all the `normal` interactions as key-value pairs.   ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "# get data from file\ndata_file = \"./kddcup.data_10_percent.gz\"\nraw_data = sc.textFile(data_file)\n\n# parse into key-value pairs\nkey_csv_data = raw_data.map(parse_interaction)\n\n# filter normal key interactions\nnormal_key_interactions = key_csv_data.filter(lambda x: x[0] == \"normal.\")\n\n# collect all\nt0 = time()\nall_normal = normal_key_interactions.collect()\ntt = time() - t0\nnormal_count = len(all_normal)\nprint \"Data collected in {} seconds\".format(round(tt,3))\nprint \"There are {} 'normal' interactions\".format(normal_count)", 
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "name": "stdout", 
                    "text": "Data collected in 12.485 seconds\nThere are 97278 normal interactions\n", 
                    "output_type": "stream"
                }
            ]
        }, 
        {
            "source": "This count matches with the previous count for `normal` interactions. The new procedure is more time consuming. This is because we retrieve all the data with `collect` and then use Python's `len` on the resulting list. Before we were just counting the total number of elements in the RDD by using `count`.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {}
}