4. Spark 2. rdd-sampling.ipynb

{
    "nbformat": 4, 
    "nbformat_minor": 0, 
    "cells": [
        {
            "source": "# Sampling RDDs", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "[Introduction to Spark with Python, by Jose A. Dianes](https://github.com/jadianes/spark-py-notebooks)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "So far we have introduced RDD creation together with some basic transformations such as `map` and `filter` and some actions such as `count`, `take`, and `collect`.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook will show how to sample RDDs. Regarding transformations, `sample` will be introduced since it will be useful in many statistical learning scenarios. Then we will compare results with the `takeSample` action.      ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Getting the data and creating the RDD", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this case we will use the complete dataset provided for the KDD Cup 1999, containing nearly half million network interactions. The file is provided as a Gzip file that we will download locally.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [], 
            "cell_type": "code", 
            "source": "import urllib\nf = urllib.urlretrieve (\"http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data.gz\", \"kddcup.data.gz\")"
        }, 
        {
            "source": "Now we can use this file to create our RDD.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [], 
            "cell_type": "code", 
            "source": "data_file = \"./kddcup.data.gz\"\nraw_data = sc.textFile(data_file)"
        }, 
        {
            "source": "## Sampling RDDs   ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In Spark, there are two sampling operations, the transformation `sample` and the action `takeSample`. By using a transformation we can tell Spark to apply successive transformation on a sample of a given RDD. By using an action we retrieve a given sample and we can have it in local memory to be used by any other standard library (e.g. Scikit-learn).  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### The `sample` transformation", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "The `sample` transformation takes up to three parameters. First is whether the sampling is done with replacement or not. Second is the sample size as a fraction. Finally we can optionally provide a *random seed*.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "text": "Sample size is 489957 of 4898431\n", 
                    "name": "stdout", 
                    "output_type": "stream"
                }
            ], 
            "cell_type": "code", 
            "source": "raw_data_sample = raw_data.sample(False, 0.1, 1234)\nsample_size = raw_data_sample.count()\ntotal_size = raw_data.count()\nprint \"Sample size is {} of {}\".format(sample_size, total_size)"
        }, 
        {
            "source": "But the power of sampling as a transformation comes from doing it as part of a sequence of additional transformations. This will show more powerful once we start doing aggregations and key-value pairs operations, and will be specially useful when using Spark's machine learning library MLlib.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In the meantime, imagine we want to have an approximation of the proportion of `normal.` interactions in our dataset. We could do this by counting the total number of tags as we did in previous notebooks. However we want a quicker response and we don't need the exact answer but just an approximation. We can do it as follows.   ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "text": "The ratio of 'normal' interactions is 0.199\nCount done in 44.523 seconds\n", 
                    "name": "stdout", 
                    "output_type": "stream"
                }
            ], 
            "cell_type": "code", 
            "source": "from time import time\n\n# transformations to be applied\nraw_data_sample_items = raw_data_sample.map(lambda x: x.split(\",\"))\nsample_normal_tags = raw_data_sample_items.filter(lambda x: \"normal.\" in x)\n\n# actions + time\nt0 = time()\nsample_normal_tags_count = sample_normal_tags.count()\ntt = time() - t0\n\nsample_normal_ratio = sample_normal_tags_count / float(sample_size)\nprint \"The ratio of 'normal' interactions is {}\".format(round(sample_normal_ratio,3)) \nprint \"Count done in {} seconds\".format(round(tt,3))"
        }, 
        {
            "source": "Let's compare this with calculating the ratio without sampling.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "text": "The ratio of 'normal' interactions is 0.199\nCount done in 91.09 seconds\n", 
                    "name": "stdout", 
                    "output_type": "stream"
                }
            ], 
            "cell_type": "code", 
            "source": "# transformations to be applied\nraw_data_items = raw_data.map(lambda x: x.split(\",\"))\nnormal_tags = raw_data_items.filter(lambda x: \"normal.\" in x)\n\n# actions + time\nt0 = time()\nnormal_tags_count = normal_tags.count()\ntt = time() - t0\n\nnormal_ratio = normal_tags_count / float(total_size)\nprint \"The ratio of 'normal' interactions is {}\".format(round(normal_ratio,3)) \nprint \"Count done in {} seconds\".format(round(tt,3))"
        }, 
        {
            "source": "We can see a gain in time. The more transformations we apply after the sampling the bigger this gain. This is because without sampling all the transformations are applied to the complete set of data.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### The `takeSample` action  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "If what we need is to grab a sample of raw data from our RDD into local memory in order to be used by other non-Spark libraries, `takeSample` can be used.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "The syntax is very similar, but in this case we specify the number of items instead of the sample size as a fraction of the complete data size.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "text": "The ratio of 'normal' interactions is 0.1988025\nCount done in 76.166 seconds\n", 
                    "name": "stdout", 
                    "output_type": "stream"
                }
            ], 
            "cell_type": "code", 
            "source": "t0 = time()\nraw_data_sample = raw_data.takeSample(False, 400000, 1234)\nnormal_data_sample = [x.split(\",\") for x in raw_data_sample if \"normal.\" in x]\ntt = time() - t0\n\nnormal_sample_size = len(normal_data_sample)\n\nnormal_ratio = normal_sample_size / 400000.0\nprint \"The ratio of 'normal' interactions is {}\".format(normal_ratio)\nprint \"Count done in {} seconds\".format(round(tt,3))"
        }, 
        {
            "source": "The process was very similar as before. We obtained a sample of about 10 percent of the data, and then filter and split.  \n\nHowever, it took longer, even with a slightly smaller sample. The reason is that Spark just distributed the execution of the sampling process. The filtering and splitting of the results were done locally in a single node.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {}
}