4. Spark 0. rdd-creation.ipynb

{
    "nbformat": 4, 
    "nbformat_minor": 0, 
    "cells": [
        {
            "source": "# RDD creation", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### [Introduction to Spark with Python, by Jose A. Dianes](https://github.com/jadianes/spark-py-notebooks)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this notebook we will introduce two different ways of getting data into the basic Spark data structure, the **Resilient Distributed Dataset** or **RDD**. An RDD is a distributed collection of elements. All work in Spark is expressed as either creating new RDDs, transforming existing RDDs, or calling actions on RDDs to compute a result. Spark automatically distributes the data contained in RDDs across your cluster and parallelizes the operations you perform on them.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### References", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "The reference book for these and other Spark related topics is *Learning Spark* by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "The KDD Cup 1999 competition dataset is described in detail [here](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Getting the data files  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this notebook we will use the reduced dataset (10 percent) provided for the KDD Cup 1999, containing nearly half million network interactions. The file is provided as a *Gzip* file that we will download locally.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 31, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [], 
            "cell_type": "code", 
            "source": "import urllib\nf = urllib.urlretrieve (\"http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz\", \"kddcup.data_10_percent.gz\")"
        }, 
        {
            "source": "## Creating a RDD from a file  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "The most common way of creating an RDD is to load it from a file. Notice that Spark's `textFile` can handle compressed files directly.    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [], 
            "cell_type": "code", 
            "source": "data_file = \"./kddcup.data_10_percent.gz\"\nraw_data = sc.textFile(data_file)"
        }, 
        {
            "source": "Now we have our data file loaded into the `raw_data` RDD.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Without getting into Spark *transformations* and *actions*, the most basic thing we can do to check that we got our RDD contents right is to `count()` the number of lines loaded from the file into the RDD.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 33, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "data": {
                        "text/plain": "494021"
                    }, 
                    "execution_count": 33, 
                    "output_type": "execute_result", 
                    "metadata": {}
                }
            ], 
            "cell_type": "code", 
            "source": "raw_data.count()"
        }, 
        {
            "source": "We can also check the first few entries in our data.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 34, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "data": {
                        "text/plain": "[u'0,tcp,http,SF,181,5450,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,9,9,1.00,0.00,0.11,0.00,0.00,0.00,0.00,0.00,normal.',\n u'0,tcp,http,SF,239,486,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,19,19,1.00,0.00,0.05,0.00,0.00,0.00,0.00,0.00,normal.',\n u'0,tcp,http,SF,235,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,29,29,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n u'0,tcp,http,SF,219,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,39,39,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n u'0,tcp,http,SF,217,2032,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,49,49,1.00,0.00,0.02,0.00,0.00,0.00,0.00,0.00,normal.']"
                    }, 
                    "execution_count": 34, 
                    "output_type": "execute_result", 
                    "metadata": {}
                }
            ], 
            "cell_type": "code", 
            "source": "raw_data.take(5)"
        }, 
        {
            "source": "In the following notebooks, we will use this raw data to learn about the different Spark transformations and actions.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Creating and RDD using `parallelize`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Another way of creating an RDD is to parallelize an already existing list.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [], 
            "cell_type": "code", 
            "source": "a = range(100)\n\ndata = sc.parallelize(a)"
        }, 
        {
            "source": "As we did before, we can `count()` the number of elements in the RDD.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "data": {
                        "text/plain": "100"
                    }, 
                    "execution_count": 36, 
                    "output_type": "execute_result", 
                    "metadata": {}
                }
            ], 
            "cell_type": "code", 
            "source": "data.count()"
        }, 
        {
            "source": "As before, we can access the first few elements on our RDD.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 37, 
            "metadata": {
                "collapsed": false
            }, 
            "outputs": [
                {
                    "data": {
                        "text/plain": "[0, 1, 2, 3, 4]"
                    }, 
                    "execution_count": 37, 
                    "output_type": "execute_result", 
                    "metadata": {}
                }
            ], 
            "cell_type": "code", 
            "source": "data.take(5)"
        }
    ], 
    "metadata": {}
}