notebooks/nb1-spark-sql-basics/nb1-spark-sql-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SparkSQL basics with SparkR  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Introduction to Apache Spark with R by J. A. Dianes**](https://github.com/jadianes/spark-r-notebooks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will introduce basic concepts about SparkSQL with R that you can find in the [SparkR documentation](http://spark.apache.org/docs/latest/sparkr.html), applied to the [2013 American Community Survey dataset](http://www.census.gov/programs-surveys/acs/data/summary-file.html). We will do two things, read data into a SparkSQL data frame, and have a quick look at the schema and what we have read. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a SparkSQL context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In further notebooks, we will explore our data by loading them into SparkSQL data frames. But first we need to init a SparkSQL context. The first thing we need to do is to set up some environment variables and library paths as follows. Remember to replace the value assigned to `SPARK_HOME` with your Spark home folder.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Set Spark home and R libs\n",
    "Sys.setenv(SPARK_HOME='/home/cluster/spark-1.5.0-bin-hadoop2.6')\n",
    ".libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can load the `SparkR` library as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Attaching package: ‘SparkR’\n",
      "\n",
      "The following objects are masked from ‘package:stats’:\n",
      "\n",
      "    filter, na.omit\n",
      "\n",
      "The following objects are masked from ‘package:base’:\n",
      "\n",
      "    intersect, rbind, sample, subset, summary, table, transform\n",
      "\n"
     ]
    }
   ],
   "source": [
    "library(SparkR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can initialise the Spark context as [in the official documentation](http://spark.apache.org/docs/latest/sparkr.html#starting-up-sparkcontext-sqlcontext). In our case we are use a standalone Spark cluster with one master and seven workers. If you are running Spark in local node, use just `master='local'`. Additionally, we require a Spark package from Databricks to read CSV files (more on this in the next section).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching java with spark-submit command /home/cluster/spark-1.5.0-bin-hadoop2.6/bin/spark-submit  --packages com.databricks:spark-csv_2.11:1.2.0 sparkr-shell /tmp/RtmpyC8rUQ/backend_port3743359adb78 \n"
     ]
    }
   ],
   "source": [
    "sc <- sparkR.init(master='spark://169.254.206.2:7077', sparkPackages=\"com.databricks:spark-csv_2.11:1.2.0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally we can start the SparkSQL context as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sqlContext <- sparkRSQL.init(sc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating SparkSQL data frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading CSV data using Databricks csv extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The easiest way to get our CSV data into a [SparkSQL dataframe](http://spark.apache.org/docs/latest/sparkr.html#creating-dataframes), is by using [Databricks CSV extension](https://github.com/databricks/spark-csv) to read SparkSQL dataframes directly from csv files. In any case, remember to set the right path for your data files in the first line, ours is `/nfs/data/2013-acs/ss13husa.csv`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "housing_a_file_path <- file.path('', 'nfs','data','2013-acs','ss13husa.csv')\n",
    "housing_b_file_path <- file.path('', 'nfs','data','2013-acs','ss13husb.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's read into a SparkSQL dataframe. We need to pass four parameters in addition to the `sqlContext`:  \n",
    "\n",
    "- The file path.  \n",
    "- `header='true'` since our `csv` files have a header with the column names. \n",
    "- Indicate that we want the library to infer the schema.  \n",
    "- And the source type (the Databricks package in this case).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.002   0.000  16.919 "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "system.time(\n",
    "    housing_a_df <- read.df(sqlContext, \n",
    "                        housing_a_file_path, \n",
    "                        header='true', \n",
    "                        source = \"com.databricks.spark.csv\", \n",
    "                        inferSchema='true')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the inferred schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- RT: string (nullable = true)\n",
      " |-- SERIALNO: integer (nullable = true)\n",
      " |-- DIVISION: integer (nullable = true)\n",
      " |-- PUMA: integer (nullable = true)\n",
      " |-- REGION: integer (nullable = true)\n",
      " |-- ST: integer (nullable = true)\n",
      " |-- ADJHSG: integer (nullable = true)\n",
      " |-- ADJINC: integer (nullable = true)\n",
      " |-- WGTP: integer (nullable = true)\n",
      " |-- NP: integer (nullable = true)\n",
      " |-- TYPE: integer (nullable = true)\n",
      " |-- ACCESS: integer (nullable = true)\n",
      " |-- ACR: integer (nullable = true)\n",
      " |-- AGS: integer (nullable = true)\n",
      " |-- BATH: integer (nullable = true)\n",
      " |-- BDSP: integer (nullable = true)\n",
      " |-- BLD: integer (nullable = true)\n",
      " |-- BROADBND: integer (nullable = true)\n",
      " |-- BUS: integer (nullable = true)\n",
      " |-- COMPOTHX: integer (nullable = true)\n",
      " |-- CONP: integer (nullable = true)\n",
      " |-- DIALUP: integer (nullable = true)\n",
      " |-- DSL: integer (nullable = true)\n",
      " |-- ELEP: integer (nullable = true)\n",
      " |-- FIBEROP: integer (nullable = true)\n",
      " |-- FS: integer (nullable = true)\n",
      " |-- FULP: integer (nullable = true)\n",
      " |-- GASP: integer (nullable = true)\n",
      " |-- HANDHELD: integer (nullable = true)\n",
      " |-- HFL: integer (nullable = true)\n",
      " |-- INSP: integer (nullable = true)\n",
      " |-- LAPTOP: integer (nullable = true)\n",
      " |-- MHP: integer (nullable = true)\n",
      " |-- MODEM: integer (nullable = true)\n",
      " |-- MRGI: integer (nullable = true)\n",
      " |-- MRGP: integer (nullable = true)\n",
      " |-- MRGT: integer (nullable = true)\n",
      " |-- MRGX: integer (nullable = true)\n",
      " |-- OTHSVCEX: integer (nullable = true)\n",
      " |-- REFR: integer (nullable = true)\n",
      " |-- RMSP: integer (nullable = true)\n",
      " |-- RNTM: integer (nullable = true)\n",
      " |-- RNTP: integer (nullable = true)\n",
      " |-- RWAT: integer (nullable = true)\n",
      " |-- RWATPR: integer (nullable = true)\n",
      " |-- SATELLITE: integer (nullable = true)\n",
      " |-- SINK: integer (nullable = true)\n",
      " |-- SMP: integer (nullable = true)\n",
      " |-- STOV: integer (nullable = true)\n",
      " |-- TEL: integer (nullable = true)\n",
      " |-- TEN: integer (nullable = true)\n",
      " |-- TOIL: integer (nullable = true)\n",
      " |-- VACS: integer (nullable = true)\n",
      " |-- VALP: integer (nullable = true)\n",
      " |-- VEH: integer (nullable = true)\n",
      " |-- WATP: integer (nullable = true)\n",
      " |-- YBL: integer (nullable = true)\n",
      " |-- FES: integer (nullable = true)\n",
      " |-- FFINCP: integer (nullable = true)\n",
      " |-- FGRNTP: integer (nullable = true)\n",
      " |-- FHINCP: integer (nullable = true)\n",
      " |-- FINCP: integer (nullable = true)\n",
      " |-- FPARC: integer (nullable = true)\n",
      " |-- FSMOCP: integer (nullable = true)\n",
      " |-- GRNTP: integer (nullable = true)\n",
      " |-- GRPIP: integer (nullable = true)\n",
      " |-- HHL: integer (nullable = true)\n",
      " |-- HHT: integer (nullable = true)\n",
      " |-- HINCP: integer (nullable = true)\n",
      " |-- HUGCL: integer (nullable = true)\n",
      " |-- HUPAC: integer (nullable = true)\n",
      " |-- HUPAOC: integer (nullable = true)\n",
      " |-- HUPARC: integer (nullable = true)\n",
      " |-- KIT: integer (nullable = true)\n",
      " |-- LNGI: integer (nullable = true)\n",
      " |-- MULTG: integer (nullable = true)\n",
      " |-- MV: integer (nullable = true)\n",
      " |-- NOC: integer (nullable = true)\n",
      " |-- NPF: integer (nullable = true)\n",
      " |-- NPP: integer (nullable = true)\n",
      " |-- NR: integer (nullable = true)\n",
      " |-- NRC: integer (nullable = true)\n",
      " |-- OCPIP: integer (nullable = true)\n",
      " |-- PARTNER: integer (nullable = true)\n",
      " |-- PLM: integer (nullable = true)\n",
      " |-- PSF: integer (nullable = true)\n",
      " |-- R18: integer (nullable = true)\n",
      " |-- R60: integer (nullable = true)\n",
      " |-- R65: integer (nullable = true)\n",
      " |-- RESMODE: integer (nullable = true)\n",
      " |-- SMOCP: integer (nullable = true)\n",
      " |-- SMX: integer (nullable = true)\n",
      " |-- SRNT: integer (nullable = true)\n",
      " |-- SSMC: integer (nullable = true)\n",
      " |-- SVAL: integer (nullable = true)\n",
      " |-- TAXP: integer (nullable = true)\n",
      " |-- WIF: integer (nullable = true)\n",
      " |-- WKEXREL: integer (nullable = true)\n",
      " |-- WORKSTAT: integer (nullable = true)\n",
      " |-- FACCESSP: integer (nullable = true)\n",
      " |-- FACRP: integer (nullable = true)\n",
      " |-- FAGSP: integer (nullable = true)\n",
      " |-- FBATHP: integer (nullable = true)\n",
      " |-- FBDSP: integer (nullable = true)\n",
      " |-- FBLDP: integer (nullable = true)\n",
      " |-- FBROADBNDP: integer (nullable = true)\n",
      " |-- FBUSP: integer (nullable = true)\n",
      " |-- FCOMPOTHXP: integer (nullable = true)\n",
      " |-- FCONP: integer (nullable = true)\n",
      " |-- FDIALUPP: integer (nullable = true)\n",
      " |-- FDSLP: integer (nullable = true)\n",
      " |-- FELEP: integer (nullable = true)\n",
      " |-- FFIBEROPP: integer (nullable = true)\n",
      " |-- FFSP: integer (nullable = true)\n",
      " |-- FFULP: integer (nullable = true)\n",
      " |-- FGASP: integer (nullable = true)\n",
      " |-- FHANDHELDP: integer (nullable = true)\n",
      " |-- FHFLP: integer (nullable = true)\n",
      " |-- FINSP: integer (nullable = true)\n",
      " |-- FKITP: integer (nullable = true)\n",
      " |-- FLAPTOPP: integer (nullable = true)\n",
      " |-- FMHP: integer (nullable = true)\n",
      " |-- FMODEMP: integer (nullable = true)\n",
      " |-- FMRGIP: integer (nullable = true)\n",
      " |-- FMRGP: integer (nullable = true)\n",
      " |-- FMRGTP: integer (nullable = true)\n",
      " |-- FMRGXP: integer (nullable = true)\n",
      " |-- FMVP: integer (nullable = true)\n",
      " |-- FOTHSVCEXP: integer (nullable = true)\n",
      " |-- FPLMP: integer (nullable = true)\n",
      " |-- FREFRP: integer (nullable = true)\n",
      " |-- FRMSP: integer (nullable = true)\n",
      " |-- FRNTMP: integer (nullable = true)\n",
      " |-- FRNTP: integer (nullable = true)\n",
      " |-- FRWATP: integer (nullable = true)\n",
      " |-- FRWATPRP: integer (nullable = true)\n",
      " |-- FSATELLITEP: integer (nullable = true)\n",
      " |-- FSINKP: integer (nullable = true)\n",
      " |-- FSMP: integer (nullable = true)\n",
      " |-- FSMXHP: integer (nullable = true)\n",
      " |-- FSMXSP: integer (nullable = true)\n",
      " |-- FSTOVP: integer (nullable = true)\n",
      " |-- FTAXP: integer (nullable = true)\n",
      " |-- FTELP: integer (nullable = true)\n",
      " |-- FTENP: integer (nullable = true)\n",
      " |-- FTOILP: integer (nullable = true)\n",
      " |-- FVACSP: integer (nullable = true)\n",
      " |-- FVALP: integer (nullable = true)\n",
      " |-- FVEHP: integer (nullable = true)\n",
      " |-- FWATP: integer (nullable = true)\n",
      " |-- FYBLP: integer (nullable = true)\n",
      " |-- wgtp1: integer (nullable = true)\n",
      " |-- wgtp2: integer (nullable = true)\n",
      " |-- wgtp3: integer (nullable = true)\n",
      " |-- wgtp4: integer (nullable = true)\n",
      " |-- wgtp5: integer (nullable = true)\n",
      " |-- wgtp6: integer (nullable = true)\n",
      " |-- wgtp7: integer (nullable = true)\n",
      " |-- wgtp8: integer (nullable = true)\n",
      " |-- wgtp9: integer (nullable = true)\n",
      " |-- wgtp10: integer (nullable = true)\n",
      " |-- wgtp11: integer (nullable = true)\n",
      " |-- wgtp12: integer (nullable = true)\n",
      " |-- wgtp13: integer (nullable = true)\n",
      " |-- wgtp14: integer (nullable = true)\n",
      " |-- wgtp15: integer (nullable = true)\n",
      " |-- wgtp16: integer (nullable = true)\n",
      " |-- wgtp17: integer (nullable = true)\n",
      " |-- wgtp18: integer (nullable = true)\n",
      " |-- wgtp19: integer (nullable = true)\n",
      " |-- wgtp20: integer (nullable = true)\n",
      " |-- wgtp21: integer (nullable = true)\n",
      " |-- wgtp22: integer (nullable = true)\n",
      " |-- wgtp23: integer (nullable = true)\n",
      " |-- wgtp24: integer (nullable = true)\n",
      " |-- wgtp25: integer (nullable = true)\n",
      " |-- wgtp26: integer (nullable = true)\n",
      " |-- wgtp27: integer (nullable = true)\n",
      " |-- wgtp28: integer (nullable = true)\n",
      " |-- wgtp29: integer (nullable = true)\n",
      " |-- wgtp30: integer (nullable = true)\n",
      " |-- wgtp31: integer (nullable = true)\n",
      " |-- wgtp32: integer (nullable = true)\n",
      " |-- wgtp33: integer (nullable = true)\n",
      " |-- wgtp34: integer (nullable = true)\n",
      " |-- wgtp35: integer (nullable = true)\n",
      " |-- wgtp36: integer (nullable = true)\n",
      " |-- wgtp37: integer (nullable = true)\n",
      " |-- wgtp38: integer (nullable = true)\n",
      " |-- wgtp39: integer (nullable = true)\n",
      " |-- wgtp40: integer (nullable = true)\n",
      " |-- wgtp41: integer (nullable = true)\n",
      " |-- wgtp42: integer (nullable = true)\n",
      " |-- wgtp43: integer (nullable = true)\n",
      " |-- wgtp44: integer (nullable = true)\n",
      " |-- wgtp45: integer (nullable = true)\n",
      " |-- wgtp46: integer (nullable = true)\n",
      " |-- wgtp47: integer (nullable = true)\n",
      " |-- wgtp48: integer (nullable = true)\n",
      " |-- wgtp49: integer (nullable = true)\n",
      " |-- wgtp50: integer (nullable = true)\n",
      " |-- wgtp51: integer (nullable = true)\n",
      " |-- wgtp52: integer (nullable = true)\n",
      " |-- wgtp53: integer (nullable = true)\n",
      " |-- wgtp54: integer (nullable = true)\n",
      " |-- wgtp55: integer (nullable = true)\n",
      " |-- wgtp56: integer (nullable = true)\n",
      " |-- wgtp57: integer (nullable = true)\n",
      " |-- wgtp58: integer (nullable = true)\n",
      " |-- wgtp59: integer (nullable = true)\n",
      " |-- wgtp60: integer (nullable = true)\n",
      " |-- wgtp61: integer (nullable = true)\n",
      " |-- wgtp62: integer (nullable = true)\n",
      " |-- wgtp63: integer (nullable = true)\n",
      " |-- wgtp64: integer (nullable = true)\n",
      " |-- wgtp65: integer (nullable = true)\n",
      " |-- wgtp66: integer (nullable = true)\n",
      " |-- wgtp67: integer (nullable = true)\n",
      " |-- wgtp68: integer (nullable = true)\n",
      " |-- wgtp69: integer (nullable = true)\n",
      " |-- wgtp70: integer (nullable = true)\n",
      " |-- wgtp71: integer (nullable = true)\n",
      " |-- wgtp72: integer (nullable = true)\n",
      " |-- wgtp73: integer (nullable = true)\n",
      " |-- wgtp74: integer (nullable = true)\n",
      " |-- wgtp75: integer (nullable = true)\n",
      " |-- wgtp76: integer (nullable = true)\n",
      " |-- wgtp77: integer (nullable = true)\n",
      " |-- wgtp78: integer (nullable = true)\n",
      " |-- wgtp79: integer (nullable = true)\n",
      " |-- wgtp80: integer (nullable = true)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.002   0.000   0.062 "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "system.time(\n",
    "    printSchema(housing_a_df)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good. Let's have a look at the first few rows.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>RT</th><th scope=col>SERIALNO</th><th scope=col>DIVISION</th><th scope=col>PUMA</th><th scope=col>REGION</th><th scope=col>ST</th><th scope=col>ADJHSG</th><th scope=col>ADJINC</th><th scope=col>WGTP</th><th scope=col>NP</th><th scope=col>ellip.h</th><th scope=col>wgtp71</th><th scope=col>wgtp72</th><th scope=col>wgtp73</th><th scope=col>wgtp74</th><th scope=col>wgtp75</th><th scope=col>wgtp76</th><th scope=col>wgtp77</th><th scope=col>wgtp78</th><th scope=col>wgtp79</th><th scope=col>wgtp80</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>H</td><td>84</td><td>6</td><td>2600</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>0</td><td>1</td><td>⋯</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>H</td><td>154</td><td>6</td><td>2500</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>51</td><td>4</td><td>⋯</td><td>86</td><td>53</td><td>59</td><td>84</td><td>49</td><td>15</td><td>15</td><td>20</td><td>50</td><td>16</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>H</td><td>156</td><td>6</td><td>1700</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>449</td><td>1</td><td>⋯</td><td>161</td><td>530</td><td>601</td><td>579</td><td>341</td><td>378</td><td>387</td><td>421</td><td>621</td><td>486</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>H</td><td>160</td><td>6</td><td>2200</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>16</td><td>3</td><td>⋯</td><td>31</td><td>24</td><td>33</td><td>7</td><td>7</td><td>13</td><td>18</td><td>23</td><td>23</td><td>5</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>H</td><td>231</td><td>6</td><td>2400</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>52</td><td>1</td><td>⋯</td><td>21</td><td>18</td><td>37</td><td>49</td><td>103</td><td>38</td><td>49</td><td>51</td><td>46</td><td>47</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>H</td><td>286</td><td>6</td><td>900</td><td>3</td><td>1</td><td>1000000</td><td>1007549</td><td>76</td><td>1</td><td>⋯</td><td>128</td><td>25</td><td>68</td><td>66</td><td>80</td><td>26</td><td>66</td><td>164</td><td>88</td><td>24</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll}\n",
       "  & RT & SERIALNO & DIVISION & PUMA & REGION & ST & ADJHSG & ADJINC & WGTP & NP & ellip.h & wgtp71 & wgtp72 & wgtp73 & wgtp74 & wgtp75 & wgtp76 & wgtp77 & wgtp78 & wgtp79 & wgtp80\\\\\n",
       "\\hline\n",
       "\t1 & H & 84 & 6 & 2600 & 3 & 1 & 1000000 & 1007549 & 0 & 1 & ⋯ & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\\\\n",
       "\t2 & H & 154 & 6 & 2500 & 3 & 1 & 1000000 & 1007549 & 51 & 4 & ⋯ & 86 & 53 & 59 & 84 & 49 & 15 & 15 & 20 & 50 & 16\\\\\n",
       "\t3 & H & 156 & 6 & 1700 & 3 & 1 & 1000000 & 1007549 & 449 & 1 & ⋯ & 161 & 530 & 601 & 579 & 341 & 378 & 387 & 421 & 621 & 486\\\\\n",
       "\t4 & H & 160 & 6 & 2200 & 3 & 1 & 1000000 & 1007549 & 16 & 3 & ⋯ & 31 & 24 & 33 & 7 & 7 & 13 & 18 & 23 & 23 & 5\\\\\n",
       "\t5 & H & 231 & 6 & 2400 & 3 & 1 & 1000000 & 1007549 & 52 & 1 & ⋯ & 21 & 18 & 37 & 49 & 103 & 38 & 49 & 51 & 46 & 47\\\\\n",
       "\t6 & H & 286 & 6 & 900 & 3 & 1 & 1000000 & 1007549 & 76 & 1 & ⋯ & 128 & 25 & 68 & 66 & 80 & 26 & 66 & 164 & 88 & 24\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  RT SERIALNO DIVISION PUMA REGION ST  ADJHSG  ADJINC WGTP NP TYPE ACCESS ACR\n",
       "1  H       84        6 2600      3  1 1000000 1007549    0  1    3     NA  NA\n",
       "2  H      154        6 2500      3  1 1000000 1007549   51  4    1      1   1\n",
       "3  H      156        6 1700      3  1 1000000 1007549  449  1    1      3   1\n",
       "4  H      160        6 2200      3  1 1000000 1007549   16  3    1      3   3\n",
       "5  H      231        6 2400      3  1 1000000 1007549   52  1    1      3  NA\n",
       "6  H      286        6  900      3  1 1000000 1007549   76  1    1      1   1\n",
       "  AGS BATH BDSP BLD BROADBND BUS COMPOTHX CONP DIALUP DSL ELEP FIBEROP FS FULP\n",
       "1  NA   NA   NA  NA       NA  NA       NA   NA     NA  NA   NA      NA  2   NA\n",
       "2  NA    1    3   2        2   2        2   NA      2   2  350       1  2    2\n",
       "3  NA    1    3   2       NA   2        2   NA     NA  NA  300      NA  2    2\n",
       "4   1    1    4   2       NA   2        2   NA     NA  NA  220      NA  1    2\n",
       "5  NA    1    1   5       NA  NA        2   NA     NA  NA   60      NA  1    1\n",
       "6  NA    1    4   2        2   2        2   NA      2   2  100       2  2    2\n",
       "  GASP HANDHELD HFL INSP LAPTOP MHP MODEM MRGI MRGP MRGT MRGX OTHSVCEX REFR\n",
       "1   NA       NA  NA   NA     NA  NA    NA   NA   NA   NA   NA       NA   NA\n",
       "2    3        2   3  350      1  NA     2   NA   NA   NA    3        2    1\n",
       "3    3        2   3  980      1  NA    NA    1  550    2    1       NA    1\n",
       "4   20        1   3   NA      2  NA    NA   NA   NA   NA   NA       NA    1\n",
       "5    3        2   1   NA      2  NA    NA   NA   NA   NA   NA       NA    1\n",
       "6   90        2   1   50      2  NA     1    2  290    2    1        2    1\n",
       "  RMSP RNTM RNTP RWAT RWATPR SATELLITE SINK SMP STOV TEL TEN TOIL VACS  VALP\n",
       "1   NA   NA   NA   NA     NA        NA   NA  NA   NA  NA  NA   NA   NA    NA\n",
       "2    9   NA   NA    1      9         2    1  NA    1   1   2    1   NA 25000\n",
       "3    6   NA   NA    1      9        NA    1  NA    1   1   1    1   NA 80000\n",
       "4    6    2  100    1      9        NA    1  NA    1   1   3    1   NA    NA\n",
       "5    3    2   80    1      9        NA    1  NA    1   1   3    1   NA    NA\n",
       "6    9   NA   NA    1      9         2    1  NA    1   1   1    1   NA 18000\n",
       "  VEH WATP YBL FES FFINCP FGRNTP FHINCP  FINCP FPARC FSMOCP GRNTP GRPIP HHL HHT\n",
       "1  NA   NA  NA  NA     NA     NA     NA     NA    NA     NA    NA    NA  NA  NA\n",
       "2   3  480   2   1      1     NA      1 151000     4      0    NA    NA   1   1\n",
       "3   1  700   5  NA      0     NA      1     NA    NA      1    NA    NA   1   6\n",
       "4   2  360   6   8      1      0      1  11400     2     NA   370    39   1   3\n",
       "5   0    1   5  NA      0      0      0     NA    NA     NA   140    43   1   6\n",
       "6   1  370   2  NA      0     NA      1     NA    NA      1    NA    NA   1   6\n",
       "   HINCP HUGCL HUPAC HUPAOC HUPARC KIT LNGI MULTG MV NOC NPF NPP NR NRC OCPIP\n",
       "1     NA    NA    NA     NA     NA  NA   NA    NA NA  NA  NA  NA NA  NA    NA\n",
       "2 151000     0     4      4      4   1    1     1  6   0   4   0  0   0     3\n",
       "3  39930     0     4      4      4   1    1     1  7   0  NA   0  0   0    28\n",
       "4  11400     1     2      4      2   1    1     1  4   0   3   1  0   1    NA\n",
       "5   3900     0     4      4      4   1    1     1  6   0  NA   0  0   0    NA\n",
       "6   5400     0     4      4      4   1    1     1  7   0  NA   0  0   0   101\n",
       "  PARTNER PLM PSF R18 R60 R65 RESMODE SMOCP SMX SRNT SSMC SVAL TAXP WIF WKEXREL\n",
       "1      NA  NA  NA  NA  NA  NA      NA    NA  NA   NA   NA   NA   NA  NA      NA\n",
       "2       0   1   0   0   0   0       2   426  NA    0    0    1    3   2       1\n",
       "3       0   1   0   0   1   0       2   926   3    0    0    1    6  NA      NA\n",
       "4       0   1   0   1   1   0       2    NA  NA    0    0    0   NA   1      15\n",
       "5       0   1   0   0   1   1       1    NA  NA    1    0    0   NA  NA      NA\n",
       "6       0   1   0   0   1   1       1   522   3    0    0    1    3  NA      NA\n",
       "  WORKSTAT FACCESSP FACRP FAGSP FBATHP FBDSP FBLDP FBROADBNDP FBUSP FCOMPOTHXP\n",
       "1       NA       NA    NA    NA     NA    NA    NA         NA    NA         NA\n",
       "2        1        0     0     0      0     0     0          0     0          0\n",
       "3       NA        0     0     0      0     0     0          0     0          0\n",
       "4       15        0     1     1      0     0     0          0     0          0\n",
       "5       NA        0     0     0      0     0     0          0     0          0\n",
       "6       NA        0     1     0      0     0     1          1     1          1\n",
       "  FCONP FDIALUPP FDSLP FELEP FFIBEROPP FFSP FFULP FGASP FHANDHELDP FHFLP FINSP\n",
       "1    NA       NA    NA    NA        NA    0    NA    NA         NA    NA    NA\n",
       "2     0        0     0     0         0    0     0     0          0     0     1\n",
       "3     0        0     0     0         0    0     0     0          0     0     1\n",
       "4     0        0     0     0         0    0     0     0          0     0     0\n",
       "5     0        0     0     0         0    0     0     0          0     0     0\n",
       "6     0        1     1     1         1    0     1     1          1     0     1\n",
       "  FKITP FLAPTOPP FMHP FMODEMP FMRGIP FMRGP FMRGTP FMRGXP FMVP FOTHSVCEXP FPLMP\n",
       "1    NA       NA   NA      NA     NA    NA     NA     NA    0         NA    NA\n",
       "2     0        0    0       0      0     0      0      0    0          0     0\n",
       "3     0        0    0       0      1     1      1      1    1          0     0\n",
       "4     0        0    0       0      0     0      0      0    0          0     0\n",
       "5     0        0    0       0      0     0      0      0    0          0     0\n",
       "6     0        1    0       1      1     1      1      1    0          1     0\n",
       "  FREFRP FRMSP FRNTMP FRNTP FRWATP FRWATPRP FSATELLITEP FSINKP FSMP FSMXHP\n",
       "1     NA    NA     NA    NA     NA       NA          NA     NA   NA     NA\n",
       "2      0     0      0     0      0        0           0      0    0      0\n",
       "3      0     0      0     0      0        0           0      0    1      1\n",
       "4      0     0      0     0      0        0           0      0    0      0\n",
       "5      0     0      0     0      0        0           0      0    0      0\n",
       "6      0     0      0     0      0        0           1      0    1      1\n",
       "  FSMXSP FSTOVP FTAXP FTELP FTENP FTOILP FVACSP FVALP FVEHP FWATP FYBLP wgtp1\n",
       "1     NA     NA    NA    NA    NA     NA     NA    NA    NA    NA    NA     0\n",
       "2      0      0     1     0     0      0      0     0     0     0     0    45\n",
       "3      1      0     1     0     1      0      0     1     0     0     0   481\n",
       "4      0      0     0     0     0      0      0     0     0     0     0    21\n",
       "5      0      0     0     0     0      0      0     0     0     0     0    74\n",
       "6      1      0     1     0     1      0      0     1     0     1     0    84\n",
       "  wgtp2 wgtp3 wgtp4 wgtp5 wgtp6 wgtp7 wgtp8 wgtp9 wgtp10 wgtp11 wgtp12 wgtp13\n",
       "1     0     0     0     0     0     0     0     0      0      0      0      0\n",
       "2    52    53    50   100    79    78    50    19     57     89     46     67\n",
       "3   576   807   739   154   173   722   455   157    671    681    470    403\n",
       "4    14     6    20    30    16     5    25     4     27     11     28     14\n",
       "5    50    43    13    12   102    80    20    60     98     17     18     52\n",
       "6    69    19   118    91    77    20    76    73    180     68     78     24\n",
       "  wgtp14 wgtp15 wgtp16 wgtp17 wgtp18 wgtp19 wgtp20 wgtp21 wgtp22 wgtp23 wgtp24\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2    109     51     18     17     18     47     87     49     50     49     64\n",
       "3    118    339    436    443    393    160    411    468    387    127    150\n",
       "4     15     27     40     31      4     16     20      9     26     39     32\n",
       "5     45     95     57     46     75     47     54     16     43     57    122\n",
       "6    131    136     73    143     82     26     67     21     27     63     74\n",
       "  wgtp25 wgtp26 wgtp27 wgtp28 wgtp29 wgtp30 wgtp31 wgtp32 wgtp33 wgtp34 wgtp35\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2     13     16     16     55     86     48     14     49     50     11     56\n",
       "3    553    514    133    613    562    151    196    432    577    710    378\n",
       "4     38      9     15      6     19     22      5     16      3     27     21\n",
       "5     93     17     13     61     59     23     82     69     50     55     20\n",
       "6     24    119     87    126    131     67    130     30     79     81     70\n",
       "  wgtp36 wgtp37 wgtp38 wgtp39 wgtp40 wgtp41 wgtp42 wgtp43 wgtp44 wgtp45 wgtp46\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2     83     71    108     54     81     43     53     50     63     17     15\n",
       "3    481    504    531    760    409    560    404    637    516    101    150\n",
       "4     33     14     11      6     31     10     21     22     11      8     22\n",
       "5     59     62     44     38     40     16     47     40    107     88     16\n",
       "6     19     94    119     65     23     84     83     23    111     70     73\n",
       "  wgtp47 wgtp48 wgtp49 wgtp50 wgtp51 wgtp52 wgtp53 wgtp54 wgtp55 wgtp56 wgtp57\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2     15     47     88     47     14     52     51     20     53     84     78\n",
       "3    659    454    179    793    724    451    527    168    355    445    465\n",
       "4     27     16     30      6     28      9     17     12     23      3      4\n",
       "5     14     86     59     16     92    113     61     64     14     74     48\n",
       "6     22     86     69    123     91     75     28    138     98     69    121\n",
       "  wgtp58 wgtp59 wgtp60 wgtp61 wgtp62 wgtp63 wgtp64 wgtp65 wgtp66 wgtp67 wgtp68\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2     74     53     15     54     55     58     65    110     82     89     50\n",
       "3    426    228    575    386    433    167    230    797    572    155    556\n",
       "4     38     15     17     33      8     10      7      7     21     22     19\n",
       "5     43     53     50     83     47     51     15     16     93     95     15\n",
       "6     87     23     67     18     24     77     62     28    125     72    121\n",
       "  wgtp69 wgtp70 wgtp71 wgtp72 wgtp73 wgtp74 wgtp75 wgtp76 wgtp77 wgtp78 wgtp79\n",
       "1      0      0      0      0      0      0      0      0      0      0      0\n",
       "2     12     55     86     53     59     84     49     15     15     20     50\n",
       "3    596    170    161    530    601    579    341    378    387    421    621\n",
       "4     12     11     31     24     33      7      7     13     18     23     23\n",
       "5     47    106     21     18     37     49    103     38     49     51     46\n",
       "6    135    100    128     25     68     66     80     26     66    164     88\n",
       "  wgtp80\n",
       "1      0\n",
       "2     16\n",
       "3    486\n",
       "4      5\n",
       "5     47\n",
       "6     24"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "head(housing_a_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's count how many rows do we have in the first dataset. For that we will use `nrow` as we do with regular R data frames. Let's have a look at the documentation.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<table width=\"100%\" summary=\"page for count {SparkR}\"><tr><td>count {SparkR}</td><td style=\"text-align: right;\">R Documentation</td></tr></table>\n",
       "\n",
       "<h2>Count</h2>\n",
       "\n",
       "<h3>Description</h3>\n",
       "\n",
       "<p>Returns the number of rows in a DataFrame\n",
       "</p>\n",
       "<p>Aggregate function: returns the number of items in a group.\n",
       "</p>\n",
       "<p>Aggregate function: returns the number of items in a group.\n",
       "</p>\n",
       "\n",
       "\n",
       "<h3>Usage</h3>\n",
       "\n",
       "<pre>\n",
       "## S4 method for signature 'DataFrame'\n",
       "count(x)\n",
       "\n",
       "## S4 method for signature 'Column'\n",
       "count(x)\n",
       "\n",
       "## S4 method for signature 'Column'\n",
       "n(x)\n",
       "\n",
       "n(x)\n",
       "</pre>\n",
       "\n",
       "\n",
       "<h3>Arguments</h3>\n",
       "\n",
       "<table summary=\"R argblock\">\n",
       "<tr valign=\"top\"><td><code>x</code></td>\n",
       "<td>\n",
       "<p>A SparkSQL DataFrame</p>\n",
       "</td></tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<h3>See Also</h3>\n",
       "\n",
       "<p>Other agg_funcs: <code>approxCountDistinct</code>,\n",
       "<code>approxCountDistinct</code>,\n",
       "<code>approxCountDistinct</code>; <code>avg</code>,\n",
       "<code>avg</code>; <code>countDistinct</code>,\n",
       "<code>countDistinct</code>, <code>countDistinct</code>,\n",
       "<code>n_distinct</code>; <code>first</code>,\n",
       "<code>first</code>; <code>last</code>,\n",
       "<code>last</code>; <code>max</code>;\n",
       "<code>mean</code>; <code>min</code>;\n",
       "<code>sumDistinct</code>, <code>sumDistinct</code>;\n",
       "<code>sum</code>\n",
       "</p>\n",
       "<p>Other agg_funcs: <code>approxCountDistinct</code>,\n",
       "<code>approxCountDistinct</code>,\n",
       "<code>approxCountDistinct</code>; <code>avg</code>,\n",
       "<code>avg</code>; <code>countDistinct</code>,\n",
       "<code>countDistinct</code>, <code>countDistinct</code>,\n",
       "<code>n_distinct</code>; <code>first</code>,\n",
       "<code>first</code>; <code>last</code>,\n",
       "<code>last</code>; <code>max</code>;\n",
       "<code>mean</code>; <code>min</code>;\n",
       "<code>sumDistinct</code>, <code>sumDistinct</code>;\n",
       "<code>sum</code>\n",
       "</p>\n",
       "\n",
       "\n",
       "<h3>Examples</h3>\n",
       "\n",
       "<pre>\n",
       "## Not run: \n",
       "sc &lt;- sparkR.init()\n",
       "sqlContext &lt;- sparkRSQL.init(sc)\n",
       "path &lt;- \"path/to/file.json\"\n",
       "df &lt;- jsonFile(sqlContext, path)\n",
       "count(df)\n",
       "\n",
       "## End(Not run)\n",
       "## Not run: count(df$c)\n",
       "</pre>\n",
       "\n",
       "<hr /><div style=\"text-align: center;\">[Package <em>SparkR</em> version 1.5.0 ]</div>"
      ],
      "text/latex": [
       "\\inputencoding{utf8}\n",
       "\\HeaderA{count}{Count}{count}\n",
       "\\aliasA{n}{count}{n}\n",
       "\\aliasA{nrow}{count}{nrow}\n",
       "%\n",
       "\\begin{Description}\\relax\n",
       "Returns the number of rows in a DataFrame\n",
       "\n",
       "Aggregate function: returns the number of items in a group.\n",
       "\n",
       "Aggregate function: returns the number of items in a group.\n",
       "\\end{Description}\n",
       "%\n",
       "\\begin{Usage}\n",
       "\\begin{verbatim}\n",
       "## S4 method for signature 'DataFrame'\n",
       "count(x)\n",
       "\n",
       "## S4 method for signature 'Column'\n",
       "count(x)\n",
       "\n",
       "## S4 method for signature 'Column'\n",
       "n(x)\n",
       "\n",
       "n(x)\n",
       "\\end{verbatim}\n",
       "\\end{Usage}\n",
       "%\n",
       "\\begin{Arguments}\n",
       "\\begin{ldescription}\n",
       "\\item[\\code{x}] A SparkSQL DataFrame\n",
       "\\end{ldescription}\n",
       "\\end{Arguments}\n",
       "%\n",
       "\\begin{SeeAlso}\\relax\n",
       "Other agg\\_funcs: \\code{\\LinkA{approxCountDistinct}{approxCountDistinct}},\n",
       "\\code{\\LinkA{approxCountDistinct}{approxCountDistinct}},\n",
       "\\code{\\LinkA{approxCountDistinct}{approxCountDistinct}}; \\code{\\LinkA{avg}{avg}},\n",
       "\\code{\\LinkA{avg}{avg}}; \\code{\\LinkA{countDistinct}{countDistinct}},\n",
       "\\code{\\LinkA{countDistinct}{countDistinct}}, \\code{\\LinkA{countDistinct}{countDistinct}},\n",
       "\\code{\\LinkA{n\\_distinct}{n.Rul.distinct}}; \\code{\\LinkA{first}{first}},\n",
       "\\code{\\LinkA{first}{first}}; \\code{\\LinkA{last}{last}},\n",
       "\\code{\\LinkA{last}{last}}; \\code{\\LinkA{max}{max}};\n",
       "\\code{\\LinkA{mean}{mean}}; \\code{\\LinkA{min}{min}};\n",
       "\\code{\\LinkA{sumDistinct}{sumDistinct}}, \\code{\\LinkA{sumDistinct}{sumDistinct}};\n",
       "\\code{\\LinkA{sum}{sum}}\n",
       "\n",
       "Other agg\\_funcs: \\code{\\LinkA{approxCountDistinct}{approxCountDistinct}},\n",
       "\\code{\\LinkA{approxCountDistinct}{approxCountDistinct}},\n",
       "\\code{\\LinkA{approxCountDistinct}{approxCountDistinct}}; \\code{\\LinkA{avg}{avg}},\n",
       "\\code{\\LinkA{avg}{avg}}; \\code{\\LinkA{countDistinct}{countDistinct}},\n",
       "\\code{\\LinkA{countDistinct}{countDistinct}}, \\code{\\LinkA{countDistinct}{countDistinct}},\n",
       "\\code{\\LinkA{n\\_distinct}{n.Rul.distinct}}; \\code{\\LinkA{first}{first}},\n",
       "\\code{\\LinkA{first}{first}}; \\code{\\LinkA{last}{last}},\n",
       "\\code{\\LinkA{last}{last}}; \\code{\\LinkA{max}{max}};\n",
       "\\code{\\LinkA{mean}{mean}}; \\code{\\LinkA{min}{min}};\n",
       "\\code{\\LinkA{sumDistinct}{sumDistinct}}, \\code{\\LinkA{sumDistinct}{sumDistinct}};\n",
       "\\code{\\LinkA{sum}{sum}}\n",
       "\\end{SeeAlso}\n",
       "%\n",
       "\\begin{Examples}\n",
       "\\begin{ExampleCode}\n",
       "## Not run: \n",
       "sc <- sparkR.init()\n",
       "sqlContext <- sparkRSQL.init(sc)\n",
       "path <- \"path/to/file.json\"\n",
       "df <- jsonFile(sqlContext, path)\n",
       "count(df)\n",
       "\n",
       "## End(Not run)\n",
       "## Not run: count(df$c)\n",
       "\\end{ExampleCode}\n",
       "\\end{Examples}"
      ],
      "text/plain": [
       "count                  package:SparkR                  R Documentation\n",
       "\n",
       "_\bC_\bo_\bu_\bn_\bt\n",
       "\n",
       "_\bD_\be_\bs_\bc_\br_\bi_\bp_\bt_\bi_\bo_\bn:\n",
       "\n",
       "     Returns the number of rows in a DataFrame\n",
       "\n",
       "     Aggregate function: returns the number of items in a group.\n",
       "\n",
       "     Aggregate function: returns the number of items in a group.\n",
       "\n",
       "_\bU_\bs_\ba_\bg_\be:\n",
       "\n",
       "     ## S4 method for signature 'DataFrame'\n",
       "     count(x)\n",
       "     \n",
       "     ## S4 method for signature 'Column'\n",
       "     count(x)\n",
       "     \n",
       "     ## S4 method for signature 'Column'\n",
       "     n(x)\n",
       "     \n",
       "     n(x)\n",
       "     \n",
       "_\bA_\br_\bg_\bu_\bm_\be_\bn_\bt_\bs:\n",
       "\n",
       "       x: A SparkSQL DataFrame\n",
       "\n",
       "_\bS_\be_\be _\bA_\bl_\bs_\bo:\n",
       "\n",
       "     Other agg_funcs: ‘approxCountDistinct’, ‘approxCountDistinct’,\n",
       "     ‘approxCountDistinct’; ‘avg’, ‘avg’; ‘countDistinct’,\n",
       "     ‘countDistinct’, ‘countDistinct’, ‘n_distinct’; ‘first’, ‘first’;\n",
       "     ‘last’, ‘last’; ‘max’; ‘mean’; ‘min’; ‘sumDistinct’,\n",
       "     ‘sumDistinct’; ‘sum’\n",
       "\n",
       "     Other agg_funcs: ‘approxCountDistinct’, ‘approxCountDistinct’,\n",
       "     ‘approxCountDistinct’; ‘avg’, ‘avg’; ‘countDistinct’,\n",
       "     ‘countDistinct’, ‘countDistinct’, ‘n_distinct’; ‘first’, ‘first’;\n",
       "     ‘last’, ‘last’; ‘max’; ‘mean’; ‘min’; ‘sumDistinct’,\n",
       "     ‘sumDistinct’; ‘sum’\n",
       "\n",
       "_\bE_\bx_\ba_\bm_\bp_\bl_\be_\bs:\n",
       "\n",
       "     ## Not run:\n",
       "     \n",
       "     sc <- sparkR.init()\n",
       "     sqlContext <- sparkRSQL.init(sc)\n",
       "     path <- \"path/to/file.json\"\n",
       "     df <- jsonFile(sqlContext, path)\n",
       "     count(df)\n",
       "     ## End(Not run)\n",
       "     \n",
       "     ## Not run: count(df$c)\n",
       "     "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "?nrow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is just a definition of `nrow` by the package *SparkR*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "756065"
      ],
      "text/latex": [
       "756065"
      ],
      "text/markdown": [
       "756065"
      ],
      "text/plain": [
       "[1] 756065"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nrow(housing_a_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the second housing data frame and count the number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.128   0.016   9.666 "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "system.time(\n",
    "    housing_b_df <- read.df(sqlContext, \n",
    "                        housing_b_file_path, \n",
    "                        header='true', \n",
    "                        source = \"com.databricks.spark.csv\", \n",
    "                        inferSchema='true')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] 720248\n"
     ]
    }
   ],
   "source": [
    "print(nrow(housing_b_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merging data frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `rbind()` as we do with regular R data frames to put both of them together. Let's have a look at the documentation.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<table width=\"100%\" summary=\"page for rbind {SparkR}\"><tr><td>rbind {SparkR}</td><td style=\"text-align: right;\">R Documentation</td></tr></table>\n",
       "\n",
       "<h2>Union two or more DataFrames</h2>\n",
       "\n",
       "<h3>Description</h3>\n",
       "\n",
       "<p>Returns a new DataFrame containing rows of all parameters.\n",
       "</p>\n",
       "\n",
       "\n",
       "<h3>Usage</h3>\n",
       "\n",
       "<pre>\n",
       "## S4 method for signature 'DataFrame'\n",
       "rbind(x, ..., deparse.level = 1)\n",
       "\n",
       "rbind(..., deparse.level = 1)\n",
       "</pre>\n",
       "\n",
       "<hr /><div style=\"text-align: center;\">[Package <em>SparkR</em> version 1.5.0 ]</div>"
      ],
      "text/latex": [
       "\\inputencoding{utf8}\n",
       "\\HeaderA{rbind}{Union two or more DataFrames}{rbind}\n",
       "\\aliasA{unionAll}{rbind}{unionAll}\n",
       "%\n",
       "\\begin{Description}\\relax\n",
       "Returns a new DataFrame containing rows of all parameters.\n",
       "\\end{Description}\n",
       "%\n",
       "\\begin{Usage}\n",
       "\\begin{verbatim}\n",
       "## S4 method for signature 'DataFrame'\n",
       "rbind(x, ..., deparse.level = 1)\n",
       "\n",
       "rbind(..., deparse.level = 1)\n",
       "\\end{verbatim}\n",
       "\\end{Usage}"
      ],
      "text/plain": [
       "rbind                  package:SparkR                  R Documentation\n",
       "\n",
       "_\bU_\bn_\bi_\bo_\bn _\bt_\bw_\bo _\bo_\br _\bm_\bo_\br_\be _\bD_\ba_\bt_\ba_\bF_\br_\ba_\bm_\be_\bs\n",
       "\n",
       "_\bD_\be_\bs_\bc_\br_\bi_\bp_\bt_\bi_\bo_\bn:\n",
       "\n",
       "     Returns a new DataFrame containing rows of all parameters.\n",
       "\n",
       "_\bU_\bs_\ba_\bg_\be:\n",
       "\n",
       "     ## S4 method for signature 'DataFrame'\n",
       "     rbind(x, ..., deparse.level = 1)\n",
       "     \n",
       "     rbind(..., deparse.level = 1)\n",
       "     "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "?rbind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, SparkR redefines many of the common R functions to work with SparkSQL data frames. Let's actually use `rbind` as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "housing_df <- rbind(housing_a_df, housing_b_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's count how many rows do we have in the complete data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.001   0.000  17.206 "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] 1476313\n"
     ]
    }
   ],
   "source": [
    "system.time(\n",
    "    housing_samples <- nrow(housing_df)\n",
    ")\n",
    "print(housing_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's get a feeling of what is to explore data using SparkR by using the `summary` function on the data frame. Let's first have a look at the documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<table width=\"100%\" summary=\"page for describe {SparkR}\"><tr><td>describe {SparkR}</td><td style=\"text-align: right;\">R Documentation</td></tr></table>\n",
       "\n",
       "<h2>describe</h2>\n",
       "\n",
       "<h3>Description</h3>\n",
       "\n",
       "<p>Computes statistics for numeric columns.\n",
       "If no columns are given, this function computes statistics for all numerical columns.\n",
       "</p>\n",
       "\n",
       "\n",
       "<h3>Usage</h3>\n",
       "\n",
       "<pre>\n",
       "## S4 method for signature 'DataFrame,character'\n",
       "describe(x, col, ...)\n",
       "\n",
       "## S4 method for signature 'DataFrame,ANY'\n",
       "describe(x)\n",
       "\n",
       "describe(x, col, ...)\n",
       "</pre>\n",
       "\n",
       "\n",
       "<h3>Arguments</h3>\n",
       "\n",
       "<table summary=\"R argblock\">\n",
       "<tr valign=\"top\"><td><code>x</code></td>\n",
       "<td>\n",
       "<p>A DataFrame to be computed.</p>\n",
       "</td></tr>\n",
       "<tr valign=\"top\"><td><code>col</code></td>\n",
       "<td>\n",
       "<p>A string of name</p>\n",
       "</td></tr>\n",
       "<tr valign=\"top\"><td><code>...</code></td>\n",
       "<td>\n",
       "<p>Additional expressions</p>\n",
       "</td></tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<h3>Value</h3>\n",
       "\n",
       "<p>A DataFrame\n",
       "</p>\n",
       "\n",
       "\n",
       "<h3>Examples</h3>\n",
       "\n",
       "<pre>\n",
       "## Not run: \n",
       "sc &lt;- sparkR.init()\n",
       "sqlContext &lt;- sparkRSQL.init(sc)\n",
       "path &lt;- \"path/to/file.json\"\n",
       "df &lt;- jsonFile(sqlContext, path)\n",
       "describe(df)\n",
       "describe(df, \"col1\")\n",
       "describe(df, \"col1\", \"col2\")\n",
       "\n",
       "## End(Not run)\n",
       "</pre>\n",
       "\n",
       "<hr /><div style=\"text-align: center;\">[Package <em>SparkR</em> version 1.5.0 ]</div>"
      ],
      "text/latex": [
       "\\inputencoding{utf8}\n",
       "\\HeaderA{describe}{describe}{describe}\n",
       "\\aliasA{summary}{describe}{summary}\n",
       "%\n",
       "\\begin{Description}\\relax\n",
       "Computes statistics for numeric columns.\n",
       "If no columns are given, this function computes statistics for all numerical columns.\n",
       "\\end{Description}\n",
       "%\n",
       "\\begin{Usage}\n",
       "\\begin{verbatim}\n",
       "## S4 method for signature 'DataFrame,character'\n",
       "describe(x, col, ...)\n",
       "\n",
       "## S4 method for signature 'DataFrame,ANY'\n",
       "describe(x)\n",
       "\n",
       "describe(x, col, ...)\n",
       "\\end{verbatim}\n",
       "\\end{Usage}\n",
       "%\n",
       "\\begin{Arguments}\n",
       "\\begin{ldescription}\n",
       "\\item[\\code{x}] A DataFrame to be computed.\n",
       "\n",
       "\\item[\\code{col}] A string of name\n",
       "\n",
       "\\item[\\code{...}] Additional expressions\n",
       "\\end{ldescription}\n",
       "\\end{Arguments}\n",
       "%\n",
       "\\begin{Value}\n",
       "A DataFrame\n",
       "\\end{Value}\n",
       "%\n",
       "\\begin{Examples}\n",
       "\\begin{ExampleCode}\n",
       "## Not run: \n",
       "sc <- sparkR.init()\n",
       "sqlContext <- sparkRSQL.init(sc)\n",
       "path <- \"path/to/file.json\"\n",
       "df <- jsonFile(sqlContext, path)\n",
       "describe(df)\n",
       "describe(df, \"col1\")\n",
       "describe(df, \"col1\", \"col2\")\n",
       "\n",
       "## End(Not run)\n",
       "\\end{ExampleCode}\n",
       "\\end{Examples}"
      ],
      "text/plain": [
       "describe                package:SparkR                 R Documentation\n",
       "\n",
       "_\bd_\be_\bs_\bc_\br_\bi_\bb_\be\n",
       "\n",
       "_\bD_\be_\bs_\bc_\br_\bi_\bp_\bt_\bi_\bo_\bn:\n",
       "\n",
       "     Computes statistics for numeric columns. If no columns are given,\n",
       "     this function computes statistics for all numerical columns.\n",
       "\n",
       "_\bU_\bs_\ba_\bg_\be:\n",
       "\n",
       "     ## S4 method for signature 'DataFrame,character'\n",
       "     describe(x, col, ...)\n",
       "     \n",
       "     ## S4 method for signature 'DataFrame,ANY'\n",
       "     describe(x)\n",
       "     \n",
       "     describe(x, col, ...)\n",
       "     \n",
       "_\bA_\br_\bg_\bu_\bm_\be_\bn_\bt_\bs:\n",
       "\n",
       "       x: A DataFrame to be computed.\n",
       "\n",
       "     col: A string of name\n",
       "\n",
       "     ...: Additional expressions\n",
       "\n",
       "_\bV_\ba_\bl_\bu_\be:\n",
       "\n",
       "     A DataFrame\n",
       "\n",
       "_\bE_\bx_\ba_\bm_\bp_\bl_\be_\bs:\n",
       "\n",
       "     ## Not run:\n",
       "     \n",
       "     sc <- sparkR.init()\n",
       "     sqlContext <- sparkRSQL.init(sc)\n",
       "     path <- \"path/to/file.json\"\n",
       "     df <- jsonFile(sqlContext, path)\n",
       "     describe(df)\n",
       "     describe(df, \"col1\")\n",
       "     describe(df, \"col1\", \"col2\")\n",
       "     ## End(Not run)\n",
       "     "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "?summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we are redirected to `describe`, that optionally accepts column names. Let's use it with the whole data frame, that is, the 231 columns and 1476313 rows. *Note*: We use `collect` here because the results of `describe` are given as a `DataFrame` object and we need to print them in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.174   0.016 197.755 "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "system.time(\n",
    "    housing_summary <- describe(housing_df)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>summary</th><th scope=col>RT</th><th scope=col>SERIALNO</th><th scope=col>DIVISION</th><th scope=col>PUMA</th><th scope=col>REGION</th><th scope=col>ST</th><th scope=col>ADJHSG</th><th scope=col>ADJINC</th><th scope=col>WGTP</th><th scope=col>ellip.h</th><th scope=col>wgtp71</th><th scope=col>wgtp72</th><th scope=col>wgtp73</th><th scope=col>wgtp74</th><th scope=col>wgtp75</th><th scope=col>wgtp76</th><th scope=col>wgtp77</th><th scope=col>wgtp78</th><th scope=col>wgtp79</th><th scope=col>wgtp80</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>count</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>⋯</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td><td>1476313</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>mean</td><td>NA</td><td>746447.7617598707</td><td>5.114928202894643</td><td>4458.457590632881</td><td>2.6305945961323918</td><td>27.8048550679971</td><td>1000000.0</td><td>1007549.0</td><td>89.95933585899468</td><td>⋯</td><td>89.95847289836234</td><td>89.9587072660066</td><td>89.95868220357065</td><td>89.95941511048132</td><td>89.95922341671448</td><td>89.95888473514763</td><td>89.95931147392186</td><td>89.95866933367111</td><td>89.95865239959276</td><td>89.95870862073286</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>stddev</td><td>NA</td><td>NaN</td><td>2.476670855019605</td><td>NaN</td><td>1.0144176501154056</td><td>15.894404191598897</td><td>NaN</td><td>NaN</td><td>80.66731786485431</td><td>⋯</td><td>99.43323120400835</td><td>100.08749152422924</td><td>98.37291356061534</td><td>99.52607767089758</td><td>98.67766416960376</td><td>99.11341616484188</td><td>99.05644177173914</td><td>98.94559614502197</td><td>97.87387141885895</td><td>100.18396018153416</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>min</td><td>H</td><td>1</td><td>1</td><td>100</td><td>1</td><td>1</td><td>1000000</td><td>1007549</td><td>0</td><td>⋯</td><td>-8</td><td>-494</td><td>-14</td><td>-151</td><td>-38</td><td>-5</td><td>-3</td><td>-16</td><td>-22</td><td>-243</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>max</td><td>H</td><td>1492845</td><td>9</td><td>70301</td><td>4</td><td>56</td><td>1000000</td><td>1007549</td><td>1829</td><td>⋯</td><td>2282</td><td>2328</td><td>2393</td><td>2348</td><td>2263</td><td>2310</td><td>2131</td><td>2794</td><td>2710</td><td>2447</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll}\n",
       "  & summary & RT & SERIALNO & DIVISION & PUMA & REGION & ST & ADJHSG & ADJINC & WGTP & ellip.h & wgtp71 & wgtp72 & wgtp73 & wgtp74 & wgtp75 & wgtp76 & wgtp77 & wgtp78 & wgtp79 & wgtp80\\\\\n",
       "\\hline\n",
       "\t1 & count & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & ⋯ & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313 & 1476313\\\\\n",
       "\t2 & mean & NA & 746447.7617598707 & 5.114928202894643 & 4458.457590632881 & 2.6305945961323918 & 27.8048550679971 & 1000000.0 & 1007549.0 & 89.95933585899468 & ⋯ & 89.95847289836234 & 89.9587072660066 & 89.95868220357065 & 89.95941511048132 & 89.95922341671448 & 89.95888473514763 & 89.95931147392186 & 89.95866933367111 & 89.95865239959276 & 89.95870862073286\\\\\n",
       "\t3 & stddev & NA & NaN & 2.476670855019605 & NaN & 1.0144176501154056 & 15.894404191598897 & NaN & NaN & 80.66731786485431 & ⋯ & 99.43323120400835 & 100.08749152422924 & 98.37291356061534 & 99.52607767089758 & 98.67766416960376 & 99.11341616484188 & 99.05644177173914 & 98.94559614502197 & 97.87387141885895 & 100.18396018153416\\\\\n",
       "\t4 & min & H & 1 & 1 & 100 & 1 & 1 & 1000000 & 1007549 & 0 & ⋯ & -8 & -494 & -14 & -151 & -38 & -5 & -3 & -16 & -22 & -243\\\\\n",
       "\t5 & max & H & 1492845 & 9 & 70301 & 4 & 56 & 1000000 & 1007549 & 1829 & ⋯ & 2282 & 2328 & 2393 & 2348 & 2263 & 2310 & 2131 & 2794 & 2710 & 2447\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  summary      RT          SERIALNO          DIVISION              PUMA\n",
       "1   count 1476313           1476313           1476313           1476313\n",
       "2    mean    <NA> 746447.7617598707 5.114928202894643 4458.457590632881\n",
       "3  stddev    <NA>               NaN 2.476670855019605               NaN\n",
       "4     min       H                 1                 1               100\n",
       "5     max       H           1492845                 9             70301\n",
       "              REGION                 ST    ADJHSG    ADJINC              WGTP\n",
       "1            1476313            1476313   1476313   1476313           1476313\n",
       "2 2.6305945961323918   27.8048550679971 1000000.0 1007549.0 89.95933585899468\n",
       "3 1.0144176501154056 15.894404191598897       NaN       NaN 80.66731786485431\n",
       "4                  1                  1   1000000   1007549                 0\n",
       "5                  4                 56   1000000   1007549              1829\n",
       "                  NP               TYPE             ACCESS                ACR\n",
       "1            1476313            1476313            1211264            1048888\n",
       "2  2.122039838435345 1.1511264887594974 1.4591451574553524 1.3099472965655055\n",
       "3 1.5066952326359986 0.4792639702444521 0.8182636016373733 0.5753510215594196\n",
       "4                  0                  1                  1                  1\n",
       "5                 20                  3                  3                  3\n",
       "                 AGS                BATH               BDSP                BLD\n",
       "1             240718             1328057            1328057            1328057\n",
       "2 1.2790609759137248  1.0097164504234382 2.7954598334258245 2.9454202643410636\n",
       "3 1.0324801853099894 0.09809200280658492 1.1626453967843118 2.0526772518004583\n",
       "4                  1                   1                  0                  1\n",
       "5                  6                   2                 19                 10\n",
       "             BROADBND                 BUS            COMPOTHX\n",
       "1              910225             1048888             1211264\n",
       "2  1.5566063335988354  1.9849879110067044  1.9530713370495614\n",
       "3 0.49678538927438015 0.12160068329311675 0.21148608357081813\n",
       "4                   1                   1                   1\n",
       "5                   2                   2                   2\n",
       "                CONP              DIALUP                DSL               ELEP\n",
       "1              45304              910225             910225            1211264\n",
       "2 321.36614868444286  1.9596121838007086 1.6977955999890137 134.60164093046603\n",
       "3 236.32344819883204 0.19686706301955062 0.4592133497896005  93.52744730809084\n",
       "4                  4                   1                  1                  1\n",
       "5               1600                   2                  2                600\n",
       "              FIBEROP                 FS               FULP              GASP\n",
       "1              910225            1359520            1211264           1211264\n",
       "2   1.890910489164767  1.879595004119101 128.69548174468986 47.62741153043432\n",
       "3 0.31175148670208863 0.3254345292801913   537.625361994371 71.87805326129471\n",
       "4                   1                  1                  1                 1\n",
       "5                   2                  2               7200               550\n",
       "             HANDHELD                HFL              INSP             LAPTOP\n",
       "1             1211264            1211264            847365            1211264\n",
       "2  1.3795308041847194 2.2461684653386875 910.6082502817558 1.2069672672514002\n",
       "3 0.48527020602919746 1.5051855327058254 915.1043685741101 0.4051318520406504\n",
       "4                   1                  1                 0                  1\n",
       "5                   2                  9              9100                  2\n",
       "                 MHP              MODEM               MRGI               MRGP\n",
       "1              55791             910225             517290             517290\n",
       "2 1299.0785610582352   1.44533933917438 1.4433663902259855 1186.7885286783041\n",
       "3  2208.088223691239 0.4970032315368849  0.496782280525347   859.228804208774\n",
       "4                  0                  1                  1                  4\n",
       "5              12800                  2                  2               7300\n",
       "                MRGT               MRGX            OTHSVCEX               REFR\n",
       "1             517290             847365              910225            1328057\n",
       "2 1.3524715343424385 1.7879980881910393  1.9832184350023345 1.0176234905580106\n",
       "3 0.4777398369622586 0.9726866158693994 0.12845210809439703 0.1315785056100068\n",
       "4                  1                  1                   1                  1\n",
       "5                  2                  3                   2                  2\n",
       "                RMSP                RNTM              RNTP                RWAT\n",
       "1            1328057              359751            359751             1328057\n",
       "2 5.9988795661631995  1.9799611397883536 859.6981245361375  1.0156145406409514\n",
       "3  2.473512938868174 0.14013316628501774 555.5034516680532 0.12397873511826024\n",
       "4                  1                   1                 4                   1\n",
       "5                 25                   2              3900                   2\n",
       "   RWATPR           SATELLITE                SINK                SMP\n",
       "1 1328057              910225             1328057              92382\n",
       "2     9.0   1.932411217006784  1.0087292939986763 385.05931891494015\n",
       "3     0.0 0.25103892010346135 0.09302200505773298  400.7238987849853\n",
       "4       9                   1                   1                  4\n",
       "5       9                   2                   2               3500\n",
       "                 STOV                TEL                TEN                TOIL\n",
       "1             1328057            1211264            1211264             1328057\n",
       "2  1.0180075102198174 1.0236183028637853  1.893678009088027  1.0107427617941098\n",
       "3 0.13297834333266625 0.1518567701277085 0.8815534411521971 0.10308906277168586\n",
       "4                   1                  1                  1                   1\n",
       "5                   2                  2                  4                   2\n",
       "                VACS               VALP               VEH               WATP\n",
       "1             116793             859691           1211264            1211264\n",
       "2  4.796417593520159 247682.84302150423  1.81503867034767 397.09110895725456\n",
       "3 2.0446465455876703                NaN 1.074267038152881 505.54512690158475\n",
       "4                  1                100                 0                  1\n",
       "5                  7            4775000                 6               4000\n",
       "                 YBL                FES              FFINCP              FGRNTP\n",
       "1            1328057             809366             1328057              339292\n",
       "2  5.044344482202194  3.123169246051848 0.18535650201760917 0.20184383952465723\n",
       "3 2.8223652839626028 2.3378140651304737  0.3885865015378239   0.401376262340714\n",
       "4                  1                  1                   0                   0\n",
       "5                 17                  8                   1                   1\n",
       "               FHINCP             FINCP              FPARC              FSMOCP\n",
       "1             1328057            812468             812468              847365\n",
       "2 0.27083250191821584 88154.62186203025  3.134906236307153 0.31369008632643547\n",
       "3  0.4443897589087033               NaN 1.0649981594677524 0.46399204310736725\n",
       "4                   0            -19770                  1                   0\n",
       "5                   1           2060000                  4                   1\n",
       "               GRNTP              GRPIP                HHL                HHT\n",
       "1             339292             331644            1211264            1211264\n",
       "2 1007.7917221744102  39.64069604756908  1.320587419423016 2.6879887456409173\n",
       "3  572.6964446082056 27.555001020885555 0.7848872588177565 2.0175271675745132\n",
       "4                  4                  1                  1                  1\n",
       "5               4663                101                  5                  7\n",
       "              HINCP               HUGCL              HUPAC             HUPAOC\n",
       "1           1211264             1211264            1211264            1211264\n",
       "2 75226.56424941219 0.03945465233012787  3.405144543220966  3.474334249181021\n",
       "3               NaN 0.19467404228514032 0.9851110359823945 0.9436759846045979\n",
       "4            -19770                   0                  1                  1\n",
       "5           2090000                   1                  4                  4\n",
       "             HUPARC                 KIT                LNGI              MULTG\n",
       "1           1211264             1328057             1211264            1211264\n",
       "2 3.411732702367114   1.023566759559266  1.0369721216844552 1.0377828450280038\n",
       "3 0.982587466935828 0.15169498146986135 0.18869335945550722 0.1906706627921392\n",
       "4                 1                   1                   1                  1\n",
       "5                 4                   2                   2                  2\n",
       "                  MV                NOC                NPF                  NPP\n",
       "1            1211259            1211264             812468              1211264\n",
       "2 4.2244953391471185 0.4825702736975589 3.0029736555778195 0.005636260963753566\n",
       "3 1.8279586940054013 0.9513097174629106 1.2870353876533343   0.0748631653492025\n",
       "4                  1                  0                  2                    0\n",
       "5                  7                 18                 20                    1\n",
       "                   NR                NRC              OCPIP            PARTNER\n",
       "1             1211264            1211264             840643            1211264\n",
       "2 0.09667752034238614 0.5410001453027581 24.084478191098956 0.1595506842438973\n",
       "3  0.2955181507163877 0.9992326766128944  21.25932465256715 0.7079723145119103\n",
       "4                   0                  0                  1                  0\n",
       "5                   1                 18                101                  4\n",
       "                 PLM                 PSF                 R18                R60\n",
       "1            1328057             1211264             1211264            1211264\n",
       "2 1.0175271091526945 0.03158271029271901 0.29500175023776815 0.5882631697136215\n",
       "3 0.1312246531618359 0.17488637083342198 0.45604354791447455 0.7651725492174631\n",
       "4                  1                   0                   0                  0\n",
       "5                  2                   1                   1                  2\n",
       "                 R65            RESMODE              SMOCP                SMX\n",
       "1            1211264            1328057             847365             517290\n",
       "2 0.4212269166754729  2.067044562093344  1227.739407457235  2.794792089543583\n",
       "3 0.6814416607367314 0.8145012909508376 1036.4845538890377 0.5054417284051073\n",
       "4                  0                  1                  0                  1\n",
       "5                  2                  3              13071                  4\n",
       "                 SRNT                  SSMC                SVAL\n",
       "1             1328057               1211264             1328057\n",
       "2   0.285497535120857 0.0049568054528162315  0.5323242902977809\n",
       "3 0.45165107390636416   0.09860996001801868 0.49895404623747147\n",
       "4                   0                     0                   0\n",
       "5                   1                     2                   1\n",
       "                TAXP                WIF            WKEXREL          WORKSTAT\n",
       "1             847365             812468             812468            809366\n",
       "2  32.50750385017082 1.4557422569258112  6.382029815328111 5.843664053098351\n",
       "3 19.756266752215417 0.8885991378269169 4.6175715515215705 4.877030986497617\n",
       "4                  1                  0                  1                 1\n",
       "5                 68                  3                 15                15\n",
       "             FACCESSP               FACRP                FAGSP\n",
       "1             1328057             1328057              1328057\n",
       "2 0.04298911868993575   0.034202598231853 0.007760962067140191\n",
       "3 0.20283257717683903 0.18174922422404846  0.08775380068653779\n",
       "4                   0                   0                    0\n",
       "5                   1                   1                    1\n",
       "                FBATHP               FBDSP                FBLDP\n",
       "1              1328057             1328057              1328057\n",
       "2 0.020873351068515886 0.04032055853024381 0.013326235244420986\n",
       "3  0.14296032415913995 0.19670996693114712  0.11466754858560176\n",
       "4                    0                   0                    0\n",
       "5                    1                   1                    1\n",
       "           FBROADBNDP               FBUSP          FCOMPOTHXP\n",
       "1             1328057             1328057             1328057\n",
       "2 0.21894165687165537 0.02172120624340672 0.03852921975487498\n",
       "3  0.4135289684628511  0.1457717237420828 0.19247004696823744\n",
       "4                   0                   0                   0\n",
       "5                   1                   1                   1\n",
       "                  FCONP            FDIALUPP               FDSLP\n",
       "1               1328057             1328057             1328057\n",
       "2 0.0044990538809704704  0.0400442149696888 0.04011951294259207\n",
       "3   0.06692392991409421 0.19606293840792594 0.19623949047997769\n",
       "4                     0                   0                   0\n",
       "5                     1                   1                   1\n",
       "                 FELEP           FFIBEROPP                 FFSP\n",
       "1              1328057             1328057              1476313\n",
       "2 0.061608048449727684 0.04010520632774045 0.020226063172240575\n",
       "3  0.24044229414964352 0.19620596003472937  0.14077275851809223\n",
       "4                    0                   0                    0\n",
       "5                    1                   1                    1\n",
       "                FFULP               FGASP          FHANDHELDP\n",
       "1             1328057             1328057             1328057\n",
       "2  0.0919809917797203 0.09274150130604333 0.03381481367140115\n",
       "3 0.28899911579612025  0.2900698454537191  0.1807522393990447\n",
       "4                   0                   0                   0\n",
       "5                   1                   1                   1\n",
       "                FHFLP               FINSP                FKITP\n",
       "1             1328057             1328057              1328057\n",
       "2 0.03258293883470363 0.14465041786610064 0.030948219842973607\n",
       "3 0.17754236376594063  0.3517480269699648  0.17317744521595352\n",
       "4                   0                   0                    0\n",
       "5                   1                   1                    1\n",
       "              FLAPTOPP                FMHP             FMODEMP\n",
       "1              1328057             1328057             1328057\n",
       "2 0.032075430497335584 0.00938062146428956 0.04010520632774045\n",
       "3  0.17620044624162046 0.09639826453434351 0.19620596003472937\n",
       "4                    0                   0                   0\n",
       "5                    1                   1                   1\n",
       "               FMRGIP                FMRGP              FMRGTP\n",
       "1             1328057              1328057             1328057\n",
       "2 0.04279861481849047 0.043326453608542405 0.04075728677308278\n",
       "3  0.2024027998524451  0.20359094288855098 0.19772741425502804\n",
       "4                   0                    0                   0\n",
       "5                   1                    1                   1\n",
       "               FMRGXP                FMVP          FOTHSVCEXP\n",
       "1             1328057             1476313             1328057\n",
       "2  0.0144451631217636 0.03443781908037117 0.04010520632774045\n",
       "3 0.11931680679665058  0.1823509136181083 0.19620596003472937\n",
       "4                   0                   0                   0\n",
       "5                   1                   1                   1\n",
       "                FPLMP               FREFRP               FRMSP\n",
       "1             1328057              1328057             1328057\n",
       "2 0.02438298958553737 0.025707480928905912 0.05212050386391548\n",
       "3 0.15423507838494116   0.1582611966124225 0.22227000909004133\n",
       "4                   0                    0                   0\n",
       "5                   1                    1                   1\n",
       "                FRNTMP                FRNTP              FRWATP FRWATPRP\n",
       "1              1328057              1328057             1328057  1328057\n",
       "2 0.006895035378752568 0.023619468140298194 0.01842315503024343      0.0\n",
       "3  0.08274958529127695  0.15186042560544744  0.1344758059614258      0.0\n",
       "4                    0                    0                   0        0\n",
       "5                    1                    1                   1        0\n",
       "          FSATELLITEP               FSINKP                FSMP\n",
       "1             1328057              1328057             1328057\n",
       "2 0.04010520632774045 0.020740826636206126 0.03478992242049852\n",
       "3 0.19620596003472937  0.14251541932946402 0.18324732936246088\n",
       "4                   0                    0                   0\n",
       "5                   1                    1                   1\n",
       "                FSMXHP               FSMXSP              FSTOVP\n",
       "1              1328057              1328057             1328057\n",
       "2 0.014446669081221664 0.012615422380214103 0.02446958225437613\n",
       "3  0.11932293506983198  0.11160767670005006 0.15450185047005893\n",
       "4                    0                    0                   0\n",
       "5                    1                    1                   1\n",
       "                FTAXP                FTELP                FTENP\n",
       "1             1328057              1328057              1328057\n",
       "2 0.10138043773723568 0.013002453960936917 0.011772838063426494\n",
       "3    0.30183181505839  0.11328455389827262   0.1078621265660882\n",
       "4                   0                    0                    0\n",
       "5                   1                    1                    1\n",
       "               FTOILP               FVACSP               FVALP\n",
       "1             1328057              1328057             1328057\n",
       "2 0.01989824231941852 0.003515662354853745 0.07633407301042049\n",
       "3  0.1396506436505619 0.059188702241731984 0.26553188567111913\n",
       "4                   0                    0                   0\n",
       "5                   1                    1                   1\n",
       "                 FVEHP               FWATP               FYBLP\n",
       "1              1328057             1328057             1328057\n",
       "2 0.011967106833516934 0.07128986180563034 0.12318145983191986\n",
       "3  0.10873773580294988   0.257308409130686  0.3286453830279637\n",
       "4                    0                   0                   0\n",
       "5                    1                   1                   1\n",
       "              wgtp1             wgtp2             wgtp3             wgtp4\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95902969085824 89.95929724929606 89.95749952753921 89.95881903092366\n",
       "3 99.49225610598178 99.50192834198567 99.52190407472071 99.23545244384397\n",
       "4              -242               -12              -190               -49\n",
       "5              2503              2271              2442              2162\n",
       "              wgtp5             wgtp6             wgtp7             wgtp8\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95899446797529 89.95882580455499 89.95919429009973  89.9595234885827\n",
       "3 99.23613584336223 99.72529062899496  98.9976594938058 99.53651134674828\n",
       "4              -258              -323              -469               -18\n",
       "5              1888              2107              2388              2473\n",
       "               wgtp9             wgtp10            wgtp11            wgtp12\n",
       "1            1476313            1476313           1476313           1476313\n",
       "2  89.95823514390241  89.95847222099921 89.95774405563049  89.9593907254085\n",
       "3 100.31475367593421 100.58691939990582 99.62940742854609 99.20197556273386\n",
       "4                -46                -51               -30               -50\n",
       "5               2261               2260              2317              2854\n",
       "             wgtp13            wgtp14            wgtp15            wgtp16\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95979917537812 89.95917329184259 89.95931282864812 89.95931011919559\n",
       "3 97.17846506691616  98.9337121155128 97.33497007224051 98.55623394462495\n",
       "4               -18               -18              -227               -12\n",
       "5              2285              2520              2074              2471\n",
       "             wgtp17            wgtp18            wgtp19            wgtp20\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95910826498175 89.95887457470062 89.95851286278723 89.95858127646373\n",
       "3 98.85358949506218 98.70179262196326 97.36286839828604  99.5174311660373\n",
       "4              -248              -106               -46              -136\n",
       "5              2207              2280              2095              2619\n",
       "             wgtp21            wgtp22            wgtp23            wgtp24\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95931011919559 89.95931147392186 89.95888338042136 89.95884951226468\n",
       "3 98.19685499329904 98.74418906104583 98.79194106724523 99.58075722983519\n",
       "4               -69              -135              -243               -94\n",
       "5              2163              1997              2108              2366\n",
       "             wgtp25            wgtp26            wgtp27            wgtp28\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95923154507209 89.95886983315869 89.95936024406748 89.95873639262135\n",
       "3 98.60288821001899 99.23577373175398 99.09669669308057 99.53192106702299\n",
       "4               -55              -176               -63               -70\n",
       "5              2160              2487              2475              2148\n",
       "             wgtp29            wgtp30            wgtp31            wgtp32\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95942256147579 89.95902359459004 89.95860362944714 89.95919429009973\n",
       "3 99.11835431039984 99.68306761926492 99.28130400908698 99.41373597902758\n",
       "4               -44               -34              -261              -216\n",
       "5              2360              2170              2456              2320\n",
       "             wgtp33            wgtp34            wgtp35            wgtp36\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95918209756333 89.95908455727206 89.95877093814117 89.95821279091899\n",
       "3  99.0608479931526 99.27229039627883 98.86932842865737 99.52568303350506\n",
       "4               -41               -81               -42               -42\n",
       "5              2164              2177              2185              2306\n",
       "             wgtp37            wgtp38            wgtp39            wgtp40\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95909945926101 89.95845190010519 89.95859008218447 89.95926134904997\n",
       "3 99.35317424304719 99.10843471220805 98.51442960485853 99.87528029218211\n",
       "4               -38              -340               -36              -269\n",
       "5              2794              2362              2128              2486\n",
       "             wgtp41            wgtp42            wgtp43            wgtp44\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95913806895963 89.95911910279189 89.95941578784445 89.96003422038551\n",
       "3 99.58936050494432 99.08076350861353 99.88154977993949 99.58468690630565\n",
       "4              -259               -11              -117               -85\n",
       "5              2438              2404              2076              2283\n",
       "             wgtp45            wgtp46            wgtp47            wgtp48\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95930605501678 89.95960409479562 89.95927354158637 89.95848509089875\n",
       "3 99.53396646191061 99.53852932747454 98.69838906136911 99.87759811145453\n",
       "4               -38              -802              -578              -293\n",
       "5              1985              2781              2679              2628\n",
       "             wgtp49             wgtp50            wgtp51            wgtp52\n",
       "1           1476313            1476313           1476313           1476313\n",
       "2 89.95880006475592  89.95879600057711 89.95920512790987 89.95923425452462\n",
       "3 99.87667140850483 100.00298757913698 97.35363509356718 98.42183922063417\n",
       "4              -118                -19               -22               -82\n",
       "5              2296               2313              2199              2222\n",
       "             wgtp53            wgtp54            wgtp55            wgtp56\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2  89.9591455199541 89.95933179481587 89.95879871002965 89.95872216799555\n",
       "3 98.05625985734078 98.61237350713262  97.4719617690276 99.22857680224587\n",
       "4              -129              -202              -382                -5\n",
       "5              2258              2212              2068              2567\n",
       "             wgtp57            wgtp58            wgtp59            wgtp60\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2  89.9587777117725  89.9584681568204  89.9579919705374 89.95936904978822\n",
       "3 99.05391359711766 98.71000563057922 97.92168419037358 99.83694994401064\n",
       "4               -52               -11               -60               -70\n",
       "5              2466              2332              2069              2108\n",
       "             wgtp61            wgtp62            wgtp63            wgtp64\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95936769506196 89.95848644562501 89.95933518163153 89.95880616102411\n",
       "3  98.0814709967849 99.11462931191114  98.5619345554546 99.16229089456633\n",
       "4               -50               -83               -49              -114\n",
       "5              2283              2209              2424              2174\n",
       "             wgtp65            wgtp66            wgtp67            wgtp68\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95848373617248 89.95870455655407  89.9586429165089 89.95856231029599\n",
       "3 98.25800607110642 99.32339518134275 99.50241206650541 99.26585588285617\n",
       "4               -21              -162               -43               -21\n",
       "5              2399              2469              2011              2213\n",
       "             wgtp69            wgtp70            wgtp71             wgtp72\n",
       "1           1476313           1476313           1476313            1476313\n",
       "2 89.95952823012465 89.95831507275219 89.95847289836234   89.9587072660066\n",
       "3 99.73191204893502 100.3266544475812 99.43323120400835 100.08749152422924\n",
       "4               -49                -3                -8               -494\n",
       "5              2601              2441              2282               2328\n",
       "             wgtp73            wgtp74            wgtp75            wgtp76\n",
       "1           1476313           1476313           1476313           1476313\n",
       "2 89.95868220357065 89.95941511048132 89.95922341671448 89.95888473514763\n",
       "3 98.37291356061534 99.52607767089758 98.67766416960376 99.11341616484188\n",
       "4               -14              -151               -38                -5\n",
       "5              2393              2348              2263              2310\n",
       "             wgtp77            wgtp78            wgtp79             wgtp80\n",
       "1           1476313           1476313           1476313            1476313\n",
       "2 89.95931147392186 89.95866933367111 89.95865239959276  89.95870862073286\n",
       "3 99.05644177173914 98.94559614502197 97.87387141885895 100.18396018153416\n",
       "4                -3               -16               -22               -243\n",
       "5              2131              2794              2710               2447"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collect(housing_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can select individual column summaries using `select` as follows (here is [a dictionary](http://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMSDataDict13.txt) of each column meaning) where `VALP` is the property value.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>VALP</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>859691</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>247682.84302150423</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>NaN</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>100</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>4775000</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       "  & VALP\\\\\n",
       "\\hline\n",
       "\t1 & 859691\\\\\n",
       "\t2 & 247682.84302150423\\\\\n",
       "\t3 & NaN\\\\\n",
       "\t4 & 100\\\\\n",
       "\t5 & 4775000\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "                VALP\n",
       "1             859691\n",
       "2 247682.84302150423\n",
       "3                NaN\n",
       "4                100\n",
       "5            4775000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collect(select(housing_summary,\"VALP\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it. In this notebook we have shown how to load a CSV file into an SparkSQL data frame using SparkR. We also had a look at the data loaded, mainly to the number of samples loaded, and the data summary."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.2.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}