From d62340e2e1ab338ea661b7ddc504cf55240ffec8 Mon Sep 17 00:00:00 2001 From: Junyang Qian Date: Sat, 27 Aug 2016 23:37:01 -0700 Subject: [PATCH 01/13] Add SparkR vignettes. --- R/pkg/vignettes/people.parquet/._SUCCESS.crc | Bin 0 -> 8 bytes R/pkg/vignettes/people.parquet/_SUCCESS | 0 R/pkg/vignettes/sparkr-vignettes.Rmd | 816 ++++++++++ R/pkg/vignettes/sparkr-vignettes.html | 1435 ++++++++++++++++++ R/pkg/vignettes/sparkr-vignettes.md | 222 +++ 5 files changed, 2473 insertions(+) create mode 100644 R/pkg/vignettes/people.parquet/._SUCCESS.crc create mode 100644 R/pkg/vignettes/people.parquet/_SUCCESS create mode 100644 R/pkg/vignettes/sparkr-vignettes.Rmd create mode 100644 R/pkg/vignettes/sparkr-vignettes.html create mode 100644 R/pkg/vignettes/sparkr-vignettes.md diff --git a/R/pkg/vignettes/people.parquet/._SUCCESS.crc b/R/pkg/vignettes/people.parquet/._SUCCESS.crc new file mode 100644 index 0000000000000000000000000000000000000000..3b7b044936a890cd8d651d349a752d819d71d22c GIT binary patch literal 8 PcmYc;N@ieSU}69O2$TUk literal 0 HcmV?d00001 diff --git a/R/pkg/vignettes/people.parquet/_SUCCESS b/R/pkg/vignettes/people.parquet/_SUCCESS new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd new file mode 100644 index 0000000000000..ff0455f23b0a2 --- /dev/null +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -0,0 +1,816 @@ +--- +title: "SparkR - Practical Guide" +output: + html_document: + theme: united + toc: true + toc_depth: 4 + toc_float: true + highlight: textmate +--- + +## Overview + +SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/). + +## Getting Started + +We begin with an example running on the local machine, trying to provide an overview of the use of SparkR: data ingestion, data processing and machine learning. + +First, let's load and attach the package. +```{r, message=FALSE} +library(SparkR) +``` + + +To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged. + +If you don't have a Spark package on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. + +```{r, eval=FALSE} +install.spark() +``` + +If you have a Spark package, you don't have to install again, but make sure to set `SPARK_HOME` environment variable to let SparkR know where the main Spark package is. + +```{r, eval=FALSE} +Sys.setenv(SPARK_HOME = "/HOME/spark") +``` + +```{r, echo=FALSE} +# Set to your own spark folder if you want to knit this Rmd. +Sys.setenv(SPARK_HOME = "/Users/junyangq/spark/") +``` + +`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any spark packages depended on, etc. We use default settings. It runs in local mode. + +```{r, message=FALSE, warning=FALSE} +sparkR.session() +``` + +The operations in SparkR are centered around an R object class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. + +`SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame, + +```{r} +cars <- cbind(model = rownames(mtcars), mtcars) +carsDF <- createDataFrame(cars) +``` + +We can view the first few rows of the `SparkDataFrame` by `showDF` or `head` function. +```{r} +showDF(carsDF) +``` + +Common data processing operations such as `filter`, `select` are supported on the `SparkDataFrame`. +```{r} +carsSubDF <- select(carsDF, "model", "mpg", "hp") +carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200) +showDF(carsSubDF) +``` + +SparkR can use many common aggregation functions after grouping. + +```{r} +carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) +showDF(carsGPDF) +``` + +The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. +```{r} +carsGP <- collect(carsGPDF) +class(carsGP) +``` + +SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models. + +SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. +```{r} +model <- spark.glm(carsDF, mpg ~ wt + cyl) +``` + +```{r} +summary(model) +``` + +The model can be saved by `write.ml` and loaded back using `read.ml`. +```{r, eval=FALSE} +write.ml(model, path = "/HOME/tmp/mlModel/glmModel") +``` + +In the end, we can stop Spark Session by running +```{r, eval=FALSE} +sparkR.session.stop() +``` + +## Setup + +### Spark Session + +The following Spark driver properties can be set in `sparkConfig`. + +Property Name | Property group | spark-submit equivalent +---------------- | ------------------ | ---------------------- +spark.driver.memory | Application Properties | --driver-memory +spark.driver.extraClassPath | Runtime Environment | --driver-class-path +spark.driver.extraJavaOptions | Runtime Environment | --driver-java-options +spark.driver.extraLibraryPath | Runtime Environment | --driver-library-path + +### Cluster Mode +SparkR can connect to remote Spark clusters. [Cluster Mode Overview](http://spark.apache.org/docs/latest/cluster-overview.html) is a good introduction to different Spark cluster modes. + +When connecting SparkR to a remote Spark cluster, make sure that the Spark version and Hadoop version on the machine match the corresponding versions on the cluster. Current SparkR package is compatible with +```{r, echo=FALSE, tidy = TRUE} +paste("Spark", packageVersion("SparkR")) +``` +It should be used both on the local computer and on the remote cluster. + +To connect, pass the URL of the master node to `sparkR.session`. A complete list can be seen in [Spark Master URLs](http://spark.apache.org/docs/latest/submitting-applications.html#master-urls). +For example, to connect to a local cluster master, we can call + +```{r, eval=FALSE} +sparkR.session(master = "spark://local:7077") +``` + +For YARN cluster, SparkR only supports the client mode. +```{r, eval=FALSE} +sparkR.session(master = "yarn") +``` + + +## Data Import + +### Local Data Frame +The simplest way is to convert a local R data frame into a `SparkDataFrame`. Specifically we can use `as.DataFrame` or `createDataFrame` and pass in the local R data frame to create a `SparkDataFrame`. As an example, the following creates a `SparkDataFrame` based using the `faithful` dataset from R. +```{r} +df <- as.DataFrame(faithful) +head(df) +``` + +### Data Sources +SparkR supports operating on a variety of data sources through the `SparkDataFrame` interface. You can check the Spark SQL programming guide for more [specific options](https://spark.apache.org/docs/latest/sql-programming-guide.html#manually-specifying-options) that are available for the built-in data sources. + +The general method for creating `SparkDataFrame` from data sources is `read.df`. This method takes in the path for the file to load and the type of data source, and the currently active Spark Session will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively and through Spark Packages you can find data source connectors for popular file formats like Avro. These packages can be added with `sparkPackages` parameter when initializing SparkSession using `sparkR.session'.` + +```{r, eval=FALSE} +sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0") +``` + +We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail. + +```{r} +people <- read.df(paste0(Sys.getenv("SPARK_HOME"), + "/examples/src/main/resources/people.json"), "json") +count(people) +head(people) +``` + +SparkR automatically infers the schema from the JSON file. +```{r} +printSchema(people) +``` + +If we want to read multiple JSON files, `read.json` can be used. +```{r} +people <- read.json(paste0(Sys.getenv("SPARK_HOME"), + c("/examples/src/main/resources/people.json", + "/examples/src/main/resources/people.json"))) +count(people) +``` +The data sources API natively supports CSV formatted input files. For more information please refer to SparkR [read.df](https://spark.apache.org/docs/latest/api/R/read.df.html) API documentation. +```{r, eval=FALSE} +df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA") +``` +The data sources API can also be used to save out `SparkDataFrames` into multiple file formats. For example we can save the `SparkDataFrame` from the previous example to a Parquet file using `write.df`. +```{r} +write.df(people, path = "people.parquet", source = "parquet", mode = "overwrite") +``` + +### Hive Tables +You can also create SparkDataFrames from Hive tables. To do this we will need to create a SparkSession with Hive support which can access tables in the Hive MetaStore. Note that Spark should have been built with Hive support and more details can be found in the [SQL programming guide](https://spark.apache.org/docs/latest/sql-programming-guide.html). In SparkR, by default it will attempt to create a SparkSession with Hive support enabled (`enableHiveSupport = TRUE`). + +```{r, eval=FALSE} +sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") + +txtPath <- paste0(Sys.getenv("SPARK_HOME"), "/examples/src/main/resources/kv1.txt") +sqlCMD <- sprintf("LOAD DATA LOCAL INPATH '%s' INTO TABLE src", txtPath) +sql(sqlCMD) + +results <- sql("FROM src SELECT key, value") + +# results is now a SparkDataFrame +head(results) +``` + + +## Data Processing + +**To dplyr users**: SparkR has similar interface as dplyr in data processing. However, some noticeable differences are worth mentioning in the first place. We use `df` to represent a `SparkDataFrame` and `col` to represent the name of column here. + +1. indicate columns. SparkR uses either a character string of the column name or a Column object constructed with `$` to indicate a column. For example, to select `col` in `df`, we can write `select(df, "col")` or `select(df, df$col)`. + +2. describe conditions. In SparkR, the Column object representation can be inserted into the condition directly, or we can use a character string to describe the condition, without referring to the `SparkDataFrame` used. For example, to select rows with value > 1, we can write `filter(df, df$col > 1)` or `filter(df, "col > 1")`. + +Here are more concrete examples. + +dplyr | SparkR +-------- | --------- +`select(mtcars, mpg, hp)` | `select(carsDF, "mpg", "hp")` +`filter(mtcars, mpg > 20, hp > 100)` | `filter(carsDF, carsDF$mpg > 20, carsDF$hp > 100)` + +Other differences will be mentioned in the specific methods. + +We use the `SparkDataFrame` `carsDF` created above. We can get basic information about the `SparkDataFrame`. +```{r} +carsDF +``` + +Print out the schema in tree format. +```{r} +printSchema(carsDF) +``` + +### SparkDataFrame Operations + +#### Selecting rows, columns + +SparkDataFrames support a number of functions to do structured data processing. Here we include some basic examples and a complete list can be found in the [API](https://spark.apache.org/docs/latest/api/R/index.html) docs: + +You can also pass in column name as strings. +```{r} +head(select(carsDF, "mpg")) +``` + +Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins. +```{r} +head(filter(carsDF, carsDF$mpg < 20)) +``` + +#### Grouping, Aggregation + +A common flow of grouping and aggregation is + +1. Use `groupBy` or `group_by` with respect to some grouping variables to create a `GroupedData` object + +2. Feed the `GroupedData` object to `agg` or `summarize` functions, with some provided aggregation functions to compute a number within each group. + +A number of widely used functions are supported to aggregate data after grouping, including `avg`, `countDistinct`, `count`, `first`, `kurtosis`, `last`, `max`, `mean`, `min`, `sd`, `skewness`, `stddev_pop`, `stddev_samp`, `sumDistinct`, `sum`, `var_pop`, `var_samp`, `var`. + +For example we can compute a histogram of the number of cylinders in the `mtcars` dataset as shown below. + +```{r} +numCyl <- summarize(groupBy(carsDF, carsDF$cyl), count = n(carsDF$cyl)) +head(numCyl) +``` + +#### Operating on Columns + +SparkR also provides a number of functions that can directly applied to columns for data processing and during aggregation. The example below shows the use of basic arithmetic functions. + +```{r} +carsDF_km <- carsDF +carsDF_km$kmpg <- carsDF_km$mpg * 1.61 +head(select(carsDF_km, "model", "mpg", "kmpg")) +``` + + +### Window Functions +A window function is a variation of aggregation function. In simple words, + +* aggregation function: `n` to `1` mapping - returns a single value for a group of entries. Examples include `sum`, `count`, `max`. + +* window function: `n` to `n` mapping - returns one value for each entry in the group, but the value may depend on all the entries of the *group*. Examples include `rank`, `lead`, `lag`. + +Formally, the *group* we mentioned is called the Frame. Every input row can have a unique frame associated with it and the output of the window function on that row is based on the rows confined in that frame. + +Window functions are often used in conjunction with the following functions: `windowPartitionBy`, `windowOrderBy`, `partitionBy`, `orderBy`, `over`. It would be easier to look at an example. + +We still use the `mtcars` dataset. The corresponding `SparkDataFrame` is `carsDF`. Suppose for each number of cylinders, we want to calculate the rank of each car in `mpg` within the group. +```{r} +carsSubDF <- select(carsDF, "model", "mpg", "cyl") +ws <- orderBy(windowPartitionBy("cyl"), "mpg") +carsRank <- withColumn(carsSubDF, "rank", over(rank(), ws)) +showDF(carsRank) +``` + +We explain in detail the above steps. + +* `windowPartitionBy` creates a Window Specification object `WindowSpec` that defines the partition. It controls which rows will be in the same partition as the given row. In this case, rows with the same value in `cyl` will be put in the same partition. `orderBy` further defines the ordering - the position a given row is in the partition. The resulting `WindowSpec` is returned as `ws`. + +More Window Specification methods include `rangeBetween`, which can define boundaries of the frame by value, and `rowsBetween`, which can define the boundaries by row indices. + +* `withColumn` appends a Column called `"rank"` to the `SparkDataFrame`. `over` returns a windowing column. The first argument is usually a Column returned by window function(s) such as `rank()`, `lead(carsDF$wt)`. That calculates the corresponding values according to the partitioned-and-ordered table. + +### User-Defined Function + +In SparkR, we support several kinds of User-Defined Functions. + +#### Apply by Partition + +`dapply` can apply a function to each partition of a `SparkDataFrame`. The function to be applied to each partition of the `SparkDataFrame` should have only one parameter, a `data.frame` corresponding to a partition, and the output should be a `data.frame` as well. Schema specifies the row format of the resulting a `SparkDataFrame`. It must match to data types of returned value. See [here](#DataTypes) for mapping between R and Spark. + +We convert `mpg` to `kmpg` (kilometers per gallon). `carsSubDF` is a `SparkDataFrame` with a subset of `carsDF` columns. + +```{r} +carsSubDF <- select(carsDF, "model", "mpg") +schema <- structType(structField("model", "string"), structField("mpg", "double"), + structField("kmpg", "double")) +out <- dapply(carsSubDF, function(x) { x <- cbind(x, x$mpg * 1.61) }, schema) +head(collect(out)) +``` + +Like `dapply`, apply a function to each partition of a `SparkDataFrame` and collect the result back. The output of function should be a `data.frame`. But, Schema is not required to be passed. Note that `dapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. + +```{r} +out <- dapplyCollect( + carsSubDF, + function(x) { + x <- cbind(x, "kmpg" = x$mpg * 1.61) + }) +head(out, 3) +``` + +#### Apply by Group +`gapply` can apply a function to each group of a `SparkDataFrame`. The function is to be applied to each group of the `SparkDataFrame` and should have only two parameters: grouping key and R `data.frame` corresponding to that key. The groups are chosen from `SparkDataFrames` column(s). The output of function should be a `data.frame`. Schema specifies the row format of the resulting `SparkDataFrame`. It must represent R function’s output schema on the basis of Spark data types. The column names of the returned `data.frame` are set by user. See [here](#DataTypes) for mapping between R and Spark. + +```{r} +schema <- structType(structField("cyl", "double"), structField("max_mpg", "double")) +result <- gapply( + carsDF, + "cyl", + function(key, x) { + y <- data.frame(key, max(x$mpg)) + }, + schema) +head(arrange(result, "max_mpg", decreasing = TRUE)) +``` + +Like gapply, `gapplyCollect` applies a function to each partition of a `SparkDataFrame` and collect the result back to R `data.frame`. The output of the function should be a `data.frame`. But, the schema is not required to be passed. Note that `gapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. + +```{r} +result <- gapplyCollect( + carsDF, + "cyl", + function(key, x) { + y <- data.frame(key, max(x$mpg)) + colnames(y) <- c("cyl", "max_mpg") + y + }) +head(result[order(result$max_mpg, decreasing = TRUE), ]) +``` + +#### Distribute Local Functions + +Similar to `lapply` in native R, `spark.lapply` runs a function over a list of elements and distributes the computations with Spark. Applies a function in a manner that is similar to `doParallel` or `lapply` to elements of a list. The results of all the computations should fit in a single machine. If that is not the case they can do something like `df <- createDataFrame(list)` and then use `dapply`. + +```{r} +families <- c("gaussian", "poisson") +train <- function(family) { + model <- glm(mpg ~ hp, mtcars, family = family) + summary(model) +} +``` + +Return a list of model's summaries. +```{r} +model.summaries <- spark.lapply(families, train) +``` + +Print the summary of each model. +```{r} +print(model.summaries) +``` + + +### SQL Queries +A `SparkDataFrame` can also be registered as a temporary view in Spark SQL and that allows you to run SQL queries over its data. The sql function enables applications to run SQL queries programmatically and returns the result as a `SparkDataFrame`. + +```{r} +people <- read.df(paste0(Sys.getenv("SPARK_HOME"), + "/examples/src/main/resources/people.json"), "json") +``` + +Register this SparkDataFrame as a temporary view. + +```{r} +createOrReplaceTempView(people, "people") +``` + +SQL statements can be run by using the sql method. +```{r} +teenagers <- sql("SELECT name FROM people WHERE age >= 13 AND age <= 19") +head(teenagers) +``` + + +## Machine Learning + +SparkR supports the following machine learning models and algorithms. + +* Generalized Linear Model (GLM) + +* Naive Bayes Model + +* $k$-means Clustering + +* Accelerated Failure Time (AFT) Survival Model + +* Gaussian Mixture Model (GMM) + +* Latent Dirichlet Allocation (LDA) + +* Multilayer Perceptron Model + +* Collaborative Filtering with Alternating Least Squares (ALS) + +* Isotonic Regression Model + +More will be added in the future. + +### R Formula + +For most above, SparkR supports **R formula operators**, including `~`, `.`, `:`, `+` and `-` for model fitting. This makes it a similar experience as using R functions. + +### Training and Test Sets + +We can easily split `SparkDataFrame` into random training and test sets by the `randomSplit` function. It returns a list of split `SparkDataFrames` with provided `weights`. We use `carsDF` as an example and want to have about $70%$ training data and $30%$ test data. +```{r} +splitDF_list <- randomSplit(carsDF, c(0.7, 0.3), seed = 0) +carsDF_train <- splitDF_list[[1]] +carsDF_test <- splitDF_list[[2]] +``` + +```{r} +count(carsDF_train) +head(carsDF_train) +``` + +```{r} +count(carsDF_test) +head(carsDF_test) +``` + + +### Models and Algorithms + +#### Generalized Linear Model + +The main function is `spark.glm`. The following families and link functions are supported. The default is gaussian. + +Family | Link Function +------ | --------- +gaussian | identity, log, inverse +binomial | logit, probit, cloglog (complementary log-log) +poisson | log, identity, sqrt +gamma | inverse, identity, log + +There are three ways to specify the `family` argument. + +* Family name as a character string, e.g. `family = "gaussian"`. + +* Family function, e.g. `family = binomial`. + +* Result returned by a family function, e.g. `family = poisson(link = log)` + +For more information regarding the families and their link functions, see the Wikipedia page [Generalized Linear Model](https://en.wikipedia.org/wiki/Generalized_linear_model). + +We use the `mtcars` dataset as an illustration. The corresponding `SparkDataFrame` is `carsDF`. After fitting the model, we print out a summary and see the fitted values by making predictions on the original dataset. We can also pass into a new `SparkDataFrame` of same schema to predict on new data. + +```{r} +gaussianGLM <- spark.glm(carsDF, mpg ~ wt + hp) +summary(gaussianGLM) +``` +When doing prediction, a new column called `prediction` will be appended. Let's look at only a subset of columns here. +```{r} +gaussianFitted <- predict(gaussianGLM, carsDF) +head(select(gaussianFitted, "model", "prediction", "mpg", "wt", "hp")) +``` + +#### Naive Bayes Model + +Naive Bayes model assumes independence among the features. `spark.naiveBayes` fits a [Bernoulli naive Bayes model](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Bernoulli_naive_Bayes) against a SparkDataFrame. The data should be all categorical. These models are often used for document classification. + +```{r} +titanic <- as.data.frame(Titanic) +titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5]) +naiveBayesModel <- spark.naiveBayes(titanicDF, Survived ~ Class + Sex + Age) +summary(naiveBayesModel) +naiveBayesPrediction <- predict(naiveBayesModel, titanicDF) +showDF(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "prediction")) +``` + +#### k-Means Clustering + +`spark.kmeans` fits a $k$-means clustering model against a `SparkDataFrame`. As an unsupervised learning method, we don't need a response variable. Hence, the left hand side of the R formula should be left blank. The clustering is based only on the variables on the right hand side. + +```{r} +kmeansModel <- spark.kmeans(carsDF, ~ mpg + hp + wt, k = 3) +summary(kmeansModel) +kmeansPredictions <- predict(kmeansModel, carsDF) +showDF(select(kmeansPredictions, "model", "mpg", "hp", "wt", "prediction")) +``` + +#### AFT Survival Model +Survival analysis studies the expected duration of time until an event happens, and often the relationship with risk factors or treatment taken on the subject. In contrast to standard regression analysis, survival modeling has to deal with special characteristics in the data including non-negative survival time and censoring. + +Accelerated Failure Time (AFT) model is a parametric survival model for censored data that assumes the effect of a covariate is to accelerate or decelerate the life course of an event by some constant. For more information, refer to the Wikipedia page [AFT Model](https://en.wikipedia.org/wiki/Accelerated_failure_time_model) and the references there. Different from a [Proportional Hazards Model](https://en.wikipedia.org/wiki/Proportional_hazards_model) designed for the same purpose, the AFT model is easier to parallelize because each instance contributes to the objective function independently. +```{r} +library(survival) +ovarianDF <- createDataFrame(ovarian) +aftModel <- spark.survreg(ovarianDF, Surv(futime, fustat) ~ ecog_ps + rx) +summary(aftModel) +aftPredictions <- predict(aftModel, ovarianDF) +head(aftPredictions) +``` + +#### Gaussian Mixture Model +`spark.gaussianMixture` fits multivariate [Gaussian Mixture Model](https://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model) (GMM) against a `SparkDataFrame`. [Expectation-Maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) (EM) is used to approximate the maximum likelihood estimator (MLE) of the model. + +We use a simulated example to demostrate the usage. +```{r} +X1 <- data.frame(V1 = rnorm(4), V2 = rnorm(4)) +X2 <- data.frame(V1 = rnorm(6, 3), V2 = rnorm(6, 4)) +data <- rbind(X1, X2) +df <- createDataFrame(data) +gmmModel <- spark.gaussianMixture(df, ~ V1 + V2, k = 2) +summary(gmmModel) +gmmFitted <- predict(gmmModel, df) +showDF(select(gmmFitted, "V1", "V2", "prediction")) +``` + + +#### Latent Dirichlet Allocation +`spark.lda` fits a [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) model on a `SparkDataFrame`. It is often used in topic modeling in which topics are inferred from a collection of text documents. LDA can be thought of as a clustering algorithm as follows: + +* Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset. + +* Topics and documents both exist in a feature space, where feature vectors are vectors of word counts (bag of words). + +* Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated. + +To use LDA, we need to specify a `features` column in `data` where each entry represents a document. There are two type options for the column: + +* character string: This can be a string of the whole document. It will be parsed automatically. Additional stop words can be added in `customizedStopWords`. + +* libSVM: Each entry is a collection of words and will be processed directly. + +There are several parameters LDA takes for fitting the model. + +* `k`: number of topics (default 10). + +* `maxIter`: maximum iterations (default 20). + +* `optimizer`: optimizer to train an LDA model, "online" (default) uses [online variational inference](https://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf). "em" uses [expectation-maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm). + +* `subsamplingRate`: For `optimizer = "online"`. Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1] (default 0.05). + +* `topicConcentration`: concentration parameter (commonly named beta or eta) for the prior placed on topic distributions over terms, default -1 to set automatically on the Spark side. Use `summary` to retrieve the effective topicConcentration. Only 1-size numeric is accepted. + +* `docConcentration`: concentration parameter (commonly named alpha) for the prior placed on documents distributions over topics (theta), default -1 to set automatically on the Spark side. Use `summary` to retrieve the effective docConcentration. Only 1-size or k-size numeric is accepted. + +* `maxVocabSize`: maximum vocabulary size, default 1 << 18. + +Two more functions are provided for the fitted model. + +* `spark.posterior` returns a `SparkDataFrame` containing a column of posterior probabilities vectors named "topicDistribution". + +* `spark.perplexity` returns the log perplexity of given `SparkDataFrame`, or the log perplexity of the training data if missing argument `data`. + +For more information, see the help document `?spark.lda`. + +Let's look an artificial example. +```{r} +corpus <- data.frame(features = c( + "1 2 6 0 2 3 1 1 0 0 3", + "1 3 0 1 3 0 0 2 0 0 1", + "1 4 1 0 0 4 9 0 1 2 0", + "2 1 0 3 0 0 5 0 2 3 9", + "3 1 1 9 3 0 2 0 0 1 3", + "4 2 0 3 4 5 1 1 1 4 0", + "2 1 0 3 0 0 5 0 2 2 9", + "1 1 1 9 2 1 2 0 0 1 3", + "4 4 0 3 4 2 1 3 0 0 0", + "2 8 2 0 3 0 2 0 2 7 2", + "1 1 1 9 0 2 2 0 0 3 3", + "4 1 0 0 4 5 1 3 0 1 0")) +corpusDF <- createDataFrame(corpus) +model <- spark.lda(data = corpusDF, k = 5, optimizer = "em") +summary(model) +``` + +```{r} +posterior <- spark.posterior(model, corpusDF) +head(posterior) +``` + +```{r} +perplexity <- spark.perplexity(model, corpusDF) +perplexity +``` + + +#### Multilayer Perceptron +Multilayer perceptron classifier (MLPC) is a classifier based on the [feedforward artificial neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network). MLPC consists of multiple layers of nodes. Each layer is fully connected to the next layer in the network. Nodes in the input layer represent the input data. All other nodes map inputs to outputs by a linear combination of the inputs with the node’s weights $w$ and bias $b$ and applying an activation function. This can be written in matrix form for MLPC with $K+1$ layers as follows: +$$ +y(x)=f_K(\ldots f_2(w_2^T f_1(w_1^T x + b_1) + b_2) \ldots + b_K). +$$ + +Nodes in intermediate layers use sigmoid (logistic) function: +$$ +f(z_i) = \frac{1}{1+e^{-z_i}}. +$$ + +Nodes in the output layer use softmax function: +$$ +f(z_i) = \frac{e^{z_i}}{\sum_{k=1}^N e^{z_k}}. +$$ + +The number of nodes $N$ in the output layer corresponds to the number of classes. + +MLPC employs backpropagation for learning the model. We use the logistic loss function for optimization and L-BFGS as an optimization routine. + +`spark.mlp` requires at least two columns in `data`: one named `"label"` and the other one `"features"`. The `"features"` column should be in libSVM-format. According to the description above, there are several additional parameters that can be set: + +* `layers`: integer vector containing the number of nodes for each layer. + +* `solver`: solver parameter, supported options: `"gd"` (minibatch gradient descent) or `"l-bfgs"`. + +* `maxIter`: maximum iteration number. + +* `tol`: convergence tolerance of iterations. + +* `stepSize`: step size for `"gd"`. + +* `seed`: seed parameter for weights initialization. + +#### Collaborative Filtering + +`spark.als` learns latent factors in [collaborative filtering](https://en.wikipedia.org/wiki/Recommender_system#Collaborative_filtering) via [alternating least squares](http://dl.acm.org/citation.cfm?id=1608614). + +There are multiple options that can be configured in `spark.als`, including `rank`, `reg`, `nonnegative`. For a complete list, refer to the help file. + +```{r} +ratings <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0), + list(2, 1, 1.0), list(2, 2, 5.0)) +df <- createDataFrame(ratings, c("user", "item", "rating")) +model <- spark.als(df, "rating", "user", "item", rank = 10, reg = 0.1, nonnegative = TRUE) +``` + +Extract latent factors. +```{r} +stats <- summary(model) +userFactors <- stats$userFactors +itemFactors <- stats$itemFactors +head(userFactors) +head(itemFactors) +``` + +Make predictions. + +```{r} +predicted <- predict(model, df) +showDF(predicted) +``` + +#### Isotonic Regression Model +`spark.isoreg` fits an [Isotonic Regression](https://en.wikipedia.org/wiki/Isotonic_regression) model against a `SparkDataFrame`. It solves a weighted univariate a regression problem under a complete order constraint. Specifically, given a set of real observed responses $y_1, \ldots, y_n$, corresponding real features $x_1, \ldots, x_n$, and optionally positive weights $w_1, \ldots, w_n$, we want to find a monotone (piecewise linear) function $f$ to minimize +$$ +\ell(f) = \sum_{i=1}^n w_i (y_i - f(x_i))^2. +$$ + +There are a few more arguments that may be useful. + +* `weightCol`: a character string specifying the weight column. + +* `isotonic`: logical value indicating whether the output sequence should be isotonic/increasing (`TRUE`) or antitonic/decreasing (`FALSE`). + +* `featureIndex`: the index of the feature on the right hand side of the formula if it is a vector column (default: 0), no effect otherwise. + +We use an artificial example to show the use. + +```{r} +y <- c(3.0, 6.0, 8.0, 5.0, 7.0) +x <- c(1.0, 2.0, 3.5, 3.0, 4.0) +w <- rep(1.0, 5) +data <- data.frame(y = y, x = x, w = w) +df <- createDataFrame(data) +isoregModel <- spark.isoreg(df, y ~ x, weightCol = "w") +isoregFitted <- predict(isoregModel, df) +head(select(isoregFitted, "x", "y", "prediction")) +``` + +In the prediction stage, based on the fitted monotone piecewise function, the rules are: + +* If the prediction input exactly matches a training feature then associated prediction is returned. In case there are multiple predictions with the same feature then one of them is returned. Which one is undefined. + +* If the prediction input is lower or higher than all training features then prediction with lowest or highest feature is returned respectively. In case there are multiple predictions with the same feature then the lowest or highest is returned respectively. + +* If the prediction input falls between two training features then prediction is treated as piecewise linear function and interpolated value is calculated from the predictions of the two closest features. In case there are multiple values with the same feature then the same rules as in previous point are used. + +For example, when the input is $3.2$, the two closest feature values are $3.0$ and $3.5$, then predicted value would be a linear interpolation between the predicted values at $3.0$ and $3.5$. + +```{r} +newDF <- createDataFrame(data.frame(x = c(1.5, 3.2))) +head(predict(isoregModel, newDF)) +``` + +### Model Persistence +The following example shows how to save/load an ML model by SparkR. +```{r} +irisDF <- suppressWarnings(createDataFrame(iris)) +gaussianGLM <- spark.glm(irisDF, Sepal_Length ~ Sepal_Width + Species, family = "gaussian") + +# Save and then load a fitted MLlib model +modelPath <- tempfile(pattern = "ml", fileext = ".tmp") +write.ml(gaussianGLM, modelPath) +gaussianGLM2 <- read.ml(modelPath) + +# Check model summary +summary(gaussianGLM2) + +# Check model prediction +gaussianPredictions <- predict(gaussianGLM2, irisDF) +showDF(gaussianPredictions) + +unlink(modelPath) +``` + + +## Advanced Topics + +### SparkR Object Classes + +There are three main object classes in SparkR you may be working with. + +* `SparkDataFrame`: the central component of SparkR. It is an S4 class representing distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R. It has two slots `sdf` and `env`. + + `sdf` stores a reference to the corresponding Spark Dataset in the Spark JVM backend. + + `env` saves the meta-information of the object such as `isCached`. + +It can be created by data import methods or by transforming an existing `SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data processing functions and feed that into machine learning algorithms. + +* `Column`: an S4 class representing column of `SparkDataFrame`. The slot `jc` saves a reference to the corresponding Column object in the Spark JVM backend. + +It can be obtained from a `SparkDataFrame` by `$` operator, `df$col`. More often, it is used together with other functions, for example, with `select` to select particular columns, with `filter` and constructed conditions to select rows, with aggregation functions to compute aggregate statistics for each group. + +* `GroupedData`: an S4 class representing grouped data created by `groupBy` or by transforming other `GroupedData`. Its `sgd` slot saves a reference to a RelationalGroupedDataset object in the backend. + +This is often an intermediate object with group information and followed up by aggregation operations. + +### Architecture + +A complete description of architecture can be seen in paper [SparkR: Scaling R Programs with Spark](https://people.csail.mit.edu/matei/papers/2016/sigmod_sparkr.pdf), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. + +Under the hood of SparkR is Spark SQL engine. This avoids the overheads of running interpreted R code, and the optimized SQL execution engine in Spark uses structural information about data and computation flow to perform a bunch of optimizations to speed up the computation. + +The main method calls of actual computation happen in the Spark JVM of the driver. We have a socket-based SparkR API that allows us to invoke functions on the JVM from R. We use a SparkR JVM backend that listens on a Netty-based socket server. + +Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and creating new objects. Method invocation can be done in two ways. + +* `invokeJMethod` takes a reference to an existing Java object and a list of arguments to be passed on to the method. + +* `invokeJStatic` takes a class name for static method and a list of arguments to be passed on to the method. + +The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method. + +To create objects, a special method name `init` is used and then similarly the appropriate constructor is invoked with provided arguments. + +Finally, we use a new R class `jobj` that refers to a Java object existing in the backend. These references are tracked on the Java side and are automatically garbage collected when they go out of scope on the R side. + +## Appendix + +### R and Spark Data Types {#DataTypes} + +R | Spark +----------- | ------------- +byte | byte +integer | integer +float | float +double | double +numeric | double +character | string +string | string +binary | binary +raw | binary +logical | boolean +POSIXct | timestamp +POSIXlt | timestamp +Date | date +array | array +list | array +env | map + +## References + +* [Spark Cluster Mode Overview](http://spark.apache.org/docs/latest/cluster-overview.html) + +* [Submitting Spark Applications](http://spark.apache.org/docs/latest/submitting-applications.html) + +* [Machine Learning Library Guide (MLlib)](http://spark.apache.org/docs/latest/ml-guide.html) + +* [SparkR: Scaling R Programs with Spark](SparkR: Scaling R Programs with Spark), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. + + + +```{r, echo=FALSE} +sparkR.session.stop() +``` diff --git a/R/pkg/vignettes/sparkr-vignettes.html b/R/pkg/vignettes/sparkr-vignettes.html new file mode 100644 index 0000000000000..65da24dc04c60 --- /dev/null +++ b/R/pkg/vignettes/sparkr-vignettes.html @@ -0,0 +1,1435 @@ + + + + + + + + + + + + + +SparkR - Practical Guide + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+
+
+
+
+ +
+ + + + + + + +
+

Overview

+

SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using MLlib.

+
+
+

Getting Started

+

We begin with an example running on the local machine, trying to provide an overview of the use of SparkR: data ingestion, data processing and machine learning.

+

First, let’s load and attach the package.

+
library(SparkR)
+

To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged.

+

If you don’t have a Spark package on the computer, you may download it from Apache Spark Website. Alternatively, we provide an easy-to-use function install.spark to complete this process.

+
install.spark()
+

If you have a Spark package, you don’t have to install again, but make sure to set SPARK_HOME environment variable to let SparkR know where the main Spark package is.

+
Sys.setenv(SPARK_HOME = "/HOME/spark")
+

SparkSession is the entry point into SparkR which connects your R program to a Spark cluster. You can create a SparkSession using sparkR.session and pass in options such as the application name, any spark packages depended on, etc. We use default settings. It runs in local mode.

+
sparkR.session()
+
## Launching java with spark-submit command /Users/junyangq/spark//bin/spark-submit   sparkr-shell /var/folders/jh/6pw_r0d51317krg8ftgy53f40000gn/T//RtmpKAtJ8U/backend_portbe3a130d80ff
+
## Java ref type org.apache.spark.sql.SparkSession id 1
+

The operations in SparkR are centered around an R object class called SparkDataFrame. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood.

+

SparkDataFrame can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a SparkDataFrame from a local R data frame,

+
cars <- cbind(model = rownames(mtcars), mtcars)
+carsDF <- createDataFrame(cars)
+

We can view the first few rows of the SparkDataFrame by showDF or head function.

+
showDF(carsDF)
+
## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
+## |              model| mpg|cyl| disp|   hp|drat|   wt| qsec| vs| am|gear|carb|
+## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
+## |          Mazda RX4|21.0|6.0|160.0|110.0| 3.9| 2.62|16.46|0.0|1.0| 4.0| 4.0|
+## |      Mazda RX4 Wag|21.0|6.0|160.0|110.0| 3.9|2.875|17.02|0.0|1.0| 4.0| 4.0|
+## |         Datsun 710|22.8|4.0|108.0| 93.0|3.85| 2.32|18.61|1.0|1.0| 4.0| 1.0|
+## |     Hornet 4 Drive|21.4|6.0|258.0|110.0|3.08|3.215|19.44|1.0|0.0| 3.0| 1.0|
+## |  Hornet Sportabout|18.7|8.0|360.0|175.0|3.15| 3.44|17.02|0.0|0.0| 3.0| 2.0|
+## |            Valiant|18.1|6.0|225.0|105.0|2.76| 3.46|20.22|1.0|0.0| 3.0| 1.0|
+## |         Duster 360|14.3|8.0|360.0|245.0|3.21| 3.57|15.84|0.0|0.0| 3.0| 4.0|
+## |          Merc 240D|24.4|4.0|146.7| 62.0|3.69| 3.19| 20.0|1.0|0.0| 4.0| 2.0|
+## |           Merc 230|22.8|4.0|140.8| 95.0|3.92| 3.15| 22.9|1.0|0.0| 4.0| 2.0|
+## |           Merc 280|19.2|6.0|167.6|123.0|3.92| 3.44| 18.3|1.0|0.0| 4.0| 4.0|
+## |          Merc 280C|17.8|6.0|167.6|123.0|3.92| 3.44| 18.9|1.0|0.0| 4.0| 4.0|
+## |         Merc 450SE|16.4|8.0|275.8|180.0|3.07| 4.07| 17.4|0.0|0.0| 3.0| 3.0|
+## |         Merc 450SL|17.3|8.0|275.8|180.0|3.07| 3.73| 17.6|0.0|0.0| 3.0| 3.0|
+## |        Merc 450SLC|15.2|8.0|275.8|180.0|3.07| 3.78| 18.0|0.0|0.0| 3.0| 3.0|
+## | Cadillac Fleetwood|10.4|8.0|472.0|205.0|2.93| 5.25|17.98|0.0|0.0| 3.0| 4.0|
+## |Lincoln Continental|10.4|8.0|460.0|215.0| 3.0|5.424|17.82|0.0|0.0| 3.0| 4.0|
+## |  Chrysler Imperial|14.7|8.0|440.0|230.0|3.23|5.345|17.42|0.0|0.0| 3.0| 4.0|
+## |           Fiat 128|32.4|4.0| 78.7| 66.0|4.08|  2.2|19.47|1.0|1.0| 4.0| 1.0|
+## |        Honda Civic|30.4|4.0| 75.7| 52.0|4.93|1.615|18.52|1.0|1.0| 4.0| 2.0|
+## |     Toyota Corolla|33.9|4.0| 71.1| 65.0|4.22|1.835| 19.9|1.0|1.0| 4.0| 1.0|
+## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
+## only showing top 20 rows
+

Common data processing operations such as filter, select are supported on the SparkDataFrame.

+
carsSubDF <- select(carsDF, "model", "mpg", "hp")
+carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200)
+showDF(carsSubDF)
+
## +-------------------+----+-----+
+## |              model| mpg|   hp|
+## +-------------------+----+-----+
+## |         Duster 360|14.3|245.0|
+## | Cadillac Fleetwood|10.4|205.0|
+## |Lincoln Continental|10.4|215.0|
+## |  Chrysler Imperial|14.7|230.0|
+## |         Camaro Z28|13.3|245.0|
+## |     Ford Pantera L|15.8|264.0|
+## |      Maserati Bora|15.0|335.0|
+## +-------------------+----+-----+
+

SparkR can use many common aggregation functions after grouping.

+
carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear))
+showDF(carsGPDF)
+
## +----+-----+
+## |gear|count|
+## +----+-----+
+## | 4.0|   12|
+## | 3.0|   15|
+## | 5.0|    5|
+## +----+-----+
+

The results carsDF and carsSubDF are SparkDataFrame objects. To convert back to R data.frame, we can use collect.

+
carsGP <- collect(carsGPDF)
+class(carsGP)
+
## [1] "data.frame"
+

SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call summary to print a summary of the fitted model, predict to make predictions on new data, and write.ml/read.ml to save/load fitted models.

+

SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example.

+
model <- spark.glm(carsDF, mpg ~ wt + cyl)
+
summary(model)
+
## 
+## Deviance Residuals: 
+## (Note: These are approximate quantiles with relative error <= 0.01)
+##     Min       1Q   Median       3Q      Max  
+## -4.2893  -1.7085  -0.4713   1.5729   6.1004  
+## 
+## Coefficients:
+##              Estimate  Std. Error  t value  Pr(>|t|)  
+## (Intercept)  39.686    1.715       23.141   0         
+## wt           -3.191    0.75691     -4.2158  0.00022202
+## cyl          -1.5078   0.41469     -3.636   0.0010643 
+## 
+## (Dispersion parameter for gaussian family taken to be 6.592137)
+## 
+##     Null deviance: 1126.05  on 31  degrees of freedom
+## Residual deviance:  191.17  on 29  degrees of freedom
+## AIC: 156
+## 
+## Number of Fisher Scoring iterations: 1
+

The model can be saved by write.ml and loaded back using read.ml.

+
write.ml(model, path = "/HOME/tmp/mlModel/glmModel")
+

In the end, we can stop Spark Session by running

+
sparkR.session.stop()
+
+
+

Setup

+
+

Spark Session

+

The following Spark driver properties can be set in sparkConfig.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Property NameProperty groupspark-submit equivalent
spark.driver.memoryApplication Properties–driver-memory
spark.driver.extraClassPathRuntime Environment–driver-class-path
spark.driver.extraJavaOptionsRuntime Environment–driver-java-options
spark.driver.extraLibraryPathRuntime Environment–driver-library-path
+
+
+

Cluster Mode

+

SparkR can connect to remote Spark clusters. Cluster Mode Overview is a good introduction to different Spark cluster modes.

+

When connecting SparkR to a remote Spark cluster, make sure that the Spark version and Hadoop version on the machine match the corresponding versions on the cluster. Current SparkR package is compatible with

+
## [1] "Spark 2.0.0"
+

It should be used both on the local computer and on the remote cluster.

+

To connect, pass the URL of the master node to sparkR.session. A complete list can be seen in Spark Master URLs. For example, to connect to a local cluster master, we can call

+
sparkR.session(master = "spark://local:7077")
+

For YARN cluster, SparkR only supports the client mode.

+
sparkR.session(master = "yarn")
+
+
+
+

Data Import

+
+

Local Data Frame

+

The simplest way is to convert a local R data frame into a SparkDataFrame. Specifically we can use as.DataFrame or createDataFrame and pass in the local R data frame to create a SparkDataFrame. As an example, the following creates a SparkDataFrame based using the faithful dataset from R.

+
df <- as.DataFrame(faithful)
+head(df)
+
##   eruptions waiting
+## 1     3.600      79
+## 2     1.800      54
+## 3     3.333      74
+## 4     2.283      62
+## 5     4.533      85
+## 6     2.883      55
+
+
+

Data Sources

+

SparkR supports operating on a variety of data sources through the SparkDataFrame interface. You can check the Spark SQL programming guide for more specific options that are available for the built-in data sources.

+

The general method for creating SparkDataFrame from data sources is read.df. This method takes in the path for the file to load and the type of data source, and the currently active Spark Session will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively and through Spark Packages you can find data source connectors for popular file formats like Avro. These packages can be added with sparkPackages parameter when initializing SparkSession using sparkR.session'.

+
sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0")
+

We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

+
people <- read.df(paste0(Sys.getenv("SPARK_HOME"), 
+                         "/examples/src/main/resources/people.json"), "json")
+count(people)
+
## [1] 3
+
head(people)
+
##   age    name
+## 1  NA Michael
+## 2  30    Andy
+## 3  19  Justin
+

SparkR automatically infers the schema from the JSON file.

+
printSchema(people)
+
## root
+##  |-- age: long (nullable = true)
+##  |-- name: string (nullable = true)
+

If we want to read multiple JSON files, read.json can be used.

+
people <- read.json(paste0(Sys.getenv("SPARK_HOME"),
+                           c("/examples/src/main/resources/people.json",
+                             "/examples/src/main/resources/people.json")))
+count(people)
+
## [1] 6
+

The data sources API natively supports CSV formatted input files. For more information please refer to SparkR read.df API documentation.

+
df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA")
+

The data sources API can also be used to save out SparkDataFrames into multiple file formats. For example we can save the SparkDataFrame from the previous example to a Parquet file using write.df.

+
write.df(people, path = "people.parquet", source = "parquet", mode = "overwrite")
+
+
+

Hive Tables

+

You can also create SparkDataFrames from Hive tables. To do this we will need to create a SparkSession with Hive support which can access tables in the Hive MetaStore. Note that Spark should have been built with Hive support and more details can be found in the SQL programming guide. In SparkR, by default it will attempt to create a SparkSession with Hive support enabled (enableHiveSupport = TRUE).

+
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
+
+txtPath <- paste0(Sys.getenv("SPARK_HOME"), "/examples/src/main/resources/kv1.txt")
+sqlCMD <- sprintf("LOAD DATA LOCAL INPATH '%s' INTO TABLE src", txtPath)
+sql(sqlCMD)
+
+results <- sql("FROM src SELECT key, value")
+
+# results is now a SparkDataFrame
+head(results)
+
+
+
+

Data Processing

+

To dplyr users: SparkR has similar interface as dplyr in data processing. However, some noticeable differences are worth mentioning in the first place. We use df to represent a SparkDataFrame and col to represent the name of column here.

+
    +

  1. indicate columns. SparkR uses either a character string of the column name or a Column object constructed with $ to indicate a column. For example, to select col in df, we can write select(df, "col") or select(df, df$col).

    2. +

  2. describe conditions. In SparkR, the Column object representation can be inserted into the condition directly, or we can use a character string to describe the condition, without referring to the SparkDataFrame used. For example, to select rows with value > 1, we can write filter(df, df$col > 1) or filter(df, "col > 1").

    3. +
+

Here are more concrete examples.

+ ++++ + + + + + + + + + + + + + + + + +
dplyrSparkR
select(mtcars, mpg, hp)select(carsDF, "mpg", "hp")
filter(mtcars, mpg > 20, hp > 100)filter(carsDF, carsDF$mpg > 20, carsDF$hp > 100)
+

Other differences will be mentioned in the specific methods.

+

We use the SparkDataFrame carsDF created above. We can get basic information about the SparkDataFrame.

+
carsDF
+
## SparkDataFrame[model:string, mpg:double, cyl:double, disp:double, hp:double, drat:double, wt:double, qsec:double, vs:double, am:double, gear:double, carb:double]
+

Print out the schema in tree format.

+
printSchema(carsDF)
+
## root
+##  |-- model: string (nullable = true)
+##  |-- mpg: double (nullable = true)
+##  |-- cyl: double (nullable = true)
+##  |-- disp: double (nullable = true)
+##  |-- hp: double (nullable = true)
+##  |-- drat: double (nullable = true)
+##  |-- wt: double (nullable = true)
+##  |-- qsec: double (nullable = true)
+##  |-- vs: double (nullable = true)
+##  |-- am: double (nullable = true)
+##  |-- gear: double (nullable = true)
+##  |-- carb: double (nullable = true)
+
+

SparkDataFrame Operations

+
+

Selecting rows, columns

+

SparkDataFrames support a number of functions to do structured data processing. Here we include some basic examples and a complete list can be found in the API docs:

+

You can also pass in column name as strings.

+
head(select(carsDF, "mpg"))
+
##    mpg
+## 1 21.0
+## 2 21.0
+## 3 22.8
+## 4 21.4
+## 5 18.7
+## 6 18.1
+

Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins.

+
head(filter(carsDF, carsDF$mpg < 20))
+
##               model  mpg cyl  disp  hp drat   wt  qsec vs am gear carb
+## 1 Hornet Sportabout 18.7   8 360.0 175 3.15 3.44 17.02  0  0    3    2
+## 2           Valiant 18.1   6 225.0 105 2.76 3.46 20.22  1  0    3    1
+## 3        Duster 360 14.3   8 360.0 245 3.21 3.57 15.84  0  0    3    4
+## 4          Merc 280 19.2   6 167.6 123 3.92 3.44 18.30  1  0    4    4
+## 5         Merc 280C 17.8   6 167.6 123 3.92 3.44 18.90  1  0    4    4
+## 6        Merc 450SE 16.4   8 275.8 180 3.07 4.07 17.40  0  0    3    3
+
+
+

Grouping, Aggregation

+

A common flow of grouping and aggregation is

+
    +

  1. Use groupBy or group_by with respect to some grouping variables to create a GroupedData object

    2. +

  2. Feed the GroupedData object to agg or summarize functions, with some provided aggregation functions to compute a number within each group.

    3. +
+

A number of widely used functions are supported to aggregate data after grouping, including avg, countDistinct, count, first, kurtosis, last, max, mean, min, sd, skewness, stddev_pop, stddev_samp, sumDistinct, sum, var_pop, var_samp, var.

+

For example we can compute a histogram of the number of cylinders in the mtcars dataset as shown below.

+
numCyl <- summarize(groupBy(carsDF, carsDF$cyl), count = n(carsDF$cyl))
+head(numCyl)
+
##   cyl count
+## 1   8    14
+## 2   4    11
+## 3   6     7
+
+
+

Operating on Columns

+

SparkR also provides a number of functions that can directly applied to columns for data processing and during aggregation. The example below shows the use of basic arithmetic functions.

+
carsDF_km <- carsDF
+carsDF_km$kmpg <- carsDF_km$mpg * 1.61
+head(select(carsDF_km, "model", "mpg", "kmpg"))
+
##               model  mpg   kmpg
+## 1         Mazda RX4 21.0 33.810
+## 2     Mazda RX4 Wag 21.0 33.810
+## 3        Datsun 710 22.8 36.708
+## 4    Hornet 4 Drive 21.4 34.454
+## 5 Hornet Sportabout 18.7 30.107
+## 6           Valiant 18.1 29.141
+
+
+
+

Window Functions

+

A window function is a variation of aggregation function. In simple words,

+
    +

  • aggregation function: n to 1 mapping - returns a single value for a group of entries. Examples include sum, count, max.

    • +

  • window function: n to n mapping - returns one value for each entry in the group, but the value may depend on all the entries of the group. Examples include rank, lead, lag.

    • +
+

Formally, the group we mentioned is called the Frame. Every input row can have a unique frame associated with it and the output of the window function on that row is based on the rows confined in that frame.

+

Window functions are often used in conjunction with the following functions: windowPartitionBy, windowOrderBy, partitionBy, orderBy, over. It would be easier to look at an example.

+

We still use the mtcars dataset. The corresponding SparkDataFrame is carsDF. Suppose for each number of cylinders, we want to calculate the rank of each car in mpg within the group.

+
carsSubDF <- select(carsDF, "model", "mpg", "cyl")
+ws <- orderBy(windowPartitionBy("cyl"), "mpg")
+carsRank <- withColumn(carsSubDF, "rank", over(rank(), ws))
+showDF(carsRank)
+
## +-------------------+----+---+----+
+## |              model| mpg|cyl|rank|
+## +-------------------+----+---+----+
+## | Cadillac Fleetwood|10.4|8.0|   1|
+## |Lincoln Continental|10.4|8.0|   1|
+## |         Camaro Z28|13.3|8.0|   3|
+## |         Duster 360|14.3|8.0|   4|
+## |  Chrysler Imperial|14.7|8.0|   5|
+## |      Maserati Bora|15.0|8.0|   6|
+## |        Merc 450SLC|15.2|8.0|   7|
+## |        AMC Javelin|15.2|8.0|   7|
+## |   Dodge Challenger|15.5|8.0|   9|
+## |     Ford Pantera L|15.8|8.0|  10|
+## |         Merc 450SE|16.4|8.0|  11|
+## |         Merc 450SL|17.3|8.0|  12|
+## |  Hornet Sportabout|18.7|8.0|  13|
+## |   Pontiac Firebird|19.2|8.0|  14|
+## |         Volvo 142E|21.4|4.0|   1|
+## |      Toyota Corona|21.5|4.0|   2|
+## |         Datsun 710|22.8|4.0|   3|
+## |           Merc 230|22.8|4.0|   3|
+## |          Merc 240D|24.4|4.0|   5|
+## |      Porsche 914-2|26.0|4.0|   6|
+## +-------------------+----+---+----+
+## only showing top 20 rows
+

We explain in detail the above steps.

+
    +
  • windowPartitionBy creates a Window Specification object WindowSpec that defines the partition. It controls which rows will be in the same partition as the given row. In this case, rows with the same value in cyl will be put in the same partition. orderBy further defines the ordering - the position a given row is in the partition. The resulting WindowSpec is returned as ws.
    • +
+

More Window Specification methods include rangeBetween, which can define boundaries of the frame by value, and rowsBetween, which can define the boundaries by row indices.

+
    +
  • withColumn appends a Column called "rank" to the SparkDataFrame. over returns a windowing column. The first argument is usually a Column returned by window function(s) such as rank(), lead(carsDF$wt). That calculates the corresponding values according to the partitioned-and-ordered table.
    • +
+
+
+

User-Defined Function

+

In SparkR, we support several kinds of User-Defined Functions.

+
+

Apply by Partition

+

dapply can apply a function to each partition of a SparkDataFrame. The function to be applied to each partition of the SparkDataFrame should have only one parameter, a data.frame corresponding to a partition, and the output should be a data.frame as well. Schema specifies the row format of the resulting a SparkDataFrame. It must match to data types of returned value. See here for mapping between R and Spark.

+

We convert mpg to kmpg (kilometers per gallon). carsSubDF is a SparkDataFrame with a subset of carsDF columns.

+
carsSubDF <- select(carsDF, "model", "mpg")
+schema <- structType(structField("model", "string"), structField("mpg", "double"),
+                     structField("kmpg", "double"))
+out <- dapply(carsSubDF, function(x) { x <- cbind(x, x$mpg * 1.61) }, schema)
+head(collect(out))
+
##               model  mpg   kmpg
+## 1         Mazda RX4 21.0 33.810
+## 2     Mazda RX4 Wag 21.0 33.810
+## 3        Datsun 710 22.8 36.708
+## 4    Hornet 4 Drive 21.4 34.454
+## 5 Hornet Sportabout 18.7 30.107
+## 6           Valiant 18.1 29.141
+

Like dapply, apply a function to each partition of a SparkDataFrame and collect the result back. The output of function should be a data.frame. But, Schema is not required to be passed. Note that dapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory.

+
out <- dapplyCollect(
+         carsSubDF,
+         function(x) {
+           x <- cbind(x, "kmpg" = x$mpg * 1.61)
+         })
+head(out, 3)
+
##           model  mpg   kmpg
+## 1     Mazda RX4 21.0 33.810
+## 2 Mazda RX4 Wag 21.0 33.810
+## 3    Datsun 710 22.8 36.708
+
+
+

Apply by Group

+

gapply can apply a function to each group of a SparkDataFrame. The function is to be applied to each group of the SparkDataFrame and should have only two parameters: grouping key and R data.frame corresponding to that key. The groups are chosen from SparkDataFrames column(s). The output of function should be a data.frame. Schema specifies the row format of the resulting SparkDataFrame. It must represent R function’s output schema on the basis of Spark data types. The column names of the returned data.frame are set by user. See here for mapping between R and Spark.

+
schema <- structType(structField("cyl", "double"), structField("max_mpg", "double"))
+result <- gapply(
+    carsDF,
+    "cyl",
+    function(key, x) {
+        y <- data.frame(key, max(x$mpg))
+    },
+    schema)
+head(arrange(result, "max_mpg", decreasing = TRUE))
+
##   cyl max_mpg
+## 1   4    33.9
+## 2   6    21.4
+## 3   8    19.2
+

Like gapply, gapplyCollect applies a function to each partition of a SparkDataFrame and collect the result back to R data.frame. The output of the function should be a data.frame. But, the schema is not required to be passed. Note that gapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory.

+
result <- gapplyCollect(
+    carsDF,
+    "cyl",
+    function(key, x) {
+         y <- data.frame(key, max(x$mpg))
+        colnames(y) <- c("cyl", "max_mpg")
+        y
+    })
+head(result[order(result$max_mpg, decreasing = TRUE), ])
+
##   cyl max_mpg
+## 2   4    33.9
+## 3   6    21.4
+## 1   8    19.2
+
+
+

Distribute Local Functions

+

Similar to lapply in native R, spark.lapply runs a function over a list of elements and distributes the computations with Spark. Applies a function in a manner that is similar to doParallel or lapply to elements of a list. The results of all the computations should fit in a single machine. If that is not the case they can do something like df <- createDataFrame(list) and then use dapply.

+
families <- c("gaussian", "poisson")
+train <- function(family) {
+  model <- glm(mpg ~ hp, mtcars, family = family)
+  summary(model)
+}
+

Return a list of model’s summaries.

+
model.summaries <- spark.lapply(families, train)
+

Print the summary of each model.

+
print(model.summaries)
+
## [[1]]
+## 
+## Call:
+## glm(formula = mpg ~ hp, family = family, data = mtcars)
+## 
+## Deviance Residuals: 
+##     Min       1Q   Median       3Q      Max  
+## -5.7121  -2.1122  -0.8854   1.5819   8.2360  
+## 
+## Coefficients:
+##             Estimate Std. Error t value Pr(>|t|)    
+## (Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
+## hp          -0.06823    0.01012  -6.742 1.79e-07 ***
+## ---
+## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
+## 
+## (Dispersion parameter for gaussian family taken to be 14.92248)
+## 
+##     Null deviance: 1126.05  on 31  degrees of freedom
+## Residual deviance:  447.67  on 30  degrees of freedom
+## AIC: 181.24
+## 
+## Number of Fisher Scoring iterations: 2
+## 
+## 
+## [[2]]
+## 
+## Call:
+## glm(formula = mpg ~ hp, family = family, data = mtcars)
+## 
+## Deviance Residuals: 
+##     Min       1Q   Median       3Q      Max  
+## -1.4179  -0.4656  -0.1878   0.3935   1.6642  
+## 
+## Coefficients:
+##               Estimate Std. Error z value Pr(>|z|)    
+## (Intercept)  3.5205346  0.0937489  37.553  < 2e-16 ***
+## hp          -0.0037514  0.0006481  -5.788 7.12e-09 ***
+## ---
+## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
+## 
+## (Dispersion parameter for poisson family taken to be 1)
+## 
+##     Null deviance: 54.524  on 31  degrees of freedom
+## Residual deviance: 18.510  on 30  degrees of freedom
+## AIC: Inf
+## 
+## Number of Fisher Scoring iterations: 4
+
+
+
+

SQL Queries

+

A SparkDataFrame can also be registered as a temporary view in Spark SQL and that allows you to run SQL queries over its data. The sql function enables applications to run SQL queries programmatically and returns the result as a SparkDataFrame.

+
people <- read.df(paste0(Sys.getenv("SPARK_HOME"), 
+                         "/examples/src/main/resources/people.json"), "json")
+

Register this SparkDataFrame as a temporary view.

+
createOrReplaceTempView(people, "people")
+

SQL statements can be run by using the sql method.

+
teenagers <- sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
+head(teenagers)
+
##     name
+## 1 Justin
+
+
+
+

Machine Learning

+

SparkR supports the following machine learning models and algorithms.

+
    +

  • Generalized Linear Model (GLM)

    • +

  • Naive Bayes Model

    • +

  • \(k\)-means Clustering

    • +

  • Accelerated Failure Time (AFT) Survival Model

    • +

  • Gaussian Mixture Model (GMM)

    • +

  • Latent Dirichlet Allocation (LDA)

    • +

  • Multilayer Perceptron Model

    • +

  • Collaborative Filtering with Alternating Least Squares (ALS)

    • +

  • Isotonic Regression Model

    • +
+

More will be added in the future.

+
+

R Formula

+

For most above, SparkR supports R formula operators, including ~, ., :, + and - for model fitting. This makes it a similar experience as using R functions.

+
+
+

Training and Test Sets

+

We can easily split SparkDataFrame into random training and test sets by the randomSplit function. It returns a list of split SparkDataFrames with provided weights. We use carsDF as an example and want to have about \(70%\) training data and \(30%\) test data.

+
splitDF_list <- randomSplit(carsDF, c(0.7, 0.3), seed = 0)
+carsDF_train <- splitDF_list[[1]]
+carsDF_test <- splitDF_list[[2]]
+
count(carsDF_train)
+
## [1] 21
+
head(carsDF_train)
+
##                model  mpg cyl  disp  hp drat    wt  qsec vs am gear carb
+## 1 Cadillac Fleetwood 10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
+## 2         Camaro Z28 13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
+## 3         Duster 360 14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
+## 4           Fiat 128 32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
+## 5          Fiat X1-9 27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
+## 6     Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
+
count(carsDF_test)
+
## [1] 11
+
head(carsDF_test)
+
##               model  mpg cyl disp  hp drat    wt  qsec vs am gear carb
+## 1       AMC Javelin 15.2   8  304 150 3.15 3.435 17.30  0  0    3    2
+## 2 Chrysler Imperial 14.7   8  440 230 3.23 5.345 17.42  0  0    3    4
+## 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
+## 4  Dodge Challenger 15.5   8  318 150 2.76 3.520 16.87  0  0    3    2
+## 5      Ferrari Dino 19.7   6  145 175 3.62 2.770 15.50  0  1    5    6
+## 6     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
+
+
+

Models and Algorithms

+
+

Generalized Linear Model

+

The main function is spark.glm. The following families and link functions are supported. The default is gaussian.

+ + + + + + + + + + + + + + + + + + + + + + + + + +
FamilyLink Function
gaussianidentity, log, inverse
binomiallogit, probit, cloglog (complementary log-log)
poissonlog, identity, sqrt
gammainverse, identity, log
+

There are three ways to specify the family argument.

+
    +

  • Family name as a character string, e.g. family = "gaussian".

    • +

  • Family function, e.g. family = binomial.

    • +

  • Result returned by a family function, e.g. family = poisson(link = log)

    • +
+

For more information regarding the families and their link functions, see the Wikipedia page Generalized Linear Model.

+

We use the mtcars dataset as an illustration. The corresponding SparkDataFrame is carsDF. After fitting the model, we print out a summary and see the fitted values by making predictions on the original dataset. We can also pass into a new SparkDataFrame of same schema to predict on new data.

+
gaussianGLM <- spark.glm(carsDF, mpg ~ wt + hp)
+summary(gaussianGLM)
+
## 
+## Deviance Residuals: 
+## (Note: These are approximate quantiles with relative error <= 0.01)
+##     Min       1Q   Median       3Q      Max  
+## -3.9410  -1.6499  -0.3267   1.0373   5.8538  
+## 
+## Coefficients:
+##              Estimate   Std. Error  t value  Pr(>|t|)  
+## (Intercept)  37.227     1.5988      23.285   0         
+## wt           -3.8778    0.63273     -6.1287  1.1196e-06
+## hp           -0.031773  0.0090297   -3.5187  0.0014512 
+## 
+## (Dispersion parameter for gaussian family taken to be 6.725785)
+## 
+##     Null deviance: 1126.05  on 31  degrees of freedom
+## Residual deviance:  195.05  on 29  degrees of freedom
+## AIC: 156.7
+## 
+## Number of Fisher Scoring iterations: 1
+

When doing prediction, a new column called prediction will be appended. Let’s look at only a subset of columns here.

+
gaussianFitted <- predict(gaussianGLM, carsDF)
+head(select(gaussianFitted, "model", "prediction", "mpg", "wt", "hp"))
+
##               model prediction  mpg    wt  hp
+## 1         Mazda RX4   23.57233 21.0 2.620 110
+## 2     Mazda RX4 Wag   22.58348 21.0 2.875 110
+## 3        Datsun 710   25.27582 22.8 2.320  93
+## 4    Hornet 4 Drive   21.26502 21.4 3.215 110
+## 5 Hornet Sportabout   18.32727 18.7 3.440 175
+## 6           Valiant   20.47382 18.1 3.460 105
+
+
+

Naive Bayes Model

+

Naive Bayes model assumes independence among the features. spark.naiveBayes fits a Bernoulli naive Bayes model against a SparkDataFrame. The data should be all categorical. These models are often used for document classification.

+
titanic <- as.data.frame(Titanic)
+titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5])
+naiveBayesModel <- spark.naiveBayes(titanicDF, Survived ~ Class + Sex + Age)
+summary(naiveBayesModel)
+
## $apriori
+##            Yes        No
+## [1,] 0.5769231 0.4230769
+## 
+## $tables
+##     Class_3rd Class_1st Class_2nd Sex_Male Age_Adult
+## Yes 0.3125    0.3125    0.3125    0.5      0.5625   
+## No  0.4166667 0.25      0.25      0.5      0.75
+
naiveBayesPrediction <- predict(naiveBayesModel, titanicDF)
+showDF(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "prediction"))
+
## +-----+------+-----+--------+----------+
+## |Class|   Sex|  Age|Survived|prediction|
+## +-----+------+-----+--------+----------+
+## |  3rd|  Male|Child|      No|       Yes|
+## |  3rd|Female|Child|      No|       Yes|
+## |  1st|  Male|Adult|      No|       Yes|
+## |  2nd|  Male|Adult|      No|       Yes|
+## |  3rd|  Male|Adult|      No|        No|
+## | Crew|  Male|Adult|      No|       Yes|
+## |  1st|Female|Adult|      No|       Yes|
+## |  2nd|Female|Adult|      No|       Yes|
+## |  3rd|Female|Adult|      No|        No|
+## | Crew|Female|Adult|      No|       Yes|
+## |  1st|  Male|Child|     Yes|       Yes|
+## |  2nd|  Male|Child|     Yes|       Yes|
+## |  3rd|  Male|Child|     Yes|       Yes|
+## |  1st|Female|Child|     Yes|       Yes|
+## |  2nd|Female|Child|     Yes|       Yes|
+## |  3rd|Female|Child|     Yes|       Yes|
+## |  1st|  Male|Adult|     Yes|       Yes|
+## |  2nd|  Male|Adult|     Yes|       Yes|
+## |  3rd|  Male|Adult|     Yes|        No|
+## | Crew|  Male|Adult|     Yes|       Yes|
+## +-----+------+-----+--------+----------+
+## only showing top 20 rows
+
+
+

k-Means Clustering

+

spark.kmeans fits a \(k\)-means clustering model against a SparkDataFrame. As an unsupervised learning method, we don’t need a response variable. Hence, the left hand side of the R formula should be left blank. The clustering is based only on the variables on the right hand side.

+
kmeansModel <- spark.kmeans(carsDF, ~ mpg + hp + wt, k = 3)
+summary(kmeansModel)
+
## $coefficients
+##   mpg      hp       wt      
+## 1 24.22353 93.52941 2.599588
+## 2 14.62    263.8    3.899   
+## 3 15.8     178.5    3.9264  
+## 
+## $size
+## $size[[1]]
+## [1] 17
+## 
+## $size[[2]]
+## [1] 5
+## 
+## $size[[3]]
+## [1] 10
+## 
+## 
+## $cluster
+## SparkDataFrame[prediction:int]
+## 
+## $is.loaded
+## [1] FALSE
+
kmeansPredictions <- predict(kmeansModel, carsDF)
+showDF(select(kmeansPredictions, "model", "mpg", "hp", "wt", "prediction"))
+
## +-------------------+----+-----+-----+----------+
+## |              model| mpg|   hp|   wt|prediction|
+## +-------------------+----+-----+-----+----------+
+## |          Mazda RX4|21.0|110.0| 2.62|         0|
+## |      Mazda RX4 Wag|21.0|110.0|2.875|         0|
+## |         Datsun 710|22.8| 93.0| 2.32|         0|
+## |     Hornet 4 Drive|21.4|110.0|3.215|         0|
+## |  Hornet Sportabout|18.7|175.0| 3.44|         2|
+## |            Valiant|18.1|105.0| 3.46|         0|
+## |         Duster 360|14.3|245.0| 3.57|         1|
+## |          Merc 240D|24.4| 62.0| 3.19|         0|
+## |           Merc 230|22.8| 95.0| 3.15|         0|
+## |           Merc 280|19.2|123.0| 3.44|         0|
+## |          Merc 280C|17.8|123.0| 3.44|         0|
+## |         Merc 450SE|16.4|180.0| 4.07|         2|
+## |         Merc 450SL|17.3|180.0| 3.73|         2|
+## |        Merc 450SLC|15.2|180.0| 3.78|         2|
+## | Cadillac Fleetwood|10.4|205.0| 5.25|         2|
+## |Lincoln Continental|10.4|215.0|5.424|         2|
+## |  Chrysler Imperial|14.7|230.0|5.345|         1|
+## |           Fiat 128|32.4| 66.0|  2.2|         0|
+## |        Honda Civic|30.4| 52.0|1.615|         0|
+## |     Toyota Corolla|33.9| 65.0|1.835|         0|
+## +-------------------+----+-----+-----+----------+
+## only showing top 20 rows
+
+
+

AFT Survival Model

+

Survival analysis studies the expected duration of time until an event happens, and often the relationship with risk factors or treatment taken on the subject. In contrast to standard regression analysis, survival modeling has to deal with special characteristics in the data including non-negative survival time and censoring.

+

Accelerated Failure Time (AFT) model is a parametric survival model for censored data that assumes the effect of a covariate is to accelerate or decelerate the life course of an event by some constant. For more information, refer to the Wikipedia page AFT Model and the references there. Different from a Proportional Hazards Model designed for the same purpose, the AFT model is easier to parallelize because each instance contributes to the objective function independently.

+
library(survival)
+ovarianDF <- createDataFrame(ovarian)
+
## Warning in FUN(X[[i]], ...): Use resid_ds instead of resid.ds as column
+## name
+
## Warning in FUN(X[[i]], ...): Use ecog_ps instead of ecog.ps as column name
+
aftModel <- spark.survreg(ovarianDF, Surv(futime, fustat) ~ ecog_ps + rx)
+summary(aftModel)
+
## $coefficients
+##                  Value
+## (Intercept)  6.8966930
+## ecog_ps     -0.3850426
+## rx           0.5286457
+## Log(scale)  -0.1234418
+
aftPredictions <- predict(aftModel, ovarianDF)
+head(aftPredictions)
+
##   futime fustat     age resid_ds rx ecog_ps label prediction
+## 1     59      1 72.3315        2  1       1    59  1141.7256
+## 2    115      1 74.4932        2  1       1   115  1141.7256
+## 3    156      1 66.4658        2  1       2   156   776.8548
+## 4    421      0 53.3644        2  2       1   421  1937.0893
+## 5    431      1 50.3397        2  1       1   431  1141.7256
+## 6    448      0 56.4301        1  1       2   448   776.8548
+
+
+

Gaussian Mixture Model

+

spark.gaussianMixture fits multivariate Gaussian Mixture Model (GMM) against a SparkDataFrame. Expectation-Maximization (EM) is used to approximate the maximum likelihood estimator (MLE) of the model.

+

We use a simulated example to demostrate the usage.

+
X1 <- data.frame(V1 = rnorm(4), V2 = rnorm(4))
+X2 <- data.frame(V1 = rnorm(6, 3), V2 = rnorm(6, 4))
+data <- rbind(X1, X2)
+df <- createDataFrame(data)
+gmmModel <- spark.gaussianMixture(df, ~ V1 + V2, k = 2)
+summary(gmmModel)
+
## $lambda
+## [1] 0.4000004 0.5999996
+## 
+## $mu
+## $mu[[1]]
+## [1] 0.8159244 0.4405515
+## 
+## $mu[[2]]
+## [1] 3.034007 4.475828
+## 
+## 
+## $sigma
+## $sigma[[1]]
+##      [,1]        [,2]       
+## [1,] 0.4763343   -0.09395206
+## [2,] -0.09395206 0.8492281  
+## 
+## $sigma[[2]]
+##      [,1]       [,2]      
+## [1,] 1.116189   -0.3408798
+## [2,] -0.3408798 0.3243061 
+## 
+## 
+## $posterior
+## SparkDataFrame[posterior:array<double>]
+## 
+## $is.loaded
+## [1] FALSE
+
gmmFitted <- predict(gmmModel, df)
+showDF(select(gmmFitted, "V1", "V2", "prediction"))
+
## +-------------------+-------------------+----------+
+## |                 V1|                 V2|prediction|
+## +-------------------+-------------------+----------+
+## | 0.7212924990296785| 0.6804778552461263|         0|
+## | 1.7023597530257268|-0.7515664802963296|         0|
+## |-0.2112969944430051|0.06082658268951577|         0|
+## |  1.051335125612143|  1.772454161411131|         0|
+## | 3.4068107905235134| 5.0720086203141435|         1|
+## | 2.4434779156754383| 4.3827744683163985|         1|
+## |  5.099806471262699| 3.8553621051014937|         1|
+## | 2.6660451400733596|  4.336997228018436|         1|
+## | 1.6976557727358053|  5.360445392662975|         1|
+## | 2.8902452466769284| 3.8473802024470407|         1|
+## +-------------------+-------------------+----------+
+
+
+

Latent Dirichlet Allocation

+

spark.lda fits a Latent Dirichlet Allocation model on a SparkDataFrame. It is often used in topic modeling in which topics are inferred from a collection of text documents. LDA can be thought of as a clustering algorithm as follows:

+
    +

  • Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset.

    • +

  • Topics and documents both exist in a feature space, where feature vectors are vectors of word counts (bag of words).

    • +

  • Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated.

    • +
+

To use LDA, we need to specify a features column in data where each entry represents a document. There are two type options for the column:

+
    +

  • character string: This can be a string of the whole document. It will be parsed automatically. Additional stop words can be added in customizedStopWords.

    • +

  • libSVM: Each entry is a collection of words and will be processed directly.

    • +
+

There are several parameters LDA takes for fitting the model.

+
    +

  • k: number of topics (default 10).

    • +

  • maxIter: maximum iterations (default 20).

    • +

  • optimizer: optimizer to train an LDA model, “online” (default) uses online variational inference. “em” uses expectation-maximization.

    • +

  • subsamplingRate: For optimizer = "online". Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1] (default 0.05).

    • +

  • topicConcentration: concentration parameter (commonly named beta or eta) for the prior placed on topic distributions over terms, default -1 to set automatically on the Spark side. Use summary to retrieve the effective topicConcentration. Only 1-size numeric is accepted.

    • +

  • docConcentration: concentration parameter (commonly named alpha) for the prior placed on documents distributions over topics (theta), default -1 to set automatically on the Spark side. Use summary to retrieve the effective docConcentration. Only 1-size or k-size numeric is accepted.

    • +

  • maxVocabSize: maximum vocabulary size, default 1 << 18.

    • +
+

Two more functions are provided for the fitted model.

+
    +

  • spark.posterior returns a SparkDataFrame containing a column of posterior probabilities vectors named “topicDistribution”.

    • +

  • spark.perplexity returns the log perplexity of given SparkDataFrame, or the log perplexity of the training data if missing argument data.

    • +
+

For more information, see the help document ?spark.lda.

+

Let’s look an artificial example.

+
corpus <- data.frame(features = c(
+  "1 2 6 0 2 3 1 1 0 0 3",
+  "1 3 0 1 3 0 0 2 0 0 1",
+  "1 4 1 0 0 4 9 0 1 2 0",
+  "2 1 0 3 0 0 5 0 2 3 9",
+  "3 1 1 9 3 0 2 0 0 1 3",
+  "4 2 0 3 4 5 1 1 1 4 0",
+  "2 1 0 3 0 0 5 0 2 2 9",
+  "1 1 1 9 2 1 2 0 0 1 3",
+  "4 4 0 3 4 2 1 3 0 0 0",
+  "2 8 2 0 3 0 2 0 2 7 2",
+  "1 1 1 9 0 2 2 0 0 3 3",
+  "4 1 0 0 4 5 1 3 0 1 0"))
+corpusDF <- createDataFrame(corpus)
+model <- spark.lda(data = corpusDF, k = 5, optimizer = "em")
+summary(model)
+
## $docConcentration
+## [1] 0.1666620 0.2475245 0.1666316 0.1666351 0.1680889
+## 
+## $topicConcentration
+## [1] 0.2
+## 
+## $logLikelihood
+## [1] -318.7387
+## 
+## $logPerplexity
+## [1] 2.414687
+## 
+## $isDistributed
+## [1] FALSE
+## 
+## $vocabSize
+## [1] 10
+## 
+## $topics
+## SparkDataFrame[topic:int, term:array<string>, termWeights:array<double>]
+## 
+## $vocabulary
+##  [1] "0" "1" "2" "3" "4" "9" "5" "8" "7" "6"
+
posterior <- spark.posterior(model, corpusDF)
+head(posterior)
+
##                features
+## 1 1 2 6 0 2 3 1 1 0 0 3
+## 2 1 3 0 1 3 0 0 2 0 0 1
+## 3 1 4 1 0 0 4 9 0 1 2 0
+## 4 2 1 0 3 0 0 5 0 2 3 9
+## 5 3 1 1 9 3 0 2 0 0 1 3
+## 6 4 2 0 3 4 5 1 1 1 4 0
+##                                            topicDistribution
+## 1 0.01409431, 0.94353632, 0.01407450, 0.01409065, 0.01420422
+## 2 0.01402469, 0.94375465, 0.01402782, 0.01402870, 0.01416414
+## 3 0.01406876, 0.94356690, 0.01405973, 0.01405508, 0.01424953
+## 4 0.01404847, 0.94367008, 0.01405272, 0.01404966, 0.01417907
+## 5 0.01403980, 0.94370443, 0.01404162, 0.01404285, 0.01417130
+## 6 0.01409286, 0.94340534, 0.01409602, 0.01408882, 0.01431697
+
perplexity <- spark.perplexity(model, corpusDF)
+perplexity
+
## [1] 2.414687
+
+
+

Multilayer Perceptron

+

Multilayer perceptron classifier (MLPC) is a classifier based on the feedforward artificial neural network. MLPC consists of multiple layers of nodes. Each layer is fully connected to the next layer in the network. Nodes in the input layer represent the input data. All other nodes map inputs to outputs by a linear combination of the inputs with the node’s weights \(w\) and bias \(b\) and applying an activation function. This can be written in matrix form for MLPC with \(K+1\) layers as follows: \[ +y(x)=f_K(\ldots f_2(w_2^T f_1(w_1^T x + b_1) + b_2) \ldots + b_K). +\]

+

Nodes in intermediate layers use sigmoid (logistic) function: \[ +f(z_i) = \frac{1}{1+e^{-z_i}}. +\]

+

Nodes in the output layer use softmax function: \[ +f(z_i) = \frac{e^{z_i}}{\sum_{k=1}^N e^{z_k}}. +\]

+

The number of nodes \(N\) in the output layer corresponds to the number of classes.

+

MLPC employs backpropagation for learning the model. We use the logistic loss function for optimization and L-BFGS as an optimization routine.

+

spark.mlp requires at least two columns in data: one named "label" and the other one "features". The "features" column should be in libSVM-format. According to the description above, there are several additional parameters that can be set:

+
    +

  • layers: integer vector containing the number of nodes for each layer.

    • +

  • solver: solver parameter, supported options: "gd" (minibatch gradient descent) or "l-bfgs".

    • +

  • maxIter: maximum iteration number.

    • +

  • tol: convergence tolerance of iterations.

    • +

  • stepSize: step size for "gd".

    • +

  • seed: seed parameter for weights initialization.

    • +
+
+
+

Collaborative Filtering

+

spark.als learns latent factors in collaborative filtering via alternating least squares.

+

There are multiple options that can be configured in spark.als, including rank, reg, nonnegative. For a complete list, refer to the help file.

+
ratings <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0),
+                list(2, 1, 1.0), list(2, 2, 5.0))
+df <- createDataFrame(ratings, c("user", "item", "rating"))
+model <- spark.als(df, "rating", "user", "item", rank = 10, reg = 0.1, nonnegative = TRUE)
+

Extract latent factors.

+
stats <- summary(model)
+userFactors <- stats$userFactors
+itemFactors <- stats$itemFactors
+head(userFactors)
+
##   id
+## 1  0
+## 2  1
+## 3  2
+##                                                                                                                           features
+## 1           0.36135060, 0.00000000, 0.14246520, 0.79222524, 0.25852802, 0.35938185, 1.02794230, 0.00000000, 0.40629458, 0.03039724
+## 2                     0.6459087, 0.0000000, 0.4240893, 0.5433040, 0.5990543, 0.2464622, 0.7952952, 0.0000000, 1.0376287, 0.3610190
+## 3 0.147648051, 0.000000000, 0.862470269, 0.008587426, 0.171784312, 0.003895568, 0.024497632, 0.000000000, 0.316428304, 1.357668996
+
head(itemFactors)
+
##   id
+## 1  0
+## 2  1
+## 3  2
+##                                                                                                                 features
+## 1 0.65061080, 0.00000000, 0.26098290, 1.38994920, 0.47113958, 0.63053089, 1.80735207, 0.00000000, 0.74440372, 0.06321633
+## 2           0.5822609, 0.0000000, 0.3250861, 0.4984071, 0.5374193, 0.2260954, 0.7286831, 0.0000000, 0.9294589, 0.2301433
+## 3           0.6116845, 0.0000000, 1.5209249, 0.3454843, 0.6182634, 0.1567241, 0.5231636, 0.0000000, 1.0985043, 2.2063246
+

Make predictions.

+
predicted <- predict(model, df)
+showDF(predicted)
+
## +----+----+------+----------+
+## |user|item|rating|prediction|
+## +----+----+------+----------+
+## | 1.0| 1.0|   3.0| 2.7894442|
+## | 2.0| 1.0|   1.0| 1.0882444|
+## | 0.0| 1.0|   2.0| 2.0054312|
+## | 1.0| 2.0|   4.0| 3.9892373|
+## | 2.0| 2.0|   5.0| 4.8677244|
+## | 0.0| 0.0|   4.0| 3.8840594|
+## +----+----+------+----------+
+
+
+

Isotonic Regression Model

+

spark.isoreg fits an Isotonic Regression model against a SparkDataFrame. It solves a weighted univariate a regression problem under a complete order constraint. Specifically, given a set of real observed responses \(y_1, \ldots, y_n\), corresponding real features \(x_1, \ldots, x_n\), and optionally positive weights \(w_1, \ldots, w_n\), we want to find a monotone (piecewise linear) function \(f\) to minimize \[ +\ell(f) = \sum_{i=1}^n w_i (y_i - f(x_i))^2. +\]

+

There are a few more arguments that may be useful.

+
    +

  • weightCol: a character string specifying the weight column.

    • +

  • isotonic: logical value indicating whether the output sequence should be isotonic/increasing (TRUE) or antitonic/decreasing (FALSE).

    • +

  • featureIndex: the index of the feature on the right hand side of the formula if it is a vector column (default: 0), no effect otherwise.

    • +
+

We use an artificial example to show the use.

+
y <- c(3.0, 6.0, 8.0, 5.0, 7.0)
+x <- c(1.0, 2.0, 3.5, 3.0, 4.0)
+w <- rep(1.0, 5)
+data <- data.frame(y = y, x = x, w = w)
+df <- createDataFrame(data)
+isoregModel <- spark.isoreg(df, y ~ x, weightCol = "w")
+isoregFitted <- predict(isoregModel, df)
+head(select(isoregFitted, "x", "y", "prediction"))
+
##     x y prediction
+## 1 1.0 3        3.0
+## 2 2.0 6        5.5
+## 3 3.5 8        7.5
+## 4 3.0 5        5.5
+## 5 4.0 7        7.5
+

In the prediction stage, based on the fitted monotone piecewise function, the rules are:

+
    +

  • If the prediction input exactly matches a training feature then associated prediction is returned. In case there are multiple predictions with the same feature then one of them is returned. Which one is undefined.

    • +

  • If the prediction input is lower or higher than all training features then prediction with lowest or highest feature is returned respectively. In case there are multiple predictions with the same feature then the lowest or highest is returned respectively.

    • +

  • If the prediction input falls between two training features then prediction is treated as piecewise linear function and interpolated value is calculated from the predictions of the two closest features. In case there are multiple values with the same feature then the same rules as in previous point are used.

    • +
+

For example, when the input is \(3.2\), the two closest feature values are \(3.0\) and \(3.5\), then predicted value would be a linear interpolation between the predicted values at \(3.0\) and \(3.5\).

+
newDF <- createDataFrame(data.frame(x = c(1.5, 3.2)))
+head(predict(isoregModel, newDF))
+
##     x prediction
+## 1 1.5       4.25
+## 2 3.2       6.30
+
+
+
+

Model Persistence

+

The following example shows how to save/load an ML model by SparkR.

+
irisDF <- suppressWarnings(createDataFrame(iris))
+gaussianGLM <- spark.glm(irisDF, Sepal_Length ~ Sepal_Width + Species, family = "gaussian")
+
+# Save and then load a fitted MLlib model
+modelPath <- tempfile(pattern = "ml", fileext = ".tmp")
+write.ml(gaussianGLM, modelPath)
+gaussianGLM2 <- read.ml(modelPath)
+
+# Check model summary
+summary(gaussianGLM2)
+
## 
+## Saved-loaded model does not support output 'Deviance Residuals'.
+## 
+## Coefficients:
+##                     Estimate  Std. Error  t value  Pr(>|t|)  
+## (Intercept)         2.2514    0.36975     6.0889   9.5681e-09
+## Sepal_Width         0.80356   0.10634     7.5566   4.1873e-12
+## Species_versicolor  1.4587    0.11211     13.012   0         
+## Species_virginica   1.9468    0.10001     19.465   0         
+## 
+## (Dispersion parameter for gaussian family taken to be 0.1918059)
+## 
+##     Null deviance: 102.168  on 149  degrees of freedom
+## Residual deviance:  28.004  on 146  degrees of freedom
+## AIC: 183.9
+## 
+## Number of Fisher Scoring iterations: 1
+
# Check model prediction
+gaussianPredictions <- predict(gaussianGLM2, irisDF)
+showDF(gaussianPredictions)
+
## +------------+-----------+------------+-----------+-------+-----+------------------+
+## |Sepal_Length|Sepal_Width|Petal_Length|Petal_Width|Species|label|        prediction|
+## +------------+-----------+------------+-----------+-------+-----+------------------+
+## |         5.1|        3.5|         1.4|        0.2| setosa|  5.1| 5.063856384860281|
+## |         4.9|        3.0|         1.4|        0.2| setosa|  4.9| 4.662075934441678|
+## |         4.7|        3.2|         1.3|        0.2| setosa|  4.7|  4.82278811460912|
+## |         4.6|        3.1|         1.5|        0.2| setosa|  4.6|4.7424320245253995|
+## |         5.0|        3.6|         1.4|        0.2| setosa|  5.0| 5.144212474944002|
+## |         5.4|        3.9|         1.7|        0.4| setosa|  5.4| 5.385280745195163|
+## |         4.6|        3.4|         1.4|        0.3| setosa|  4.6|  4.98350029477656|
+## |         5.0|        3.4|         1.5|        0.2| setosa|  5.0|  4.98350029477656|
+## |         4.4|        2.9|         1.4|        0.2| setosa|  4.4| 4.581719844357957|
+## |         4.9|        3.1|         1.5|        0.1| setosa|  4.9|4.7424320245253995|
+## |         5.4|        3.7|         1.5|        0.2| setosa|  5.4| 5.224568565027722|
+## |         4.8|        3.4|         1.6|        0.2| setosa|  4.8|  4.98350029477656|
+## |         4.8|        3.0|         1.4|        0.1| setosa|  4.8| 4.662075934441678|
+## |         4.3|        3.0|         1.1|        0.1| setosa|  4.3| 4.662075934441678|
+## |         5.8|        4.0|         1.2|        0.2| setosa|  5.8| 5.465636835278884|
+## |         5.7|        4.4|         1.5|        0.4| setosa|  5.7|5.7870611956137665|
+## |         5.4|        3.9|         1.3|        0.4| setosa|  5.4| 5.385280745195163|
+## |         5.1|        3.5|         1.4|        0.3| setosa|  5.1| 5.063856384860281|
+## |         5.7|        3.8|         1.7|        0.3| setosa|  5.7| 5.304924655111442|
+## |         5.1|        3.8|         1.5|        0.3| setosa|  5.1| 5.304924655111442|
+## +------------+-----------+------------+-----------+-------+-----+------------------+
+## only showing top 20 rows
+
unlink(modelPath)
+
+
+
+

Advanced Topics

+
+

SparkR Object Classes

+

There are three main object classes in SparkR you may be working with.

+
    +
  • SparkDataFrame: the central component of SparkR. It is an S4 class representing distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R. It has two slots sdf and env. +
      +
    • sdf stores a reference to the corresponding Spark Dataset in the Spark JVM backend.
      • +
    • env saves the meta-information of the object such as isCached.
      • +
    • +
+

It can be created by data import methods or by transforming an existing SparkDataFrame. We can manipulate SparkDataFrame by numerous data processing functions and feed that into machine learning algorithms.

+
    +
  • Column: an S4 class representing column of SparkDataFrame. The slot jc saves a reference to the corresponding Column object in the Spark JVM backend.
    • +
+

It can be obtained from a SparkDataFrame by $ operator, df$col. More often, it is used together with other functions, for example, with select to select particular columns, with filter and constructed conditions to select rows, with aggregation functions to compute aggregate statistics for each group.

+
    +
  • GroupedData: an S4 class representing grouped data created by groupBy or by transforming other GroupedData. Its sgd slot saves a reference to a RelationalGroupedDataset object in the backend.
    • +
+

This is often an intermediate object with group information and followed up by aggregation operations.

+
+
+

Architecture

+

A complete description of architecture can be seen in paper SparkR: Scaling R Programs with Spark, Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016.

+

Under the hood of SparkR is Spark SQL engine. This avoids the overheads of running interpreted R code, and the optimized SQL execution engine in Spark uses structural information about data and computation flow to perform a bunch of optimizations to speed up the computation.

+

The main method calls of actual computation happen in the Spark JVM of the driver. We have a socket-based SparkR API that allows us to invoke functions on the JVM from R. We use a SparkR JVM backend that listens on a Netty-based socket server.

+

Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and creating new objects. Method invocation can be done in two ways.

+
    +

  • invokeJMethod takes a reference to an existing Java object and a list of arguments to be passed on to the method.

    • +

  • invokeJStatic takes a class name for static method and a list of arguments to be passed on to the method.

    • +
+

The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method.

+

To create objects, a special method name init is used and then similarly the appropriate constructor is invoked with provided arguments.

+

Finally, we use a new R class jobj that refers to a Java object existing in the backend. These references are tracked on the Java side and are automatically garbage collected when they go out of scope on the R side.

+
+
+
+

Appendix

+
+

R and Spark Data Types

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RSpark
bytebyte
integerinteger
floatfloat
doubledouble
numericdouble
characterstring
stringstring
binarybinary
rawbinary
logicalboolean
POSIXcttimestamp
POSIXlttimestamp
Datedate
arrayarray
listarray
envmap
+
+
+
+

References

+ +
+ + + +
+
+ +
+ + + + + + + + diff --git a/R/pkg/vignettes/sparkr-vignettes.md b/R/pkg/vignettes/sparkr-vignettes.md new file mode 100644 index 0000000000000..1a23560869acd --- /dev/null +++ b/R/pkg/vignettes/sparkr-vignettes.md @@ -0,0 +1,222 @@ +Untitled +================ + +Overview +-------- + +SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using MLlib. + +Getting Started +--------------- + +We start with an example running on the local machine and provide an overview of SparkR in multiple dimensions: data ingestion, data processing and machine learning. + +First, let's load and attach the package. + +``` r +library(SparkR) +``` + +To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged. You may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. + +``` r +install.spark(overwrite = TRUE) +``` + +If you have a Spark package, you don't have to install again, but an environment variable should be set to let SparkR know where it is. If you have run the `install.spark` function, this has already been done for you. + +``` r +Sys.setenv(SPARK_HOME = "/HOME/spark") +``` + +`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any spark packages depended on, etc. We use default settings. + +``` r +sparkR.session() +``` + + ## Launching java with spark-submit command /Users/junyangq/spark//bin/spark-submit sparkr-shell /var/folders/jh/6pw_r0d51317krg8ftgy53f40000gn/T//RtmpT7vIHb/backend_portb8c54afe73fa + + ## Java ref type org.apache.spark.sql.SparkSession id 1 + +The operations in SparkR are centered around a class of R object called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. + +`SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame, + +``` r +cars <- cbind(model = rownames(mtcars), mtcars) +carsDF <- createDataFrame(cars) +``` + +We can view the first few rows of the `SparkDataFrame` by `showDF` or `head` function. + +``` r +showDF(carsDF) +``` + + ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ + ## | model| mpg|cyl| disp| hp|drat| wt| qsec| vs| am|gear|carb| + ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ + ## | Mazda RX4|21.0|6.0|160.0|110.0| 3.9| 2.62|16.46|0.0|1.0| 4.0| 4.0| + ## | Mazda RX4 Wag|21.0|6.0|160.0|110.0| 3.9|2.875|17.02|0.0|1.0| 4.0| 4.0| + ## | Datsun 710|22.8|4.0|108.0| 93.0|3.85| 2.32|18.61|1.0|1.0| 4.0| 1.0| + ## | Hornet 4 Drive|21.4|6.0|258.0|110.0|3.08|3.215|19.44|1.0|0.0| 3.0| 1.0| + ## | Hornet Sportabout|18.7|8.0|360.0|175.0|3.15| 3.44|17.02|0.0|0.0| 3.0| 2.0| + ## | Valiant|18.1|6.0|225.0|105.0|2.76| 3.46|20.22|1.0|0.0| 3.0| 1.0| + ## | Duster 360|14.3|8.0|360.0|245.0|3.21| 3.57|15.84|0.0|0.0| 3.0| 4.0| + ## | Merc 240D|24.4|4.0|146.7| 62.0|3.69| 3.19| 20.0|1.0|0.0| 4.0| 2.0| + ## | Merc 230|22.8|4.0|140.8| 95.0|3.92| 3.15| 22.9|1.0|0.0| 4.0| 2.0| + ## | Merc 280|19.2|6.0|167.6|123.0|3.92| 3.44| 18.3|1.0|0.0| 4.0| 4.0| + ## | Merc 280C|17.8|6.0|167.6|123.0|3.92| 3.44| 18.9|1.0|0.0| 4.0| 4.0| + ## | Merc 450SE|16.4|8.0|275.8|180.0|3.07| 4.07| 17.4|0.0|0.0| 3.0| 3.0| + ## | Merc 450SL|17.3|8.0|275.8|180.0|3.07| 3.73| 17.6|0.0|0.0| 3.0| 3.0| + ## | Merc 450SLC|15.2|8.0|275.8|180.0|3.07| 3.78| 18.0|0.0|0.0| 3.0| 3.0| + ## | Cadillac Fleetwood|10.4|8.0|472.0|205.0|2.93| 5.25|17.98|0.0|0.0| 3.0| 4.0| + ## |Lincoln Continental|10.4|8.0|460.0|215.0| 3.0|5.424|17.82|0.0|0.0| 3.0| 4.0| + ## | Chrysler Imperial|14.7|8.0|440.0|230.0|3.23|5.345|17.42|0.0|0.0| 3.0| 4.0| + ## | Fiat 128|32.4|4.0| 78.7| 66.0|4.08| 2.2|19.47|1.0|1.0| 4.0| 1.0| + ## | Honda Civic|30.4|4.0| 75.7| 52.0|4.93|1.615|18.52|1.0|1.0| 4.0| 2.0| + ## | Toyota Corolla|33.9|4.0| 71.1| 65.0|4.22|1.835| 19.9|1.0|1.0| 4.0| 1.0| + ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ + ## only showing top 20 rows + +We use `magrittr` package to chain operations when necessary in the rest of the document. + +``` r +library(magrittr) +``` + +Common data processing operations such as `filter`, `select` are supported on the `SparkDataFrame`. + +``` r +carsSubDF <- select(carsDF, "model", "mpg", "hp") +carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200) +showDF(carsSubDF) +``` + + ## +-------------------+----+-----+ + ## | model| mpg| hp| + ## +-------------------+----+-----+ + ## | Duster 360|14.3|245.0| + ## | Cadillac Fleetwood|10.4|205.0| + ## |Lincoln Continental|10.4|215.0| + ## | Chrysler Imperial|14.7|230.0| + ## | Camaro Z28|13.3|245.0| + ## | Ford Pantera L|15.8|264.0| + ## | Maserati Bora|15.0|335.0| + ## +-------------------+----+-----+ + +SparkR support a number of commonly used functions to aggregate data after grouping. + +``` r +carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) +showDF(carsGPDF) +``` + + ## +----+-----+ + ## |gear|count| + ## +----+-----+ + ## | 4.0| 12| + ## | 3.0| 15| + ## | 5.0| 5| + ## +----+-----+ + +SparkR supports a number of widely used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models. + +SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. + +``` r +fit <- spark.glm(carsDF, mpg ~ wt + cyl) +``` + +``` r +summary(fit) +``` + + ## + ## Deviance Residuals: + ## (Note: These are approximate quantiles with relative error <= 0.01) + ## Min 1Q Median 3Q Max + ## -4.2893 -1.7085 -0.4713 1.5729 6.1004 + ## + ## Coefficients: + ## Estimate Std. Error t value Pr(>|t|) + ## (Intercept) 39.686 1.715 23.141 0 + ## wt -3.191 0.75691 -4.2158 0.00022202 + ## cyl -1.5078 0.41469 -3.636 0.0010643 + ## + ## (Dispersion parameter for gaussian family taken to be 6.592137) + ## + ## Null deviance: 1126.05 on 31 degrees of freedom + ## Residual deviance: 191.17 on 29 degrees of freedom + ## AIC: 156 + ## + ## Number of Fisher Scoring iterations: 1 + +``` r +sparkR.session.stop() +``` + +Vignettes are long form documentation commonly included in packages. Because they are part of the distribution of the package, they need to be as compact as possible. The `html_vignette` output type provides a custom style sheet (and tweaks some options) to ensure that the resulting html is as small as possible. The `html_vignette` format: + +- Never uses retina figures +- Has a smaller default figure size +- Uses a custom CSS stylesheet instead of the default Twitter Bootstrap style + +Vignette Info +------------- + +Note the various macros within the `vignette` section of the metadata block above. These are required in order to instruct R how to build the vignette. Note that you should change the `title` field and the `\VignetteIndexEntry` to match the title of your vignette. + +Styles +------ + +The `html_vignette` template includes a basic CSS theme. To override this theme you can specify your own CSS in the document metadata as follows: + + output: + rmarkdown::html_vignette: + css: mystyles.css + +Figures +------- + +The figure sizes have been customised so that you can easily put two images side-by-side. + +``` r +plot(1:10) +plot(10:1) +``` + +![](sparkr-vignettes_files/figure-markdown_github/unnamed-chunk-14-1.png)![](sparkr-vignettes_files/figure-markdown_github/unnamed-chunk-14-2.png) + +You can enable figure captions by `fig_caption: yes` in YAML: + + output: + rmarkdown::html_vignette: + fig_caption: yes + +Then you can use the chunk option `fig.cap = "Your figure caption."` in **knitr**. + +More Examples +------------- + +You can write math expressions, e.g. *Y* = *X**β* + *ϵ*, footnotes[1], and tables, e.g. using `knitr::kable()`. + +| | mpg| cyl| disp| hp| drat| wt| qsec| vs| am| gear| carb| +|-------------------|-----:|----:|------:|----:|-----:|------:|------:|----:|----:|-----:|-----:| +| Mazda RX4 | 21.0| 6| 160.0| 110| 3.90| 2.620| 16.46| 0| 1| 4| 4| +| Mazda RX4 Wag | 21.0| 6| 160.0| 110| 3.90| 2.875| 17.02| 0| 1| 4| 4| +| Datsun 710 | 22.8| 4| 108.0| 93| 3.85| 2.320| 18.61| 1| 1| 4| 1| +| Hornet 4 Drive | 21.4| 6| 258.0| 110| 3.08| 3.215| 19.44| 1| 0| 3| 1| +| Hornet Sportabout | 18.7| 8| 360.0| 175| 3.15| 3.440| 17.02| 0| 0| 3| 2| +| Valiant | 18.1| 6| 225.0| 105| 2.76| 3.460| 20.22| 1| 0| 3| 1| +| Duster 360 | 14.3| 8| 360.0| 245| 3.21| 3.570| 15.84| 0| 0| 3| 4| +| Merc 240D | 24.4| 4| 146.7| 62| 3.69| 3.190| 20.00| 1| 0| 4| 2| +| Merc 230 | 22.8| 4| 140.8| 95| 3.92| 3.150| 22.90| 1| 0| 4| 2| +| Merc 280 | 19.2| 6| 167.6| 123| 3.92| 3.440| 18.30| 1| 0| 4| 4| + +Also a quote using `>`: + +> "He who gives up \[code\] safety for \[code\] speed deserves neither." ([via](https://twitter.com/hadleywickham/status/504368538874703872)) + +[1] A footnote here. From b50907001c1ef68d34859c5628856aba455de2f4 Mon Sep 17 00:00:00 2001 From: Shivaram Venkataraman Date: Mon, 5 Sep 2016 12:00:42 -0700 Subject: [PATCH 02/13] Minor updates to SparkR vignette --- R/pkg/vignettes/sparkr-vignettes.Rmd | 79 +++++++++++++--------------- 1 file changed, 37 insertions(+), 42 deletions(-) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index ff0455f23b0a2..2778b9420e244 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -15,7 +15,7 @@ SparkR is an R package that provides a light-weight frontend to use Apache Spark ## Getting Started -We begin with an example running on the local machine, trying to provide an overview of the use of SparkR: data ingestion, data processing and machine learning. +We begin with an example running on the local machine and provide an overview of the use of SparkR: data ingestion, data processing and machine learning. First, let's load and attach the package. ```{r, message=FALSE} @@ -23,32 +23,27 @@ library(SparkR) ``` -To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged. +To use SparkR, you need an Apache Spark installation which will execute SparkR programs. -If you don't have a Spark package on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. +If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. ```{r, eval=FALSE} install.spark() ``` -If you have a Spark package, you don't have to install again, but make sure to set `SPARK_HOME` environment variable to let SparkR know where the main Spark package is. +`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. We use default settings in which it runs in local mode. -```{r, eval=FALSE} -Sys.setenv(SPARK_HOME = "/HOME/spark") -``` - -```{r, echo=FALSE} -# Set to your own spark folder if you want to knit this Rmd. -Sys.setenv(SPARK_HOME = "/Users/junyangq/spark/") +```{r, message=FALSE, warning=FALSE} +sparkR.session() ``` -`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any spark packages depended on, etc. We use default settings. It runs in local mode. +If you already have Spark installed, you don't have to install again and can pass the `sparkHome` argument to `sparkR.session` to let SparkR know where the Spark installation is. -```{r, message=FALSE, warning=FALSE} -sparkR.session() +```{r, eval=FALSE} +sparkR.session(sparkHome = "/HOME/spark") ``` -The operations in SparkR are centered around an R object class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. +The operations in SparkR are centered around an R class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. `SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame, @@ -82,7 +77,7 @@ carsGP <- collect(carsGPDF) class(carsGP) ``` -SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models. +SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models. SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. ```{r} @@ -125,14 +120,14 @@ paste("Spark", packageVersion("SparkR")) ``` It should be used both on the local computer and on the remote cluster. -To connect, pass the URL of the master node to `sparkR.session`. A complete list can be seen in [Spark Master URLs](http://spark.apache.org/docs/latest/submitting-applications.html#master-urls). -For example, to connect to a local cluster master, we can call +To connect, pass the URL of the master node to `sparkR.session`. A complete list can be seen in [Spark Master URLs](http://spark.apache.org/docs/latest/submitting-applications.html#master-urls). +For example, to connect to a standalone Spark master, we can call ```{r, eval=FALSE} sparkR.session(master = "spark://local:7077") ``` -For YARN cluster, SparkR only supports the client mode. +For YARN cluster, SparkR supports the client mode with the master set as "yarn". ```{r, eval=FALSE} sparkR.session(master = "yarn") ``` @@ -159,7 +154,7 @@ sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0") We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail. ```{r} -people <- read.df(paste0(Sys.getenv("SPARK_HOME"), +people <- read.df(paste0(sparkR.conf("spark.home"), "/examples/src/main/resources/people.json"), "json") count(people) head(people) @@ -192,7 +187,7 @@ You can also create SparkDataFrames from Hive tables. To do this we will need to ```{r, eval=FALSE} sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") -txtPath <- paste0(Sys.getenv("SPARK_HOME"), "/examples/src/main/resources/kv1.txt") +txtPath <- paste0(sparkR.conf("spark.home"), "/examples/src/main/resources/kv1.txt") sqlCMD <- sprintf("LOAD DATA LOCAL INPATH '%s' INTO TABLE src", txtPath) sql(sqlCMD) @@ -215,7 +210,7 @@ Here are more concrete examples. dplyr | SparkR -------- | --------- -`select(mtcars, mpg, hp)` | `select(carsDF, "mpg", "hp")` +`select(mtcars, mpg, hp)` | `select(carsDF, "mpg", "hp")` `filter(mtcars, mpg > 20, hp > 100)` | `filter(carsDF, carsDF$mpg > 20, carsDF$hp > 100)` Other differences will be mentioned in the specific methods. @@ -281,11 +276,11 @@ A window function is a variation of aggregation function. In simple words, * window function: `n` to `n` mapping - returns one value for each entry in the group, but the value may depend on all the entries of the *group*. Examples include `rank`, `lead`, `lag`. -Formally, the *group* we mentioned is called the Frame. Every input row can have a unique frame associated with it and the output of the window function on that row is based on the rows confined in that frame. +Formally, the *group* mentioned above is called the *frame*. Every input row can have a unique frame associated with it and the output of the window function on that row is based on the rows confined in that frame. -Window functions are often used in conjunction with the following functions: `windowPartitionBy`, `windowOrderBy`, `partitionBy`, `orderBy`, `over`. It would be easier to look at an example. +Window functions are often used in conjunction with the following functions: `windowPartitionBy`, `windowOrderBy`, `partitionBy`, `orderBy`, `over`. To illustrate this we next look at an example. -We still use the `mtcars` dataset. The corresponding `SparkDataFrame` is `carsDF`. Suppose for each number of cylinders, we want to calculate the rank of each car in `mpg` within the group. +We still use the `mtcars` dataset. The corresponding `SparkDataFrame` is `carsDF`. Suppose for each number of cylinders, we want to calculate the rank of each car in `mpg` within the group. ```{r} carsSubDF <- select(carsDF, "model", "mpg", "cyl") ws <- orderBy(windowPartitionBy("cyl"), "mpg") @@ -295,15 +290,15 @@ showDF(carsRank) We explain in detail the above steps. -* `windowPartitionBy` creates a Window Specification object `WindowSpec` that defines the partition. It controls which rows will be in the same partition as the given row. In this case, rows with the same value in `cyl` will be put in the same partition. `orderBy` further defines the ordering - the position a given row is in the partition. The resulting `WindowSpec` is returned as `ws`. +* `windowPartitionBy` creates a window specification object `WindowSpec` that defines the partition. It controls which rows will be in the same partition as the given row. In this case, rows with the same value in `cyl` will be put in the same partition. `orderBy` further defines the ordering - the position a given row is in the partition. The resulting `WindowSpec` is returned as `ws`. -More Window Specification methods include `rangeBetween`, which can define boundaries of the frame by value, and `rowsBetween`, which can define the boundaries by row indices. +More window specification methods include `rangeBetween`, which can define boundaries of the frame by value, and `rowsBetween`, which can define the boundaries by row indices. * `withColumn` appends a Column called `"rank"` to the `SparkDataFrame`. `over` returns a windowing column. The first argument is usually a Column returned by window function(s) such as `rank()`, `lead(carsDF$wt)`. That calculates the corresponding values according to the partitioned-and-ordered table. ### User-Defined Function -In SparkR, we support several kinds of User-Defined Functions. +In SparkR, we support several kinds of user-defined functions (UDFs). #### Apply by Partition @@ -319,7 +314,7 @@ out <- dapply(carsSubDF, function(x) { x <- cbind(x, x$mpg * 1.61) }, schema) head(collect(out)) ``` -Like `dapply`, apply a function to each partition of a `SparkDataFrame` and collect the result back. The output of function should be a `data.frame`. But, Schema is not required to be passed. Note that `dapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. +Like `dapply`, apply a function to each partition of a `SparkDataFrame` and collect the result back. The output of function should be a `data.frame`, but no schema is required in this case. Note that `dapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. ```{r} out <- dapplyCollect( @@ -345,7 +340,7 @@ result <- gapply( head(arrange(result, "max_mpg", decreasing = TRUE)) ``` -Like gapply, `gapplyCollect` applies a function to each partition of a `SparkDataFrame` and collect the result back to R `data.frame`. The output of the function should be a `data.frame`. But, the schema is not required to be passed. Note that `gapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. +Like gapply, `gapplyCollect` applies a function to each partition of a `SparkDataFrame` and collect the result back to R `data.frame`. The output of the function should be a `data.frame` but no schema is required in this case. Note that `gapplyCollect` can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory. ```{r} result <- gapplyCollect( @@ -361,7 +356,7 @@ head(result[order(result$max_mpg, decreasing = TRUE), ]) #### Distribute Local Functions -Similar to `lapply` in native R, `spark.lapply` runs a function over a list of elements and distributes the computations with Spark. Applies a function in a manner that is similar to `doParallel` or `lapply` to elements of a list. The results of all the computations should fit in a single machine. If that is not the case they can do something like `df <- createDataFrame(list)` and then use `dapply`. +Similar to `lapply` in native R, `spark.lapply` runs a function over a list of elements and distributes the computations with Spark. `spark.lapply` works in a manner that is similar to `doParallel` or `lapply` to elements of a list. The results of all the computations should fit in a single machine. If that is not the case you can do something like `df <- createDataFrame(list)` and then use `dapply`. ```{r} families <- c("gaussian", "poisson") @@ -386,7 +381,7 @@ print(model.summaries) A `SparkDataFrame` can also be registered as a temporary view in Spark SQL and that allows you to run SQL queries over its data. The sql function enables applications to run SQL queries programmatically and returns the result as a `SparkDataFrame`. ```{r} -people <- read.df(paste0(Sys.getenv("SPARK_HOME"), +people <- read.df(paste0(sparkR.conf("spark.home"), "/examples/src/main/resources/people.json"), "json") ``` @@ -433,7 +428,7 @@ For most above, SparkR supports **R formula operators**, including `~`, `.`, `:` ### Training and Test Sets -We can easily split `SparkDataFrame` into random training and test sets by the `randomSplit` function. It returns a list of split `SparkDataFrames` with provided `weights`. We use `carsDF` as an example and want to have about $70%$ training data and $30%$ test data. +We can easily split `SparkDataFrame` into random training and test sets by the `randomSplit` function. It returns a list of split `SparkDataFrames` with provided `weights`. We use `carsDF` as an example and want to have about $70%$ training data and $30%$ test data. ```{r} splitDF_list <- randomSplit(carsDF, c(0.7, 0.3), seed = 0) carsDF_train <- splitDF_list[[1]] @@ -474,7 +469,7 @@ There are three ways to specify the `family` argument. For more information regarding the families and their link functions, see the Wikipedia page [Generalized Linear Model](https://en.wikipedia.org/wiki/Generalized_linear_model). -We use the `mtcars` dataset as an illustration. The corresponding `SparkDataFrame` is `carsDF`. After fitting the model, we print out a summary and see the fitted values by making predictions on the original dataset. We can also pass into a new `SparkDataFrame` of same schema to predict on new data. +We use the `mtcars` dataset as an illustration. The corresponding `SparkDataFrame` is `carsDF`. After fitting the model, we print out a summary and see the fitted values by making predictions on the original dataset. We can also pass into a new `SparkDataFrame` of same schema to predict on new data. ```{r} gaussianGLM <- spark.glm(carsDF, mpg ~ wt + hp) @@ -546,7 +541,7 @@ showDF(select(gmmFitted, "V1", "V2", "prediction")) * Topics and documents both exist in a feature space, where feature vectors are vectors of word counts (bag of words). -* Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated. +* Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated. To use LDA, we need to specify a `features` column in `data` where each entry represents a document. There are two type options for the column: @@ -742,15 +737,15 @@ unlink(modelPath) There are three main object classes in SparkR you may be working with. -* `SparkDataFrame`: the central component of SparkR. It is an S4 class representing distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R. It has two slots `sdf` and `env`. +* `SparkDataFrame`: the central component of SparkR. It is an S4 class representing distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R. It has two slots `sdf` and `env`. + `sdf` stores a reference to the corresponding Spark Dataset in the Spark JVM backend. + `env` saves the meta-information of the object such as `isCached`. - + It can be created by data import methods or by transforming an existing `SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data processing functions and feed that into machine learning algorithms. * `Column`: an S4 class representing column of `SparkDataFrame`. The slot `jc` saves a reference to the corresponding Column object in the Spark JVM backend. -It can be obtained from a `SparkDataFrame` by `$` operator, `df$col`. More often, it is used together with other functions, for example, with `select` to select particular columns, with `filter` and constructed conditions to select rows, with aggregation functions to compute aggregate statistics for each group. +It can be obtained from a `SparkDataFrame` by `$` operator, `df$col`. More often, it is used together with other functions, for example, with `select` to select particular columns, with `filter` and constructed conditions to select rows, with aggregation functions to compute aggregate statistics for each group. * `GroupedData`: an S4 class representing grouped data created by `groupBy` or by transforming other `GroupedData`. Its `sgd` slot saves a reference to a RelationalGroupedDataset object in the backend. @@ -762,7 +757,7 @@ A complete description of architecture can be seen in paper [SparkR: Scaling R P Under the hood of SparkR is Spark SQL engine. This avoids the overheads of running interpreted R code, and the optimized SQL execution engine in Spark uses structural information about data and computation flow to perform a bunch of optimizations to speed up the computation. -The main method calls of actual computation happen in the Spark JVM of the driver. We have a socket-based SparkR API that allows us to invoke functions on the JVM from R. We use a SparkR JVM backend that listens on a Netty-based socket server. +The main method calls of actual computation happen in the Spark JVM of the driver. We have a socket-based SparkR API that allows us to invoke functions on the JVM from R. We use a SparkR JVM backend that listens on a Netty-based socket server. Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and creating new objects. Method invocation can be done in two ways. @@ -770,9 +765,9 @@ Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and * `invokeJStatic` takes a class name for static method and a list of arguments to be passed on to the method. -The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method. +The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method. -To create objects, a special method name `init` is used and then similarly the appropriate constructor is invoked with provided arguments. +To create objects, a special method name `init` is used and then similarly the appropriate constructor is invoked with provided arguments. Finally, we use a new R class `jobj` that refers to a Java object existing in the backend. These references are tracked on the Java side and are automatically garbage collected when they go out of scope on the R side. @@ -807,7 +802,7 @@ env | map * [Machine Learning Library Guide (MLlib)](http://spark.apache.org/docs/latest/ml-guide.html) -* [SparkR: Scaling R Programs with Spark](SparkR: Scaling R Programs with Spark), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. +* [SparkR: Scaling R Programs with Spark](https://people.csail.mit.edu/matei/papers/2016/sigmod_sparkr.pdf), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. From f93b69ebd7525833f48ab508088baed387743167 Mon Sep 17 00:00:00 2001 From: Shivaram Venkataraman Date: Tue, 6 Sep 2016 09:33:45 -0700 Subject: [PATCH 03/13] Address comments --- R/pkg/vignettes/sparkr-vignettes.Rmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 2778b9420e244..7fae5c60c0cd3 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -23,7 +23,7 @@ library(SparkR) ``` -To use SparkR, you need an Apache Spark installation which will execute SparkR programs. +To use SparkR, you need an Apache Spark installation. The Spark installation will be used to run a backend process that will compile and execute SparkR programs. If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. @@ -121,7 +121,7 @@ paste("Spark", packageVersion("SparkR")) It should be used both on the local computer and on the remote cluster. To connect, pass the URL of the master node to `sparkR.session`. A complete list can be seen in [Spark Master URLs](http://spark.apache.org/docs/latest/submitting-applications.html#master-urls). -For example, to connect to a standalone Spark master, we can call +For example, to connect to a local standalone Spark master, we can call ```{r, eval=FALSE} sparkR.session(master = "spark://local:7077") From cbd0b414e7dcf2b2b291da836760a36c9e2f5608 Mon Sep 17 00:00:00 2001 From: junyangq Date: Wed, 7 Sep 2016 02:20:24 +0800 Subject: [PATCH 04/13] Update to make more R-user-friendly. Be precise about version of ML algorithms. Add Windows-specific workaround. --- R/pkg/vignettes/people.parquet/._SUCCESS.crc | Bin 8 -> 0 bytes R/pkg/vignettes/people.parquet/_SUCCESS | 0 R/pkg/vignettes/sparkr-vignettes.Rmd | 94 +- R/pkg/vignettes/sparkr-vignettes.html | 1435 ------------------ R/pkg/vignettes/sparkr-vignettes.md | 222 --- 5 files changed, 68 insertions(+), 1683 deletions(-) delete mode 100644 R/pkg/vignettes/people.parquet/._SUCCESS.crc delete mode 100644 R/pkg/vignettes/people.parquet/_SUCCESS delete mode 100644 R/pkg/vignettes/sparkr-vignettes.html delete mode 100644 R/pkg/vignettes/sparkr-vignettes.md diff --git a/R/pkg/vignettes/people.parquet/._SUCCESS.crc b/R/pkg/vignettes/people.parquet/._SUCCESS.crc deleted file mode 100644 index 3b7b044936a890cd8d651d349a752d819d71d22c..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8 PcmYc;N@ieSU}69O2$TUk diff --git a/R/pkg/vignettes/people.parquet/_SUCCESS b/R/pkg/vignettes/people.parquet/_SUCCESS deleted file mode 100644 index e69de29bb2d1d..0000000000000 diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 7fae5c60c0cd3..6543a2094c9c0 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -22,27 +22,14 @@ First, let's load and attach the package. library(SparkR) ``` +`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. -To use SparkR, you need an Apache Spark installation. The Spark installation will be used to run a backend process that will compile and execute SparkR programs. - -If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. - -```{r, eval=FALSE} -install.spark() -``` - -`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. We use default settings in which it runs in local mode. +We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession). ```{r, message=FALSE, warning=FALSE} sparkR.session() ``` -If you already have Spark installed, you don't have to install again and can pass the `sparkHome` argument to `sparkR.session` to let SparkR know where the Spark installation is. - -```{r, eval=FALSE} -sparkR.session(sparkHome = "/HOME/spark") -``` - The operations in SparkR are centered around an R class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. `SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame, @@ -100,9 +87,34 @@ sparkR.session.stop() ## Setup -### Spark Session +### Installation + +Different from many other R packages, to use SparkR, you need an additional installation of Apache Spark. The Spark installation will be used to run a backend process that will compile and execute SparkR programs. + +If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. + +```{r, eval=FALSE} +install.spark() +``` + +If you already have Spark installed, you don't have to install again and can pass the `sparkHome` argument to `sparkR.session` to let SparkR know where the Spark installation is. + +```{r, eval=FALSE} +sparkR.session(sparkHome = "/HOME/spark") +``` + +### Spark Session {#SetupSparkSession} + +**For Windows users**: Due to different file prefixes across operating systems, to avoid the issue of potential wrong prefix, a current workaround is to specify `spark.sql.warehouse.dir` when starting the `SparkSession`. + +```{r, eval=FALSE} +spark_warehouse_path <- file.path(path.expand('~'), "spark-warehouse") +sparkR.session(spark.sql.warehouse.dir = spark_warehouse_path) +``` -The following Spark driver properties can be set in `sparkConfig`. +In addition to `sparkHome`, many other options can be specified in `sparkR.session`. For a complete list, see the [SparkR API doc](http://spark.apache.org/docs/latest/api/R/sparkR.session.html). + +In particular, the following Spark driver properties can be set in `sparkConfig`. Property Name | Property group | spark-submit equivalent ---------------- | ------------------ | ---------------------- @@ -111,6 +123,8 @@ spark.driver.extraClassPath | Runtime Environment | --driver-class-path spark.driver.extraJavaOptions | Runtime Environment | --driver-java-options spark.driver.extraLibraryPath | Runtime Environment | --driver-library-path + + ### Cluster Mode SparkR can connect to remote Spark clusters. [Cluster Mode Overview](http://spark.apache.org/docs/latest/cluster-overview.html) is a good introduction to different Spark cluster modes. @@ -145,17 +159,31 @@ head(df) ### Data Sources SparkR supports operating on a variety of data sources through the `SparkDataFrame` interface. You can check the Spark SQL programming guide for more [specific options](https://spark.apache.org/docs/latest/sql-programming-guide.html#manually-specifying-options) that are available for the built-in data sources. -The general method for creating `SparkDataFrame` from data sources is `read.df`. This method takes in the path for the file to load and the type of data source, and the currently active Spark Session will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively and through Spark Packages you can find data source connectors for popular file formats like Avro. These packages can be added with `sparkPackages` parameter when initializing SparkSession using `sparkR.session'.` +The general method for creating `SparkDataFrame` from data sources is `read.df`. This method takes in the path for the file to load and the type of data source, and the currently active Spark Session will be used automatically. SparkR supports reading CSV, JSON and Parquet files natively and through Spark Packages you can find data source connectors for popular file formats like Avro. These packages can be added with `sparkPackages` parameter when initializing SparkSession using `sparkR.session'.` ```{r, eval=FALSE} sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0") ``` -We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail. +We can see how to use data sources using an example CSV input file. For more information please refer to SparkR [read.df](https://spark.apache.org/docs/latest/api/R/read.df.html) API documentation. +```{r, eval=FALSE} +df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA") +``` + +The data sources API natively supports JSON formatted input files. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail. + +Let's take a look at the first two lines of the raw JSON file used here. ```{r} -people <- read.df(paste0(sparkR.conf("spark.home"), - "/examples/src/main/resources/people.json"), "json") +filePath <- paste0(sparkR.conf("spark.home"), + "/examples/src/main/resources/people.json") +readLines(filePath, n = 2L) +``` + +We use `read.df` to read that into a `SparkDataFrame`. + +```{r} +people <- read.df(filePath, "json") count(people) head(people) ``` @@ -172,12 +200,9 @@ people <- read.json(paste0(Sys.getenv("SPARK_HOME"), "/examples/src/main/resources/people.json"))) count(people) ``` -The data sources API natively supports CSV formatted input files. For more information please refer to SparkR [read.df](https://spark.apache.org/docs/latest/api/R/read.df.html) API documentation. -```{r, eval=FALSE} -df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA") -``` + The data sources API can also be used to save out `SparkDataFrames` into multiple file formats. For example we can save the `SparkDataFrame` from the previous example to a Parquet file using `write.df`. -```{r} +```{r, eval=FALSE} write.df(people, path = "people.parquet", source = "parquet", mode = "overwrite") ``` @@ -519,6 +544,9 @@ head(aftPredictions) ``` #### Gaussian Mixture Model + +(Coming in 2.1.0) + `spark.gaussianMixture` fits multivariate [Gaussian Mixture Model](https://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model) (GMM) against a `SparkDataFrame`. [Expectation-Maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) (EM) is used to approximate the maximum likelihood estimator (MLE) of the model. We use a simulated example to demostrate the usage. @@ -535,6 +563,9 @@ showDF(select(gmmFitted, "V1", "V2", "prediction")) #### Latent Dirichlet Allocation + +(Coming in 2.1.0) + `spark.lda` fits a [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) model on a `SparkDataFrame`. It is often used in topic modeling in which topics are inferred from a collection of text documents. LDA can be thought of as a clustering algorithm as follows: * Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset. @@ -605,6 +636,9 @@ perplexity #### Multilayer Perceptron + +(Coming in 2.1.0) + Multilayer perceptron classifier (MLPC) is a classifier based on the [feedforward artificial neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network). MLPC consists of multiple layers of nodes. Each layer is fully connected to the next layer in the network. Nodes in the input layer represent the input data. All other nodes map inputs to outputs by a linear combination of the inputs with the node’s weights $w$ and bias $b$ and applying an activation function. This can be written in matrix form for MLPC with $K+1$ layers as follows: $$ y(x)=f_K(\ldots f_2(w_2^T f_1(w_1^T x + b_1) + b_2) \ldots + b_K). @@ -640,6 +674,8 @@ MLPC employs backpropagation for learning the model. We use the logistic loss fu #### Collaborative Filtering +(Coming in 2.1.0) + `spark.als` learns latent factors in [collaborative filtering](https://en.wikipedia.org/wiki/Recommender_system#Collaborative_filtering) via [alternating least squares](http://dl.acm.org/citation.cfm?id=1608614). There are multiple options that can be configured in `spark.als`, including `rank`, `reg`, `nonnegative`. For a complete list, refer to the help file. @@ -668,6 +704,9 @@ showDF(predicted) ``` #### Isotonic Regression Model + +(Coming in 2.1.0) + `spark.isoreg` fits an [Isotonic Regression](https://en.wikipedia.org/wiki/Isotonic_regression) model against a `SparkDataFrame`. It solves a weighted univariate a regression problem under a complete order constraint. Specifically, given a set of real observed responses $y_1, \ldots, y_n$, corresponding real features $x_1, \ldots, x_n$, and optionally positive weights $w_1, \ldots, w_n$, we want to find a monotone (piecewise linear) function $f$ to minimize $$ \ell(f) = \sum_{i=1}^n w_i (y_i - f(x_i))^2. @@ -709,6 +748,9 @@ newDF <- createDataFrame(data.frame(x = c(1.5, 3.2))) head(predict(isoregModel, newDF)) ``` +#### What's More? +We also expect Decision Tree, Random Forest, Kolmogorov-Smirnov Test coming in the next version 2.1.0. + ### Model Persistence The following example shows how to save/load an ML model by SparkR. ```{r} diff --git a/R/pkg/vignettes/sparkr-vignettes.html b/R/pkg/vignettes/sparkr-vignettes.html deleted file mode 100644 index 65da24dc04c60..0000000000000 --- a/R/pkg/vignettes/sparkr-vignettes.html +++ /dev/null @@ -1,1435 +0,0 @@ - - - - - - - - - - - - - -SparkR - Practical Guide - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
-
-
-
-
- -
- - - - - - - -
-

Overview

-

SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using MLlib.

-
-
-

Getting Started

-

We begin with an example running on the local machine, trying to provide an overview of the use of SparkR: data ingestion, data processing and machine learning.

-

First, let’s load and attach the package.

-
library(SparkR)
-

To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged.

-

If you don’t have a Spark package on the computer, you may download it from Apache Spark Website. Alternatively, we provide an easy-to-use function install.spark to complete this process.

-
install.spark()
-

If you have a Spark package, you don’t have to install again, but make sure to set SPARK_HOME environment variable to let SparkR know where the main Spark package is.

-
Sys.setenv(SPARK_HOME = "/HOME/spark")
-

SparkSession is the entry point into SparkR which connects your R program to a Spark cluster. You can create a SparkSession using sparkR.session and pass in options such as the application name, any spark packages depended on, etc. We use default settings. It runs in local mode.

-
sparkR.session()
-
## Launching java with spark-submit command /Users/junyangq/spark//bin/spark-submit   sparkr-shell /var/folders/jh/6pw_r0d51317krg8ftgy53f40000gn/T//RtmpKAtJ8U/backend_portbe3a130d80ff
-
## Java ref type org.apache.spark.sql.SparkSession id 1
-

The operations in SparkR are centered around an R object class called SparkDataFrame. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood.

-

SparkDataFrame can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a SparkDataFrame from a local R data frame,

-
cars <- cbind(model = rownames(mtcars), mtcars)
-carsDF <- createDataFrame(cars)
-

We can view the first few rows of the SparkDataFrame by showDF or head function.

-
showDF(carsDF)
-
## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
-## |              model| mpg|cyl| disp|   hp|drat|   wt| qsec| vs| am|gear|carb|
-## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
-## |          Mazda RX4|21.0|6.0|160.0|110.0| 3.9| 2.62|16.46|0.0|1.0| 4.0| 4.0|
-## |      Mazda RX4 Wag|21.0|6.0|160.0|110.0| 3.9|2.875|17.02|0.0|1.0| 4.0| 4.0|
-## |         Datsun 710|22.8|4.0|108.0| 93.0|3.85| 2.32|18.61|1.0|1.0| 4.0| 1.0|
-## |     Hornet 4 Drive|21.4|6.0|258.0|110.0|3.08|3.215|19.44|1.0|0.0| 3.0| 1.0|
-## |  Hornet Sportabout|18.7|8.0|360.0|175.0|3.15| 3.44|17.02|0.0|0.0| 3.0| 2.0|
-## |            Valiant|18.1|6.0|225.0|105.0|2.76| 3.46|20.22|1.0|0.0| 3.0| 1.0|
-## |         Duster 360|14.3|8.0|360.0|245.0|3.21| 3.57|15.84|0.0|0.0| 3.0| 4.0|
-## |          Merc 240D|24.4|4.0|146.7| 62.0|3.69| 3.19| 20.0|1.0|0.0| 4.0| 2.0|
-## |           Merc 230|22.8|4.0|140.8| 95.0|3.92| 3.15| 22.9|1.0|0.0| 4.0| 2.0|
-## |           Merc 280|19.2|6.0|167.6|123.0|3.92| 3.44| 18.3|1.0|0.0| 4.0| 4.0|
-## |          Merc 280C|17.8|6.0|167.6|123.0|3.92| 3.44| 18.9|1.0|0.0| 4.0| 4.0|
-## |         Merc 450SE|16.4|8.0|275.8|180.0|3.07| 4.07| 17.4|0.0|0.0| 3.0| 3.0|
-## |         Merc 450SL|17.3|8.0|275.8|180.0|3.07| 3.73| 17.6|0.0|0.0| 3.0| 3.0|
-## |        Merc 450SLC|15.2|8.0|275.8|180.0|3.07| 3.78| 18.0|0.0|0.0| 3.0| 3.0|
-## | Cadillac Fleetwood|10.4|8.0|472.0|205.0|2.93| 5.25|17.98|0.0|0.0| 3.0| 4.0|
-## |Lincoln Continental|10.4|8.0|460.0|215.0| 3.0|5.424|17.82|0.0|0.0| 3.0| 4.0|
-## |  Chrysler Imperial|14.7|8.0|440.0|230.0|3.23|5.345|17.42|0.0|0.0| 3.0| 4.0|
-## |           Fiat 128|32.4|4.0| 78.7| 66.0|4.08|  2.2|19.47|1.0|1.0| 4.0| 1.0|
-## |        Honda Civic|30.4|4.0| 75.7| 52.0|4.93|1.615|18.52|1.0|1.0| 4.0| 2.0|
-## |     Toyota Corolla|33.9|4.0| 71.1| 65.0|4.22|1.835| 19.9|1.0|1.0| 4.0| 1.0|
-## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+
-## only showing top 20 rows
-

Common data processing operations such as filter, select are supported on the SparkDataFrame.

-
carsSubDF <- select(carsDF, "model", "mpg", "hp")
-carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200)
-showDF(carsSubDF)
-
## +-------------------+----+-----+
-## |              model| mpg|   hp|
-## +-------------------+----+-----+
-## |         Duster 360|14.3|245.0|
-## | Cadillac Fleetwood|10.4|205.0|
-## |Lincoln Continental|10.4|215.0|
-## |  Chrysler Imperial|14.7|230.0|
-## |         Camaro Z28|13.3|245.0|
-## |     Ford Pantera L|15.8|264.0|
-## |      Maserati Bora|15.0|335.0|
-## +-------------------+----+-----+
-

SparkR can use many common aggregation functions after grouping.

-
carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear))
-showDF(carsGPDF)
-
## +----+-----+
-## |gear|count|
-## +----+-----+
-## | 4.0|   12|
-## | 3.0|   15|
-## | 5.0|    5|
-## +----+-----+
-

The results carsDF and carsSubDF are SparkDataFrame objects. To convert back to R data.frame, we can use collect.

-
carsGP <- collect(carsGPDF)
-class(carsGP)
-
## [1] "data.frame"
-

SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call summary to print a summary of the fitted model, predict to make predictions on new data, and write.ml/read.ml to save/load fitted models.

-

SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example.

-
model <- spark.glm(carsDF, mpg ~ wt + cyl)
-
summary(model)
-
## 
-## Deviance Residuals: 
-## (Note: These are approximate quantiles with relative error <= 0.01)
-##     Min       1Q   Median       3Q      Max  
-## -4.2893  -1.7085  -0.4713   1.5729   6.1004  
-## 
-## Coefficients:
-##              Estimate  Std. Error  t value  Pr(>|t|)  
-## (Intercept)  39.686    1.715       23.141   0         
-## wt           -3.191    0.75691     -4.2158  0.00022202
-## cyl          -1.5078   0.41469     -3.636   0.0010643 
-## 
-## (Dispersion parameter for gaussian family taken to be 6.592137)
-## 
-##     Null deviance: 1126.05  on 31  degrees of freedom
-## Residual deviance:  191.17  on 29  degrees of freedom
-## AIC: 156
-## 
-## Number of Fisher Scoring iterations: 1
-

The model can be saved by write.ml and loaded back using read.ml.

-
write.ml(model, path = "/HOME/tmp/mlModel/glmModel")
-

In the end, we can stop Spark Session by running

-
sparkR.session.stop()
-
-
-

Setup

-
-

Spark Session

-

The following Spark driver properties can be set in sparkConfig.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Property NameProperty groupspark-submit equivalent
spark.driver.memoryApplication Properties–driver-memory
spark.driver.extraClassPathRuntime Environment–driver-class-path
spark.driver.extraJavaOptionsRuntime Environment–driver-java-options
spark.driver.extraLibraryPathRuntime Environment–driver-library-path
-
-
-

Cluster Mode

-

SparkR can connect to remote Spark clusters. Cluster Mode Overview is a good introduction to different Spark cluster modes.

-

When connecting SparkR to a remote Spark cluster, make sure that the Spark version and Hadoop version on the machine match the corresponding versions on the cluster. Current SparkR package is compatible with

-
## [1] "Spark 2.0.0"
-

It should be used both on the local computer and on the remote cluster.

-

To connect, pass the URL of the master node to sparkR.session. A complete list can be seen in Spark Master URLs. For example, to connect to a local cluster master, we can call

-
sparkR.session(master = "spark://local:7077")
-

For YARN cluster, SparkR only supports the client mode.

-
sparkR.session(master = "yarn")
-
-
-
-

Data Import

-
-

Local Data Frame

-

The simplest way is to convert a local R data frame into a SparkDataFrame. Specifically we can use as.DataFrame or createDataFrame and pass in the local R data frame to create a SparkDataFrame. As an example, the following creates a SparkDataFrame based using the faithful dataset from R.

-
df <- as.DataFrame(faithful)
-head(df)
-
##   eruptions waiting
-## 1     3.600      79
-## 2     1.800      54
-## 3     3.333      74
-## 4     2.283      62
-## 5     4.533      85
-## 6     2.883      55
-
-
-

Data Sources

-

SparkR supports operating on a variety of data sources through the SparkDataFrame interface. You can check the Spark SQL programming guide for more specific options that are available for the built-in data sources.

-

The general method for creating SparkDataFrame from data sources is read.df. This method takes in the path for the file to load and the type of data source, and the currently active Spark Session will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively and through Spark Packages you can find data source connectors for popular file formats like Avro. These packages can be added with sparkPackages parameter when initializing SparkSession using sparkR.session'.

-
sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0")
-

We can see how to use data sources using an example JSON input file. Note that the file that is used here is not a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

-
people <- read.df(paste0(Sys.getenv("SPARK_HOME"), 
-                         "/examples/src/main/resources/people.json"), "json")
-count(people)
-
## [1] 3
-
head(people)
-
##   age    name
-## 1  NA Michael
-## 2  30    Andy
-## 3  19  Justin
-

SparkR automatically infers the schema from the JSON file.

-
printSchema(people)
-
## root
-##  |-- age: long (nullable = true)
-##  |-- name: string (nullable = true)
-

If we want to read multiple JSON files, read.json can be used.

-
people <- read.json(paste0(Sys.getenv("SPARK_HOME"),
-                           c("/examples/src/main/resources/people.json",
-                             "/examples/src/main/resources/people.json")))
-count(people)
-
## [1] 6
-

The data sources API natively supports CSV formatted input files. For more information please refer to SparkR read.df API documentation.

-
df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA")
-

The data sources API can also be used to save out SparkDataFrames into multiple file formats. For example we can save the SparkDataFrame from the previous example to a Parquet file using write.df.

-
write.df(people, path = "people.parquet", source = "parquet", mode = "overwrite")
-
-
-

Hive Tables

-

You can also create SparkDataFrames from Hive tables. To do this we will need to create a SparkSession with Hive support which can access tables in the Hive MetaStore. Note that Spark should have been built with Hive support and more details can be found in the SQL programming guide. In SparkR, by default it will attempt to create a SparkSession with Hive support enabled (enableHiveSupport = TRUE).

-
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
-
-txtPath <- paste0(Sys.getenv("SPARK_HOME"), "/examples/src/main/resources/kv1.txt")
-sqlCMD <- sprintf("LOAD DATA LOCAL INPATH '%s' INTO TABLE src", txtPath)
-sql(sqlCMD)
-
-results <- sql("FROM src SELECT key, value")
-
-# results is now a SparkDataFrame
-head(results)
-
-
-
-

Data Processing

-

To dplyr users: SparkR has similar interface as dplyr in data processing. However, some noticeable differences are worth mentioning in the first place. We use df to represent a SparkDataFrame and col to represent the name of column here.

-
    -

  1. indicate columns. SparkR uses either a character string of the column name or a Column object constructed with $ to indicate a column. For example, to select col in df, we can write select(df, "col") or select(df, df$col).

    2. -

  2. describe conditions. In SparkR, the Column object representation can be inserted into the condition directly, or we can use a character string to describe the condition, without referring to the SparkDataFrame used. For example, to select rows with value > 1, we can write filter(df, df$col > 1) or filter(df, "col > 1").

    3. -
-

Here are more concrete examples.

- ---- - - - - - - - - - - - - - - - - -
dplyrSparkR
select(mtcars, mpg, hp)select(carsDF, "mpg", "hp")
filter(mtcars, mpg > 20, hp > 100)filter(carsDF, carsDF$mpg > 20, carsDF$hp > 100)
-

Other differences will be mentioned in the specific methods.

-

We use the SparkDataFrame carsDF created above. We can get basic information about the SparkDataFrame.

-
carsDF
-
## SparkDataFrame[model:string, mpg:double, cyl:double, disp:double, hp:double, drat:double, wt:double, qsec:double, vs:double, am:double, gear:double, carb:double]
-

Print out the schema in tree format.

-
printSchema(carsDF)
-
## root
-##  |-- model: string (nullable = true)
-##  |-- mpg: double (nullable = true)
-##  |-- cyl: double (nullable = true)
-##  |-- disp: double (nullable = true)
-##  |-- hp: double (nullable = true)
-##  |-- drat: double (nullable = true)
-##  |-- wt: double (nullable = true)
-##  |-- qsec: double (nullable = true)
-##  |-- vs: double (nullable = true)
-##  |-- am: double (nullable = true)
-##  |-- gear: double (nullable = true)
-##  |-- carb: double (nullable = true)
-
-

SparkDataFrame Operations

-
-

Selecting rows, columns

-

SparkDataFrames support a number of functions to do structured data processing. Here we include some basic examples and a complete list can be found in the API docs:

-

You can also pass in column name as strings.

-
head(select(carsDF, "mpg"))
-
##    mpg
-## 1 21.0
-## 2 21.0
-## 3 22.8
-## 4 21.4
-## 5 18.7
-## 6 18.1
-

Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins.

-
head(filter(carsDF, carsDF$mpg < 20))
-
##               model  mpg cyl  disp  hp drat   wt  qsec vs am gear carb
-## 1 Hornet Sportabout 18.7   8 360.0 175 3.15 3.44 17.02  0  0    3    2
-## 2           Valiant 18.1   6 225.0 105 2.76 3.46 20.22  1  0    3    1
-## 3        Duster 360 14.3   8 360.0 245 3.21 3.57 15.84  0  0    3    4
-## 4          Merc 280 19.2   6 167.6 123 3.92 3.44 18.30  1  0    4    4
-## 5         Merc 280C 17.8   6 167.6 123 3.92 3.44 18.90  1  0    4    4
-## 6        Merc 450SE 16.4   8 275.8 180 3.07 4.07 17.40  0  0    3    3
-
-
-

Grouping, Aggregation

-

A common flow of grouping and aggregation is

-
    -

  1. Use groupBy or group_by with respect to some grouping variables to create a GroupedData object

    2. -

  2. Feed the GroupedData object to agg or summarize functions, with some provided aggregation functions to compute a number within each group.

    3. -
-

A number of widely used functions are supported to aggregate data after grouping, including avg, countDistinct, count, first, kurtosis, last, max, mean, min, sd, skewness, stddev_pop, stddev_samp, sumDistinct, sum, var_pop, var_samp, var.

-

For example we can compute a histogram of the number of cylinders in the mtcars dataset as shown below.

-
numCyl <- summarize(groupBy(carsDF, carsDF$cyl), count = n(carsDF$cyl))
-head(numCyl)
-
##   cyl count
-## 1   8    14
-## 2   4    11
-## 3   6     7
-
-
-

Operating on Columns

-

SparkR also provides a number of functions that can directly applied to columns for data processing and during aggregation. The example below shows the use of basic arithmetic functions.

-
carsDF_km <- carsDF
-carsDF_km$kmpg <- carsDF_km$mpg * 1.61
-head(select(carsDF_km, "model", "mpg", "kmpg"))
-
##               model  mpg   kmpg
-## 1         Mazda RX4 21.0 33.810
-## 2     Mazda RX4 Wag 21.0 33.810
-## 3        Datsun 710 22.8 36.708
-## 4    Hornet 4 Drive 21.4 34.454
-## 5 Hornet Sportabout 18.7 30.107
-## 6           Valiant 18.1 29.141
-
-
-
-

Window Functions

-

A window function is a variation of aggregation function. In simple words,

-
    -

  • aggregation function: n to 1 mapping - returns a single value for a group of entries. Examples include sum, count, max.

    • -

  • window function: n to n mapping - returns one value for each entry in the group, but the value may depend on all the entries of the group. Examples include rank, lead, lag.

    • -
-

Formally, the group we mentioned is called the Frame. Every input row can have a unique frame associated with it and the output of the window function on that row is based on the rows confined in that frame.

-

Window functions are often used in conjunction with the following functions: windowPartitionBy, windowOrderBy, partitionBy, orderBy, over. It would be easier to look at an example.

-

We still use the mtcars dataset. The corresponding SparkDataFrame is carsDF. Suppose for each number of cylinders, we want to calculate the rank of each car in mpg within the group.

-
carsSubDF <- select(carsDF, "model", "mpg", "cyl")
-ws <- orderBy(windowPartitionBy("cyl"), "mpg")
-carsRank <- withColumn(carsSubDF, "rank", over(rank(), ws))
-showDF(carsRank)
-
## +-------------------+----+---+----+
-## |              model| mpg|cyl|rank|
-## +-------------------+----+---+----+
-## | Cadillac Fleetwood|10.4|8.0|   1|
-## |Lincoln Continental|10.4|8.0|   1|
-## |         Camaro Z28|13.3|8.0|   3|
-## |         Duster 360|14.3|8.0|   4|
-## |  Chrysler Imperial|14.7|8.0|   5|
-## |      Maserati Bora|15.0|8.0|   6|
-## |        Merc 450SLC|15.2|8.0|   7|
-## |        AMC Javelin|15.2|8.0|   7|
-## |   Dodge Challenger|15.5|8.0|   9|
-## |     Ford Pantera L|15.8|8.0|  10|
-## |         Merc 450SE|16.4|8.0|  11|
-## |         Merc 450SL|17.3|8.0|  12|
-## |  Hornet Sportabout|18.7|8.0|  13|
-## |   Pontiac Firebird|19.2|8.0|  14|
-## |         Volvo 142E|21.4|4.0|   1|
-## |      Toyota Corona|21.5|4.0|   2|
-## |         Datsun 710|22.8|4.0|   3|
-## |           Merc 230|22.8|4.0|   3|
-## |          Merc 240D|24.4|4.0|   5|
-## |      Porsche 914-2|26.0|4.0|   6|
-## +-------------------+----+---+----+
-## only showing top 20 rows
-

We explain in detail the above steps.

-
    -
  • windowPartitionBy creates a Window Specification object WindowSpec that defines the partition. It controls which rows will be in the same partition as the given row. In this case, rows with the same value in cyl will be put in the same partition. orderBy further defines the ordering - the position a given row is in the partition. The resulting WindowSpec is returned as ws.
    • -
-

More Window Specification methods include rangeBetween, which can define boundaries of the frame by value, and rowsBetween, which can define the boundaries by row indices.

-
    -
  • withColumn appends a Column called "rank" to the SparkDataFrame. over returns a windowing column. The first argument is usually a Column returned by window function(s) such as rank(), lead(carsDF$wt). That calculates the corresponding values according to the partitioned-and-ordered table.
    • -
-
-
-

User-Defined Function

-

In SparkR, we support several kinds of User-Defined Functions.

-
-

Apply by Partition

-

dapply can apply a function to each partition of a SparkDataFrame. The function to be applied to each partition of the SparkDataFrame should have only one parameter, a data.frame corresponding to a partition, and the output should be a data.frame as well. Schema specifies the row format of the resulting a SparkDataFrame. It must match to data types of returned value. See here for mapping between R and Spark.

-

We convert mpg to kmpg (kilometers per gallon). carsSubDF is a SparkDataFrame with a subset of carsDF columns.

-
carsSubDF <- select(carsDF, "model", "mpg")
-schema <- structType(structField("model", "string"), structField("mpg", "double"),
-                     structField("kmpg", "double"))
-out <- dapply(carsSubDF, function(x) { x <- cbind(x, x$mpg * 1.61) }, schema)
-head(collect(out))
-
##               model  mpg   kmpg
-## 1         Mazda RX4 21.0 33.810
-## 2     Mazda RX4 Wag 21.0 33.810
-## 3        Datsun 710 22.8 36.708
-## 4    Hornet 4 Drive 21.4 34.454
-## 5 Hornet Sportabout 18.7 30.107
-## 6           Valiant 18.1 29.141
-

Like dapply, apply a function to each partition of a SparkDataFrame and collect the result back. The output of function should be a data.frame. But, Schema is not required to be passed. Note that dapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory.

-
out <- dapplyCollect(
-         carsSubDF,
-         function(x) {
-           x <- cbind(x, "kmpg" = x$mpg * 1.61)
-         })
-head(out, 3)
-
##           model  mpg   kmpg
-## 1     Mazda RX4 21.0 33.810
-## 2 Mazda RX4 Wag 21.0 33.810
-## 3    Datsun 710 22.8 36.708
-
-
-

Apply by Group

-

gapply can apply a function to each group of a SparkDataFrame. The function is to be applied to each group of the SparkDataFrame and should have only two parameters: grouping key and R data.frame corresponding to that key. The groups are chosen from SparkDataFrames column(s). The output of function should be a data.frame. Schema specifies the row format of the resulting SparkDataFrame. It must represent R function’s output schema on the basis of Spark data types. The column names of the returned data.frame are set by user. See here for mapping between R and Spark.

-
schema <- structType(structField("cyl", "double"), structField("max_mpg", "double"))
-result <- gapply(
-    carsDF,
-    "cyl",
-    function(key, x) {
-        y <- data.frame(key, max(x$mpg))
-    },
-    schema)
-head(arrange(result, "max_mpg", decreasing = TRUE))
-
##   cyl max_mpg
-## 1   4    33.9
-## 2   6    21.4
-## 3   8    19.2
-

Like gapply, gapplyCollect applies a function to each partition of a SparkDataFrame and collect the result back to R data.frame. The output of the function should be a data.frame. But, the schema is not required to be passed. Note that gapplyCollect can fail if the output of UDF run on all the partition cannot be pulled to the driver and fit in driver memory.

-
result <- gapplyCollect(
-    carsDF,
-    "cyl",
-    function(key, x) {
-         y <- data.frame(key, max(x$mpg))
-        colnames(y) <- c("cyl", "max_mpg")
-        y
-    })
-head(result[order(result$max_mpg, decreasing = TRUE), ])
-
##   cyl max_mpg
-## 2   4    33.9
-## 3   6    21.4
-## 1   8    19.2
-
-
-

Distribute Local Functions

-

Similar to lapply in native R, spark.lapply runs a function over a list of elements and distributes the computations with Spark. Applies a function in a manner that is similar to doParallel or lapply to elements of a list. The results of all the computations should fit in a single machine. If that is not the case they can do something like df <- createDataFrame(list) and then use dapply.

-
families <- c("gaussian", "poisson")
-train <- function(family) {
-  model <- glm(mpg ~ hp, mtcars, family = family)
-  summary(model)
-}
-

Return a list of model’s summaries.

-
model.summaries <- spark.lapply(families, train)
-

Print the summary of each model.

-
print(model.summaries)
-
## [[1]]
-## 
-## Call:
-## glm(formula = mpg ~ hp, family = family, data = mtcars)
-## 
-## Deviance Residuals: 
-##     Min       1Q   Median       3Q      Max  
-## -5.7121  -2.1122  -0.8854   1.5819   8.2360  
-## 
-## Coefficients:
-##             Estimate Std. Error t value Pr(>|t|)    
-## (Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
-## hp          -0.06823    0.01012  -6.742 1.79e-07 ***
-## ---
-## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
-## 
-## (Dispersion parameter for gaussian family taken to be 14.92248)
-## 
-##     Null deviance: 1126.05  on 31  degrees of freedom
-## Residual deviance:  447.67  on 30  degrees of freedom
-## AIC: 181.24
-## 
-## Number of Fisher Scoring iterations: 2
-## 
-## 
-## [[2]]
-## 
-## Call:
-## glm(formula = mpg ~ hp, family = family, data = mtcars)
-## 
-## Deviance Residuals: 
-##     Min       1Q   Median       3Q      Max  
-## -1.4179  -0.4656  -0.1878   0.3935   1.6642  
-## 
-## Coefficients:
-##               Estimate Std. Error z value Pr(>|z|)    
-## (Intercept)  3.5205346  0.0937489  37.553  < 2e-16 ***
-## hp          -0.0037514  0.0006481  -5.788 7.12e-09 ***
-## ---
-## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
-## 
-## (Dispersion parameter for poisson family taken to be 1)
-## 
-##     Null deviance: 54.524  on 31  degrees of freedom
-## Residual deviance: 18.510  on 30  degrees of freedom
-## AIC: Inf
-## 
-## Number of Fisher Scoring iterations: 4
-
-
-
-

SQL Queries

-

A SparkDataFrame can also be registered as a temporary view in Spark SQL and that allows you to run SQL queries over its data. The sql function enables applications to run SQL queries programmatically and returns the result as a SparkDataFrame.

-
people <- read.df(paste0(Sys.getenv("SPARK_HOME"), 
-                         "/examples/src/main/resources/people.json"), "json")
-

Register this SparkDataFrame as a temporary view.

-
createOrReplaceTempView(people, "people")
-

SQL statements can be run by using the sql method.

-
teenagers <- sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
-head(teenagers)
-
##     name
-## 1 Justin
-
-
-
-

Machine Learning

-

SparkR supports the following machine learning models and algorithms.

-
    -

  • Generalized Linear Model (GLM)

    • -

  • Naive Bayes Model

    • -

  • \(k\)-means Clustering

    • -

  • Accelerated Failure Time (AFT) Survival Model

    • -

  • Gaussian Mixture Model (GMM)

    • -

  • Latent Dirichlet Allocation (LDA)

    • -

  • Multilayer Perceptron Model

    • -

  • Collaborative Filtering with Alternating Least Squares (ALS)

    • -

  • Isotonic Regression Model

    • -
-

More will be added in the future.

-
-

R Formula

-

For most above, SparkR supports R formula operators, including ~, ., :, + and - for model fitting. This makes it a similar experience as using R functions.

-
-
-

Training and Test Sets

-

We can easily split SparkDataFrame into random training and test sets by the randomSplit function. It returns a list of split SparkDataFrames with provided weights. We use carsDF as an example and want to have about \(70%\) training data and \(30%\) test data.

-
splitDF_list <- randomSplit(carsDF, c(0.7, 0.3), seed = 0)
-carsDF_train <- splitDF_list[[1]]
-carsDF_test <- splitDF_list[[2]]
-
count(carsDF_train)
-
## [1] 21
-
head(carsDF_train)
-
##                model  mpg cyl  disp  hp drat    wt  qsec vs am gear carb
-## 1 Cadillac Fleetwood 10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
-## 2         Camaro Z28 13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
-## 3         Duster 360 14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
-## 4           Fiat 128 32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
-## 5          Fiat X1-9 27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
-## 6     Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
-
count(carsDF_test)
-
## [1] 11
-
head(carsDF_test)
-
##               model  mpg cyl disp  hp drat    wt  qsec vs am gear carb
-## 1       AMC Javelin 15.2   8  304 150 3.15 3.435 17.30  0  0    3    2
-## 2 Chrysler Imperial 14.7   8  440 230 3.23 5.345 17.42  0  0    3    4
-## 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
-## 4  Dodge Challenger 15.5   8  318 150 2.76 3.520 16.87  0  0    3    2
-## 5      Ferrari Dino 19.7   6  145 175 3.62 2.770 15.50  0  1    5    6
-## 6     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
-
-
-

Models and Algorithms

-
-

Generalized Linear Model

-

The main function is spark.glm. The following families and link functions are supported. The default is gaussian.

- - - - - - - - - - - - - - - - - - - - - - - - - -
FamilyLink Function
gaussianidentity, log, inverse
binomiallogit, probit, cloglog (complementary log-log)
poissonlog, identity, sqrt
gammainverse, identity, log
-

There are three ways to specify the family argument.

-
    -

  • Family name as a character string, e.g. family = "gaussian".

    • -

  • Family function, e.g. family = binomial.

    • -

  • Result returned by a family function, e.g. family = poisson(link = log)

    • -
-

For more information regarding the families and their link functions, see the Wikipedia page Generalized Linear Model.

-

We use the mtcars dataset as an illustration. The corresponding SparkDataFrame is carsDF. After fitting the model, we print out a summary and see the fitted values by making predictions on the original dataset. We can also pass into a new SparkDataFrame of same schema to predict on new data.

-
gaussianGLM <- spark.glm(carsDF, mpg ~ wt + hp)
-summary(gaussianGLM)
-
## 
-## Deviance Residuals: 
-## (Note: These are approximate quantiles with relative error <= 0.01)
-##     Min       1Q   Median       3Q      Max  
-## -3.9410  -1.6499  -0.3267   1.0373   5.8538  
-## 
-## Coefficients:
-##              Estimate   Std. Error  t value  Pr(>|t|)  
-## (Intercept)  37.227     1.5988      23.285   0         
-## wt           -3.8778    0.63273     -6.1287  1.1196e-06
-## hp           -0.031773  0.0090297   -3.5187  0.0014512 
-## 
-## (Dispersion parameter for gaussian family taken to be 6.725785)
-## 
-##     Null deviance: 1126.05  on 31  degrees of freedom
-## Residual deviance:  195.05  on 29  degrees of freedom
-## AIC: 156.7
-## 
-## Number of Fisher Scoring iterations: 1
-

When doing prediction, a new column called prediction will be appended. Let’s look at only a subset of columns here.

-
gaussianFitted <- predict(gaussianGLM, carsDF)
-head(select(gaussianFitted, "model", "prediction", "mpg", "wt", "hp"))
-
##               model prediction  mpg    wt  hp
-## 1         Mazda RX4   23.57233 21.0 2.620 110
-## 2     Mazda RX4 Wag   22.58348 21.0 2.875 110
-## 3        Datsun 710   25.27582 22.8 2.320  93
-## 4    Hornet 4 Drive   21.26502 21.4 3.215 110
-## 5 Hornet Sportabout   18.32727 18.7 3.440 175
-## 6           Valiant   20.47382 18.1 3.460 105
-
-
-

Naive Bayes Model

-

Naive Bayes model assumes independence among the features. spark.naiveBayes fits a Bernoulli naive Bayes model against a SparkDataFrame. The data should be all categorical. These models are often used for document classification.

-
titanic <- as.data.frame(Titanic)
-titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5])
-naiveBayesModel <- spark.naiveBayes(titanicDF, Survived ~ Class + Sex + Age)
-summary(naiveBayesModel)
-
## $apriori
-##            Yes        No
-## [1,] 0.5769231 0.4230769
-## 
-## $tables
-##     Class_3rd Class_1st Class_2nd Sex_Male Age_Adult
-## Yes 0.3125    0.3125    0.3125    0.5      0.5625   
-## No  0.4166667 0.25      0.25      0.5      0.75
-
naiveBayesPrediction <- predict(naiveBayesModel, titanicDF)
-showDF(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "prediction"))
-
## +-----+------+-----+--------+----------+
-## |Class|   Sex|  Age|Survived|prediction|
-## +-----+------+-----+--------+----------+
-## |  3rd|  Male|Child|      No|       Yes|
-## |  3rd|Female|Child|      No|       Yes|
-## |  1st|  Male|Adult|      No|       Yes|
-## |  2nd|  Male|Adult|      No|       Yes|
-## |  3rd|  Male|Adult|      No|        No|
-## | Crew|  Male|Adult|      No|       Yes|
-## |  1st|Female|Adult|      No|       Yes|
-## |  2nd|Female|Adult|      No|       Yes|
-## |  3rd|Female|Adult|      No|        No|
-## | Crew|Female|Adult|      No|       Yes|
-## |  1st|  Male|Child|     Yes|       Yes|
-## |  2nd|  Male|Child|     Yes|       Yes|
-## |  3rd|  Male|Child|     Yes|       Yes|
-## |  1st|Female|Child|     Yes|       Yes|
-## |  2nd|Female|Child|     Yes|       Yes|
-## |  3rd|Female|Child|     Yes|       Yes|
-## |  1st|  Male|Adult|     Yes|       Yes|
-## |  2nd|  Male|Adult|     Yes|       Yes|
-## |  3rd|  Male|Adult|     Yes|        No|
-## | Crew|  Male|Adult|     Yes|       Yes|
-## +-----+------+-----+--------+----------+
-## only showing top 20 rows
-
-
-

k-Means Clustering

-

spark.kmeans fits a \(k\)-means clustering model against a SparkDataFrame. As an unsupervised learning method, we don’t need a response variable. Hence, the left hand side of the R formula should be left blank. The clustering is based only on the variables on the right hand side.

-
kmeansModel <- spark.kmeans(carsDF, ~ mpg + hp + wt, k = 3)
-summary(kmeansModel)
-
## $coefficients
-##   mpg      hp       wt      
-## 1 24.22353 93.52941 2.599588
-## 2 14.62    263.8    3.899   
-## 3 15.8     178.5    3.9264  
-## 
-## $size
-## $size[[1]]
-## [1] 17
-## 
-## $size[[2]]
-## [1] 5
-## 
-## $size[[3]]
-## [1] 10
-## 
-## 
-## $cluster
-## SparkDataFrame[prediction:int]
-## 
-## $is.loaded
-## [1] FALSE
-
kmeansPredictions <- predict(kmeansModel, carsDF)
-showDF(select(kmeansPredictions, "model", "mpg", "hp", "wt", "prediction"))
-
## +-------------------+----+-----+-----+----------+
-## |              model| mpg|   hp|   wt|prediction|
-## +-------------------+----+-----+-----+----------+
-## |          Mazda RX4|21.0|110.0| 2.62|         0|
-## |      Mazda RX4 Wag|21.0|110.0|2.875|         0|
-## |         Datsun 710|22.8| 93.0| 2.32|         0|
-## |     Hornet 4 Drive|21.4|110.0|3.215|         0|
-## |  Hornet Sportabout|18.7|175.0| 3.44|         2|
-## |            Valiant|18.1|105.0| 3.46|         0|
-## |         Duster 360|14.3|245.0| 3.57|         1|
-## |          Merc 240D|24.4| 62.0| 3.19|         0|
-## |           Merc 230|22.8| 95.0| 3.15|         0|
-## |           Merc 280|19.2|123.0| 3.44|         0|
-## |          Merc 280C|17.8|123.0| 3.44|         0|
-## |         Merc 450SE|16.4|180.0| 4.07|         2|
-## |         Merc 450SL|17.3|180.0| 3.73|         2|
-## |        Merc 450SLC|15.2|180.0| 3.78|         2|
-## | Cadillac Fleetwood|10.4|205.0| 5.25|         2|
-## |Lincoln Continental|10.4|215.0|5.424|         2|
-## |  Chrysler Imperial|14.7|230.0|5.345|         1|
-## |           Fiat 128|32.4| 66.0|  2.2|         0|
-## |        Honda Civic|30.4| 52.0|1.615|         0|
-## |     Toyota Corolla|33.9| 65.0|1.835|         0|
-## +-------------------+----+-----+-----+----------+
-## only showing top 20 rows
-
-
-

AFT Survival Model

-

Survival analysis studies the expected duration of time until an event happens, and often the relationship with risk factors or treatment taken on the subject. In contrast to standard regression analysis, survival modeling has to deal with special characteristics in the data including non-negative survival time and censoring.

-

Accelerated Failure Time (AFT) model is a parametric survival model for censored data that assumes the effect of a covariate is to accelerate or decelerate the life course of an event by some constant. For more information, refer to the Wikipedia page AFT Model and the references there. Different from a Proportional Hazards Model designed for the same purpose, the AFT model is easier to parallelize because each instance contributes to the objective function independently.

-
library(survival)
-ovarianDF <- createDataFrame(ovarian)
-
## Warning in FUN(X[[i]], ...): Use resid_ds instead of resid.ds as column
-## name
-
## Warning in FUN(X[[i]], ...): Use ecog_ps instead of ecog.ps as column name
-
aftModel <- spark.survreg(ovarianDF, Surv(futime, fustat) ~ ecog_ps + rx)
-summary(aftModel)
-
## $coefficients
-##                  Value
-## (Intercept)  6.8966930
-## ecog_ps     -0.3850426
-## rx           0.5286457
-## Log(scale)  -0.1234418
-
aftPredictions <- predict(aftModel, ovarianDF)
-head(aftPredictions)
-
##   futime fustat     age resid_ds rx ecog_ps label prediction
-## 1     59      1 72.3315        2  1       1    59  1141.7256
-## 2    115      1 74.4932        2  1       1   115  1141.7256
-## 3    156      1 66.4658        2  1       2   156   776.8548
-## 4    421      0 53.3644        2  2       1   421  1937.0893
-## 5    431      1 50.3397        2  1       1   431  1141.7256
-## 6    448      0 56.4301        1  1       2   448   776.8548
-
-
-

Gaussian Mixture Model

-

spark.gaussianMixture fits multivariate Gaussian Mixture Model (GMM) against a SparkDataFrame. Expectation-Maximization (EM) is used to approximate the maximum likelihood estimator (MLE) of the model.

-

We use a simulated example to demostrate the usage.

-
X1 <- data.frame(V1 = rnorm(4), V2 = rnorm(4))
-X2 <- data.frame(V1 = rnorm(6, 3), V2 = rnorm(6, 4))
-data <- rbind(X1, X2)
-df <- createDataFrame(data)
-gmmModel <- spark.gaussianMixture(df, ~ V1 + V2, k = 2)
-summary(gmmModel)
-
## $lambda
-## [1] 0.4000004 0.5999996
-## 
-## $mu
-## $mu[[1]]
-## [1] 0.8159244 0.4405515
-## 
-## $mu[[2]]
-## [1] 3.034007 4.475828
-## 
-## 
-## $sigma
-## $sigma[[1]]
-##      [,1]        [,2]       
-## [1,] 0.4763343   -0.09395206
-## [2,] -0.09395206 0.8492281  
-## 
-## $sigma[[2]]
-##      [,1]       [,2]      
-## [1,] 1.116189   -0.3408798
-## [2,] -0.3408798 0.3243061 
-## 
-## 
-## $posterior
-## SparkDataFrame[posterior:array<double>]
-## 
-## $is.loaded
-## [1] FALSE
-
gmmFitted <- predict(gmmModel, df)
-showDF(select(gmmFitted, "V1", "V2", "prediction"))
-
## +-------------------+-------------------+----------+
-## |                 V1|                 V2|prediction|
-## +-------------------+-------------------+----------+
-## | 0.7212924990296785| 0.6804778552461263|         0|
-## | 1.7023597530257268|-0.7515664802963296|         0|
-## |-0.2112969944430051|0.06082658268951577|         0|
-## |  1.051335125612143|  1.772454161411131|         0|
-## | 3.4068107905235134| 5.0720086203141435|         1|
-## | 2.4434779156754383| 4.3827744683163985|         1|
-## |  5.099806471262699| 3.8553621051014937|         1|
-## | 2.6660451400733596|  4.336997228018436|         1|
-## | 1.6976557727358053|  5.360445392662975|         1|
-## | 2.8902452466769284| 3.8473802024470407|         1|
-## +-------------------+-------------------+----------+
-
-
-

Latent Dirichlet Allocation

-

spark.lda fits a Latent Dirichlet Allocation model on a SparkDataFrame. It is often used in topic modeling in which topics are inferred from a collection of text documents. LDA can be thought of as a clustering algorithm as follows:

-
    -

  • Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset.

    • -

  • Topics and documents both exist in a feature space, where feature vectors are vectors of word counts (bag of words).

    • -

  • Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated.

    • -
-

To use LDA, we need to specify a features column in data where each entry represents a document. There are two type options for the column:

-
    -

  • character string: This can be a string of the whole document. It will be parsed automatically. Additional stop words can be added in customizedStopWords.

    • -

  • libSVM: Each entry is a collection of words and will be processed directly.

    • -
-

There are several parameters LDA takes for fitting the model.

-
    -

  • k: number of topics (default 10).

    • -

  • maxIter: maximum iterations (default 20).

    • -

  • optimizer: optimizer to train an LDA model, “online” (default) uses online variational inference. “em” uses expectation-maximization.

    • -

  • subsamplingRate: For optimizer = "online". Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1] (default 0.05).

    • -

  • topicConcentration: concentration parameter (commonly named beta or eta) for the prior placed on topic distributions over terms, default -1 to set automatically on the Spark side. Use summary to retrieve the effective topicConcentration. Only 1-size numeric is accepted.

    • -

  • docConcentration: concentration parameter (commonly named alpha) for the prior placed on documents distributions over topics (theta), default -1 to set automatically on the Spark side. Use summary to retrieve the effective docConcentration. Only 1-size or k-size numeric is accepted.

    • -

  • maxVocabSize: maximum vocabulary size, default 1 << 18.

    • -
-

Two more functions are provided for the fitted model.

-
    -

  • spark.posterior returns a SparkDataFrame containing a column of posterior probabilities vectors named “topicDistribution”.

    • -

  • spark.perplexity returns the log perplexity of given SparkDataFrame, or the log perplexity of the training data if missing argument data.

    • -
-

For more information, see the help document ?spark.lda.

-

Let’s look an artificial example.

-
corpus <- data.frame(features = c(
-  "1 2 6 0 2 3 1 1 0 0 3",
-  "1 3 0 1 3 0 0 2 0 0 1",
-  "1 4 1 0 0 4 9 0 1 2 0",
-  "2 1 0 3 0 0 5 0 2 3 9",
-  "3 1 1 9 3 0 2 0 0 1 3",
-  "4 2 0 3 4 5 1 1 1 4 0",
-  "2 1 0 3 0 0 5 0 2 2 9",
-  "1 1 1 9 2 1 2 0 0 1 3",
-  "4 4 0 3 4 2 1 3 0 0 0",
-  "2 8 2 0 3 0 2 0 2 7 2",
-  "1 1 1 9 0 2 2 0 0 3 3",
-  "4 1 0 0 4 5 1 3 0 1 0"))
-corpusDF <- createDataFrame(corpus)
-model <- spark.lda(data = corpusDF, k = 5, optimizer = "em")
-summary(model)
-
## $docConcentration
-## [1] 0.1666620 0.2475245 0.1666316 0.1666351 0.1680889
-## 
-## $topicConcentration
-## [1] 0.2
-## 
-## $logLikelihood
-## [1] -318.7387
-## 
-## $logPerplexity
-## [1] 2.414687
-## 
-## $isDistributed
-## [1] FALSE
-## 
-## $vocabSize
-## [1] 10
-## 
-## $topics
-## SparkDataFrame[topic:int, term:array<string>, termWeights:array<double>]
-## 
-## $vocabulary
-##  [1] "0" "1" "2" "3" "4" "9" "5" "8" "7" "6"
-
posterior <- spark.posterior(model, corpusDF)
-head(posterior)
-
##                features
-## 1 1 2 6 0 2 3 1 1 0 0 3
-## 2 1 3 0 1 3 0 0 2 0 0 1
-## 3 1 4 1 0 0 4 9 0 1 2 0
-## 4 2 1 0 3 0 0 5 0 2 3 9
-## 5 3 1 1 9 3 0 2 0 0 1 3
-## 6 4 2 0 3 4 5 1 1 1 4 0
-##                                            topicDistribution
-## 1 0.01409431, 0.94353632, 0.01407450, 0.01409065, 0.01420422
-## 2 0.01402469, 0.94375465, 0.01402782, 0.01402870, 0.01416414
-## 3 0.01406876, 0.94356690, 0.01405973, 0.01405508, 0.01424953
-## 4 0.01404847, 0.94367008, 0.01405272, 0.01404966, 0.01417907
-## 5 0.01403980, 0.94370443, 0.01404162, 0.01404285, 0.01417130
-## 6 0.01409286, 0.94340534, 0.01409602, 0.01408882, 0.01431697
-
perplexity <- spark.perplexity(model, corpusDF)
-perplexity
-
## [1] 2.414687
-
-
-

Multilayer Perceptron

-

Multilayer perceptron classifier (MLPC) is a classifier based on the feedforward artificial neural network. MLPC consists of multiple layers of nodes. Each layer is fully connected to the next layer in the network. Nodes in the input layer represent the input data. All other nodes map inputs to outputs by a linear combination of the inputs with the node’s weights \(w\) and bias \(b\) and applying an activation function. This can be written in matrix form for MLPC with \(K+1\) layers as follows: \[ -y(x)=f_K(\ldots f_2(w_2^T f_1(w_1^T x + b_1) + b_2) \ldots + b_K). -\]

-

Nodes in intermediate layers use sigmoid (logistic) function: \[ -f(z_i) = \frac{1}{1+e^{-z_i}}. -\]

-

Nodes in the output layer use softmax function: \[ -f(z_i) = \frac{e^{z_i}}{\sum_{k=1}^N e^{z_k}}. -\]

-

The number of nodes \(N\) in the output layer corresponds to the number of classes.

-

MLPC employs backpropagation for learning the model. We use the logistic loss function for optimization and L-BFGS as an optimization routine.

-

spark.mlp requires at least two columns in data: one named "label" and the other one "features". The "features" column should be in libSVM-format. According to the description above, there are several additional parameters that can be set:

-
    -

  • layers: integer vector containing the number of nodes for each layer.

    • -

  • solver: solver parameter, supported options: "gd" (minibatch gradient descent) or "l-bfgs".

    • -

  • maxIter: maximum iteration number.

    • -

  • tol: convergence tolerance of iterations.

    • -

  • stepSize: step size for "gd".

    • -

  • seed: seed parameter for weights initialization.

    • -
-
-
-

Collaborative Filtering

-

spark.als learns latent factors in collaborative filtering via alternating least squares.

-

There are multiple options that can be configured in spark.als, including rank, reg, nonnegative. For a complete list, refer to the help file.

-
ratings <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0),
-                list(2, 1, 1.0), list(2, 2, 5.0))
-df <- createDataFrame(ratings, c("user", "item", "rating"))
-model <- spark.als(df, "rating", "user", "item", rank = 10, reg = 0.1, nonnegative = TRUE)
-

Extract latent factors.

-
stats <- summary(model)
-userFactors <- stats$userFactors
-itemFactors <- stats$itemFactors
-head(userFactors)
-
##   id
-## 1  0
-## 2  1
-## 3  2
-##                                                                                                                           features
-## 1           0.36135060, 0.00000000, 0.14246520, 0.79222524, 0.25852802, 0.35938185, 1.02794230, 0.00000000, 0.40629458, 0.03039724
-## 2                     0.6459087, 0.0000000, 0.4240893, 0.5433040, 0.5990543, 0.2464622, 0.7952952, 0.0000000, 1.0376287, 0.3610190
-## 3 0.147648051, 0.000000000, 0.862470269, 0.008587426, 0.171784312, 0.003895568, 0.024497632, 0.000000000, 0.316428304, 1.357668996
-
head(itemFactors)
-
##   id
-## 1  0
-## 2  1
-## 3  2
-##                                                                                                                 features
-## 1 0.65061080, 0.00000000, 0.26098290, 1.38994920, 0.47113958, 0.63053089, 1.80735207, 0.00000000, 0.74440372, 0.06321633
-## 2           0.5822609, 0.0000000, 0.3250861, 0.4984071, 0.5374193, 0.2260954, 0.7286831, 0.0000000, 0.9294589, 0.2301433
-## 3           0.6116845, 0.0000000, 1.5209249, 0.3454843, 0.6182634, 0.1567241, 0.5231636, 0.0000000, 1.0985043, 2.2063246
-

Make predictions.

-
predicted <- predict(model, df)
-showDF(predicted)
-
## +----+----+------+----------+
-## |user|item|rating|prediction|
-## +----+----+------+----------+
-## | 1.0| 1.0|   3.0| 2.7894442|
-## | 2.0| 1.0|   1.0| 1.0882444|
-## | 0.0| 1.0|   2.0| 2.0054312|
-## | 1.0| 2.0|   4.0| 3.9892373|
-## | 2.0| 2.0|   5.0| 4.8677244|
-## | 0.0| 0.0|   4.0| 3.8840594|
-## +----+----+------+----------+
-
-
-

Isotonic Regression Model

-

spark.isoreg fits an Isotonic Regression model against a SparkDataFrame. It solves a weighted univariate a regression problem under a complete order constraint. Specifically, given a set of real observed responses \(y_1, \ldots, y_n\), corresponding real features \(x_1, \ldots, x_n\), and optionally positive weights \(w_1, \ldots, w_n\), we want to find a monotone (piecewise linear) function \(f\) to minimize \[ -\ell(f) = \sum_{i=1}^n w_i (y_i - f(x_i))^2. -\]

-

There are a few more arguments that may be useful.

-
    -

  • weightCol: a character string specifying the weight column.

    • -

  • isotonic: logical value indicating whether the output sequence should be isotonic/increasing (TRUE) or antitonic/decreasing (FALSE).

    • -

  • featureIndex: the index of the feature on the right hand side of the formula if it is a vector column (default: 0), no effect otherwise.

    • -
-

We use an artificial example to show the use.

-
y <- c(3.0, 6.0, 8.0, 5.0, 7.0)
-x <- c(1.0, 2.0, 3.5, 3.0, 4.0)
-w <- rep(1.0, 5)
-data <- data.frame(y = y, x = x, w = w)
-df <- createDataFrame(data)
-isoregModel <- spark.isoreg(df, y ~ x, weightCol = "w")
-isoregFitted <- predict(isoregModel, df)
-head(select(isoregFitted, "x", "y", "prediction"))
-
##     x y prediction
-## 1 1.0 3        3.0
-## 2 2.0 6        5.5
-## 3 3.5 8        7.5
-## 4 3.0 5        5.5
-## 5 4.0 7        7.5
-

In the prediction stage, based on the fitted monotone piecewise function, the rules are:

-
    -

  • If the prediction input exactly matches a training feature then associated prediction is returned. In case there are multiple predictions with the same feature then one of them is returned. Which one is undefined.

    • -

  • If the prediction input is lower or higher than all training features then prediction with lowest or highest feature is returned respectively. In case there are multiple predictions with the same feature then the lowest or highest is returned respectively.

    • -

  • If the prediction input falls between two training features then prediction is treated as piecewise linear function and interpolated value is calculated from the predictions of the two closest features. In case there are multiple values with the same feature then the same rules as in previous point are used.

    • -
-

For example, when the input is \(3.2\), the two closest feature values are \(3.0\) and \(3.5\), then predicted value would be a linear interpolation between the predicted values at \(3.0\) and \(3.5\).

-
newDF <- createDataFrame(data.frame(x = c(1.5, 3.2)))
-head(predict(isoregModel, newDF))
-
##     x prediction
-## 1 1.5       4.25
-## 2 3.2       6.30
-
-
-
-

Model Persistence

-

The following example shows how to save/load an ML model by SparkR.

-
irisDF <- suppressWarnings(createDataFrame(iris))
-gaussianGLM <- spark.glm(irisDF, Sepal_Length ~ Sepal_Width + Species, family = "gaussian")
-
-# Save and then load a fitted MLlib model
-modelPath <- tempfile(pattern = "ml", fileext = ".tmp")
-write.ml(gaussianGLM, modelPath)
-gaussianGLM2 <- read.ml(modelPath)
-
-# Check model summary
-summary(gaussianGLM2)
-
## 
-## Saved-loaded model does not support output 'Deviance Residuals'.
-## 
-## Coefficients:
-##                     Estimate  Std. Error  t value  Pr(>|t|)  
-## (Intercept)         2.2514    0.36975     6.0889   9.5681e-09
-## Sepal_Width         0.80356   0.10634     7.5566   4.1873e-12
-## Species_versicolor  1.4587    0.11211     13.012   0         
-## Species_virginica   1.9468    0.10001     19.465   0         
-## 
-## (Dispersion parameter for gaussian family taken to be 0.1918059)
-## 
-##     Null deviance: 102.168  on 149  degrees of freedom
-## Residual deviance:  28.004  on 146  degrees of freedom
-## AIC: 183.9
-## 
-## Number of Fisher Scoring iterations: 1
-
# Check model prediction
-gaussianPredictions <- predict(gaussianGLM2, irisDF)
-showDF(gaussianPredictions)
-
## +------------+-----------+------------+-----------+-------+-----+------------------+
-## |Sepal_Length|Sepal_Width|Petal_Length|Petal_Width|Species|label|        prediction|
-## +------------+-----------+------------+-----------+-------+-----+------------------+
-## |         5.1|        3.5|         1.4|        0.2| setosa|  5.1| 5.063856384860281|
-## |         4.9|        3.0|         1.4|        0.2| setosa|  4.9| 4.662075934441678|
-## |         4.7|        3.2|         1.3|        0.2| setosa|  4.7|  4.82278811460912|
-## |         4.6|        3.1|         1.5|        0.2| setosa|  4.6|4.7424320245253995|
-## |         5.0|        3.6|         1.4|        0.2| setosa|  5.0| 5.144212474944002|
-## |         5.4|        3.9|         1.7|        0.4| setosa|  5.4| 5.385280745195163|
-## |         4.6|        3.4|         1.4|        0.3| setosa|  4.6|  4.98350029477656|
-## |         5.0|        3.4|         1.5|        0.2| setosa|  5.0|  4.98350029477656|
-## |         4.4|        2.9|         1.4|        0.2| setosa|  4.4| 4.581719844357957|
-## |         4.9|        3.1|         1.5|        0.1| setosa|  4.9|4.7424320245253995|
-## |         5.4|        3.7|         1.5|        0.2| setosa|  5.4| 5.224568565027722|
-## |         4.8|        3.4|         1.6|        0.2| setosa|  4.8|  4.98350029477656|
-## |         4.8|        3.0|         1.4|        0.1| setosa|  4.8| 4.662075934441678|
-## |         4.3|        3.0|         1.1|        0.1| setosa|  4.3| 4.662075934441678|
-## |         5.8|        4.0|         1.2|        0.2| setosa|  5.8| 5.465636835278884|
-## |         5.7|        4.4|         1.5|        0.4| setosa|  5.7|5.7870611956137665|
-## |         5.4|        3.9|         1.3|        0.4| setosa|  5.4| 5.385280745195163|
-## |         5.1|        3.5|         1.4|        0.3| setosa|  5.1| 5.063856384860281|
-## |         5.7|        3.8|         1.7|        0.3| setosa|  5.7| 5.304924655111442|
-## |         5.1|        3.8|         1.5|        0.3| setosa|  5.1| 5.304924655111442|
-## +------------+-----------+------------+-----------+-------+-----+------------------+
-## only showing top 20 rows
-
unlink(modelPath)
-
-
-
-

Advanced Topics

-
-

SparkR Object Classes

-

There are three main object classes in SparkR you may be working with.

-
    -
  • SparkDataFrame: the central component of SparkR. It is an S4 class representing distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R. It has two slots sdf and env. -
      -
    • sdf stores a reference to the corresponding Spark Dataset in the Spark JVM backend.
      • -
    • env saves the meta-information of the object such as isCached.
      • -
    • -
-

It can be created by data import methods or by transforming an existing SparkDataFrame. We can manipulate SparkDataFrame by numerous data processing functions and feed that into machine learning algorithms.

-
    -
  • Column: an S4 class representing column of SparkDataFrame. The slot jc saves a reference to the corresponding Column object in the Spark JVM backend.
    • -
-

It can be obtained from a SparkDataFrame by $ operator, df$col. More often, it is used together with other functions, for example, with select to select particular columns, with filter and constructed conditions to select rows, with aggregation functions to compute aggregate statistics for each group.

-
    -
  • GroupedData: an S4 class representing grouped data created by groupBy or by transforming other GroupedData. Its sgd slot saves a reference to a RelationalGroupedDataset object in the backend.
    • -
-

This is often an intermediate object with group information and followed up by aggregation operations.

-
-
-

Architecture

-

A complete description of architecture can be seen in paper SparkR: Scaling R Programs with Spark, Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016.

-

Under the hood of SparkR is Spark SQL engine. This avoids the overheads of running interpreted R code, and the optimized SQL execution engine in Spark uses structural information about data and computation flow to perform a bunch of optimizations to speed up the computation.

-

The main method calls of actual computation happen in the Spark JVM of the driver. We have a socket-based SparkR API that allows us to invoke functions on the JVM from R. We use a SparkR JVM backend that listens on a Netty-based socket server.

-

Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and creating new objects. Method invocation can be done in two ways.

-
    -

  • invokeJMethod takes a reference to an existing Java object and a list of arguments to be passed on to the method.

    • -

  • invokeJStatic takes a class name for static method and a list of arguments to be passed on to the method.

    • -
-

The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method.

-

To create objects, a special method name init is used and then similarly the appropriate constructor is invoked with provided arguments.

-

Finally, we use a new R class jobj that refers to a Java object existing in the backend. These references are tracked on the Java side and are automatically garbage collected when they go out of scope on the R side.

-
-
-
-

Appendix

-
-

R and Spark Data Types

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RSpark
bytebyte
integerinteger
floatfloat
doubledouble
numericdouble
characterstring
stringstring
binarybinary
rawbinary
logicalboolean
POSIXcttimestamp
POSIXlttimestamp
Datedate
arrayarray
listarray
envmap
-
-
-
-

References

- -
- - - -
-
- -
- - - - - - - - diff --git a/R/pkg/vignettes/sparkr-vignettes.md b/R/pkg/vignettes/sparkr-vignettes.md deleted file mode 100644 index 1a23560869acd..0000000000000 --- a/R/pkg/vignettes/sparkr-vignettes.md +++ /dev/null @@ -1,222 +0,0 @@ -Untitled -================ - -Overview --------- - -SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using MLlib. - -Getting Started ---------------- - -We start with an example running on the local machine and provide an overview of SparkR in multiple dimensions: data ingestion, data processing and machine learning. - -First, let's load and attach the package. - -``` r -library(SparkR) -``` - -To use SparkR, you need an Apache Spark package where backend codes to be called are compiled and packaged. You may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. - -``` r -install.spark(overwrite = TRUE) -``` - -If you have a Spark package, you don't have to install again, but an environment variable should be set to let SparkR know where it is. If you have run the `install.spark` function, this has already been done for you. - -``` r -Sys.setenv(SPARK_HOME = "/HOME/spark") -``` - -`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any spark packages depended on, etc. We use default settings. - -``` r -sparkR.session() -``` - - ## Launching java with spark-submit command /Users/junyangq/spark//bin/spark-submit sparkr-shell /var/folders/jh/6pw_r0d51317krg8ftgy53f40000gn/T//RtmpT7vIHb/backend_portb8c54afe73fa - - ## Java ref type org.apache.spark.sql.SparkSession id 1 - -The operations in SparkR are centered around a class of R object called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood. - -`SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame, - -``` r -cars <- cbind(model = rownames(mtcars), mtcars) -carsDF <- createDataFrame(cars) -``` - -We can view the first few rows of the `SparkDataFrame` by `showDF` or `head` function. - -``` r -showDF(carsDF) -``` - - ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ - ## | model| mpg|cyl| disp| hp|drat| wt| qsec| vs| am|gear|carb| - ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ - ## | Mazda RX4|21.0|6.0|160.0|110.0| 3.9| 2.62|16.46|0.0|1.0| 4.0| 4.0| - ## | Mazda RX4 Wag|21.0|6.0|160.0|110.0| 3.9|2.875|17.02|0.0|1.0| 4.0| 4.0| - ## | Datsun 710|22.8|4.0|108.0| 93.0|3.85| 2.32|18.61|1.0|1.0| 4.0| 1.0| - ## | Hornet 4 Drive|21.4|6.0|258.0|110.0|3.08|3.215|19.44|1.0|0.0| 3.0| 1.0| - ## | Hornet Sportabout|18.7|8.0|360.0|175.0|3.15| 3.44|17.02|0.0|0.0| 3.0| 2.0| - ## | Valiant|18.1|6.0|225.0|105.0|2.76| 3.46|20.22|1.0|0.0| 3.0| 1.0| - ## | Duster 360|14.3|8.0|360.0|245.0|3.21| 3.57|15.84|0.0|0.0| 3.0| 4.0| - ## | Merc 240D|24.4|4.0|146.7| 62.0|3.69| 3.19| 20.0|1.0|0.0| 4.0| 2.0| - ## | Merc 230|22.8|4.0|140.8| 95.0|3.92| 3.15| 22.9|1.0|0.0| 4.0| 2.0| - ## | Merc 280|19.2|6.0|167.6|123.0|3.92| 3.44| 18.3|1.0|0.0| 4.0| 4.0| - ## | Merc 280C|17.8|6.0|167.6|123.0|3.92| 3.44| 18.9|1.0|0.0| 4.0| 4.0| - ## | Merc 450SE|16.4|8.0|275.8|180.0|3.07| 4.07| 17.4|0.0|0.0| 3.0| 3.0| - ## | Merc 450SL|17.3|8.0|275.8|180.0|3.07| 3.73| 17.6|0.0|0.0| 3.0| 3.0| - ## | Merc 450SLC|15.2|8.0|275.8|180.0|3.07| 3.78| 18.0|0.0|0.0| 3.0| 3.0| - ## | Cadillac Fleetwood|10.4|8.0|472.0|205.0|2.93| 5.25|17.98|0.0|0.0| 3.0| 4.0| - ## |Lincoln Continental|10.4|8.0|460.0|215.0| 3.0|5.424|17.82|0.0|0.0| 3.0| 4.0| - ## | Chrysler Imperial|14.7|8.0|440.0|230.0|3.23|5.345|17.42|0.0|0.0| 3.0| 4.0| - ## | Fiat 128|32.4|4.0| 78.7| 66.0|4.08| 2.2|19.47|1.0|1.0| 4.0| 1.0| - ## | Honda Civic|30.4|4.0| 75.7| 52.0|4.93|1.615|18.52|1.0|1.0| 4.0| 2.0| - ## | Toyota Corolla|33.9|4.0| 71.1| 65.0|4.22|1.835| 19.9|1.0|1.0| 4.0| 1.0| - ## +-------------------+----+---+-----+-----+----+-----+-----+---+---+----+----+ - ## only showing top 20 rows - -We use `magrittr` package to chain operations when necessary in the rest of the document. - -``` r -library(magrittr) -``` - -Common data processing operations such as `filter`, `select` are supported on the `SparkDataFrame`. - -``` r -carsSubDF <- select(carsDF, "model", "mpg", "hp") -carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200) -showDF(carsSubDF) -``` - - ## +-------------------+----+-----+ - ## | model| mpg| hp| - ## +-------------------+----+-----+ - ## | Duster 360|14.3|245.0| - ## | Cadillac Fleetwood|10.4|205.0| - ## |Lincoln Continental|10.4|215.0| - ## | Chrysler Imperial|14.7|230.0| - ## | Camaro Z28|13.3|245.0| - ## | Ford Pantera L|15.8|264.0| - ## | Maserati Bora|15.0|335.0| - ## +-------------------+----+-----+ - -SparkR support a number of commonly used functions to aggregate data after grouping. - -``` r -carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) -showDF(carsGPDF) -``` - - ## +----+-----+ - ## |gear|count| - ## +----+-----+ - ## | 4.0| 12| - ## | 3.0| 15| - ## | 5.0| 5| - ## +----+-----+ - -SparkR supports a number of widely used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models. - -SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. - -``` r -fit <- spark.glm(carsDF, mpg ~ wt + cyl) -``` - -``` r -summary(fit) -``` - - ## - ## Deviance Residuals: - ## (Note: These are approximate quantiles with relative error <= 0.01) - ## Min 1Q Median 3Q Max - ## -4.2893 -1.7085 -0.4713 1.5729 6.1004 - ## - ## Coefficients: - ## Estimate Std. Error t value Pr(>|t|) - ## (Intercept) 39.686 1.715 23.141 0 - ## wt -3.191 0.75691 -4.2158 0.00022202 - ## cyl -1.5078 0.41469 -3.636 0.0010643 - ## - ## (Dispersion parameter for gaussian family taken to be 6.592137) - ## - ## Null deviance: 1126.05 on 31 degrees of freedom - ## Residual deviance: 191.17 on 29 degrees of freedom - ## AIC: 156 - ## - ## Number of Fisher Scoring iterations: 1 - -``` r -sparkR.session.stop() -``` - -Vignettes are long form documentation commonly included in packages. Because they are part of the distribution of the package, they need to be as compact as possible. The `html_vignette` output type provides a custom style sheet (and tweaks some options) to ensure that the resulting html is as small as possible. The `html_vignette` format: - -- Never uses retina figures -- Has a smaller default figure size -- Uses a custom CSS stylesheet instead of the default Twitter Bootstrap style - -Vignette Info -------------- - -Note the various macros within the `vignette` section of the metadata block above. These are required in order to instruct R how to build the vignette. Note that you should change the `title` field and the `\VignetteIndexEntry` to match the title of your vignette. - -Styles ------- - -The `html_vignette` template includes a basic CSS theme. To override this theme you can specify your own CSS in the document metadata as follows: - - output: - rmarkdown::html_vignette: - css: mystyles.css - -Figures -------- - -The figure sizes have been customised so that you can easily put two images side-by-side. - -``` r -plot(1:10) -plot(10:1) -``` - -![](sparkr-vignettes_files/figure-markdown_github/unnamed-chunk-14-1.png)![](sparkr-vignettes_files/figure-markdown_github/unnamed-chunk-14-2.png) - -You can enable figure captions by `fig_caption: yes` in YAML: - - output: - rmarkdown::html_vignette: - fig_caption: yes - -Then you can use the chunk option `fig.cap = "Your figure caption."` in **knitr**. - -More Examples -------------- - -You can write math expressions, e.g. *Y* = *X**β* + *ϵ*, footnotes[1], and tables, e.g. using `knitr::kable()`. - -| | mpg| cyl| disp| hp| drat| wt| qsec| vs| am| gear| carb| -|-------------------|-----:|----:|------:|----:|-----:|------:|------:|----:|----:|-----:|-----:| -| Mazda RX4 | 21.0| 6| 160.0| 110| 3.90| 2.620| 16.46| 0| 1| 4| 4| -| Mazda RX4 Wag | 21.0| 6| 160.0| 110| 3.90| 2.875| 17.02| 0| 1| 4| 4| -| Datsun 710 | 22.8| 4| 108.0| 93| 3.85| 2.320| 18.61| 1| 1| 4| 1| -| Hornet 4 Drive | 21.4| 6| 258.0| 110| 3.08| 3.215| 19.44| 1| 0| 3| 1| -| Hornet Sportabout | 18.7| 8| 360.0| 175| 3.15| 3.440| 17.02| 0| 0| 3| 2| -| Valiant | 18.1| 6| 225.0| 105| 2.76| 3.460| 20.22| 1| 0| 3| 1| -| Duster 360 | 14.3| 8| 360.0| 245| 3.21| 3.570| 15.84| 0| 0| 3| 4| -| Merc 240D | 24.4| 4| 146.7| 62| 3.69| 3.190| 20.00| 1| 0| 4| 2| -| Merc 230 | 22.8| 4| 140.8| 95| 3.92| 3.150| 22.90| 1| 0| 4| 2| -| Merc 280 | 19.2| 6| 167.6| 123| 3.92| 3.440| 18.30| 1| 0| 4| 4| - -Also a quote using `>`: - -> "He who gives up \[code\] safety for \[code\] speed deserves neither." ([via](https://twitter.com/hadleywickham/status/504368538874703872)) - -[1] A footnote here. From 4386ab35b13c361bb471cbf039a980e8bddc1cc3 Mon Sep 17 00:00:00 2001 From: junyangq Date: Wed, 7 Sep 2016 03:37:21 +0800 Subject: [PATCH 05/13] Modify bash script to create vignettes. --- R/create-docs.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index d2ae160b50021..4707f2727433e 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -17,11 +17,13 @@ # limitations under the License. # -# Script to create API docs for SparkR -# This requires `devtools` and `knitr` to be installed on the machine. +# Script to create API docs and vignettes for SparkR +# This requires `devtools`, `knitr` and `rmarkdown` to be installed on the machine. # After running this script the html docs can be found in # $SPARK_HOME/R/pkg/html +# The vignettes can be found in +# $SPARK_HOME/R/pkg/vignettes/sparkr_vignettes.html set -o pipefail set -e @@ -43,4 +45,7 @@ Rscript -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); library(knit popd +# render creates SparkR vignettes +Rscript -e 'library(rmarkdown); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' + popd From efe56d21dde7003b891e128bfc1fea096a09d3ab Mon Sep 17 00:00:00 2001 From: junyangq Date: Wed, 7 Sep 2016 19:35:02 +0800 Subject: [PATCH 06/13] Resolve commented issues. --- R/pkg/vignettes/sparkr-vignettes.Rmd | 60 ++++++++++++++-------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 6543a2094c9c0..af73f3324a679 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -11,7 +11,7 @@ output: ## Overview -SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/). +SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. With Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/). ## Getting Started @@ -26,7 +26,7 @@ library(SparkR) We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession). -```{r, message=FALSE, warning=FALSE} +```{r, message=FALSE} sparkR.session() ``` @@ -39,26 +39,26 @@ cars <- cbind(model = rownames(mtcars), mtcars) carsDF <- createDataFrame(cars) ``` -We can view the first few rows of the `SparkDataFrame` by `showDF` or `head` function. +We can view the first few rows of the `SparkDataFrame` by `head` or `showDF` function. ```{r} -showDF(carsDF) +head(carsDF) ``` Common data processing operations such as `filter`, `select` are supported on the `SparkDataFrame`. ```{r} carsSubDF <- select(carsDF, "model", "mpg", "hp") carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200) -showDF(carsSubDF) +head(carsSubDF) ``` SparkR can use many common aggregation functions after grouping. ```{r} carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) -showDF(carsGPDF) +head(carsGPDF) ``` -The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. +The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. *Caution*: This can cause the driver to run out of memory, though, because `collect()` fetches the entire distributed `DataFrame` to a single machine; ```{r} carsGP <- collect(carsGPDF) class(carsGP) @@ -68,7 +68,7 @@ SparkR supports a number of commonly used machine learning algorithms. Under the SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. ```{r} -model <- spark.glm(carsDF, mpg ~ wt + cyl) +model <- spark.glm(data = carsDF, formula = mpg ~ wt + cyl) ``` ```{r} @@ -91,7 +91,7 @@ sparkR.session.stop() Different from many other R packages, to use SparkR, you need an additional installation of Apache Spark. The Spark installation will be used to run a backend process that will compile and execute SparkR programs. -If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. +If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process. You don't have to call it explicitly. We will check the installation when `sparkR.session` is called and `install.spark` function will be triggered automatically if no installation is found. ```{r, eval=FALSE} install.spark() @@ -105,12 +105,6 @@ sparkR.session(sparkHome = "/HOME/spark") ### Spark Session {#SetupSparkSession} -**For Windows users**: Due to different file prefixes across operating systems, to avoid the issue of potential wrong prefix, a current workaround is to specify `spark.sql.warehouse.dir` when starting the `SparkSession`. - -```{r, eval=FALSE} -spark_warehouse_path <- file.path(path.expand('~'), "spark-warehouse") -sparkR.session(spark.sql.warehouse.dir = spark_warehouse_path) -``` In addition to `sparkHome`, many other options can be specified in `sparkR.session`. For a complete list, see the [SparkR API doc](http://spark.apache.org/docs/latest/api/R/sparkR.session.html). @@ -123,9 +117,15 @@ spark.driver.extraClassPath | Runtime Environment | --driver-class-path spark.driver.extraJavaOptions | Runtime Environment | --driver-java-options spark.driver.extraLibraryPath | Runtime Environment | --driver-library-path +**For Windows users**: Due to different file prefixes across operating systems, to avoid the issue of potential wrong prefix, a current workaround is to specify `spark.sql.warehouse.dir` when starting the `SparkSession`. + +```{r, eval=FALSE} +spark_warehouse_path <- file.path(path.expand('~'), "spark-warehouse") +sparkR.session(spark.sql.warehouse.dir = spark_warehouse_path) +``` -### Cluster Mode +#### Cluster Mode SparkR can connect to remote Spark clusters. [Cluster Mode Overview](http://spark.apache.org/docs/latest/cluster-overview.html) is a good introduction to different Spark cluster modes. When connecting SparkR to a remote Spark cluster, make sure that the Spark version and Hadoop version on the machine match the corresponding versions on the cluster. Current SparkR package is compatible with @@ -145,7 +145,7 @@ For YARN cluster, SparkR supports the client mode with the master set as "yarn". ```{r, eval=FALSE} sparkR.session(master = "yarn") ``` - +Yarn cluster mode is not supported in the current version. ## Data Import @@ -261,7 +261,7 @@ You can also pass in column name as strings. head(select(carsDF, "mpg")) ``` -Filter the SparkDataFrame to only retain rows with wait times shorter than 50 mins. +Filter the SparkDataFrame to only retain rows with mpg less than 20 miles/gallon. ```{r} head(filter(carsDF, carsDF$mpg < 20)) ``` @@ -274,7 +274,7 @@ A common flow of grouping and aggregation is 2. Feed the `GroupedData` object to `agg` or `summarize` functions, with some provided aggregation functions to compute a number within each group. -A number of widely used functions are supported to aggregate data after grouping, including `avg`, `countDistinct`, `count`, `first`, `kurtosis`, `last`, `max`, `mean`, `min`, `sd`, `skewness`, `stddev_pop`, `stddev_samp`, `sumDistinct`, `sum`, `var_pop`, `var_samp`, `var`. +A number of widely used functions are supported to aggregate data after grouping, including `avg`, `countDistinct`, `count`, `first`, `kurtosis`, `last`, `max`, `mean`, `min`, `sd`, `skewness`, `stddev_pop`, `stddev_samp`, `sumDistinct`, `sum`, `var_pop`, `var_samp`, `var`. See the [API doc for `mean`](http://spark.apache.org/docs/latest/api/R/mean.html) and other `agg_funcs` linked there. For example we can compute a histogram of the number of cylinders in the `mtcars` dataset as shown below. @@ -310,7 +310,7 @@ We still use the `mtcars` dataset. The corresponding `SparkDataFrame` is `carsDF carsSubDF <- select(carsDF, "model", "mpg", "cyl") ws <- orderBy(windowPartitionBy("cyl"), "mpg") carsRank <- withColumn(carsSubDF, "rank", over(rank(), ws)) -showDF(carsRank) +head(carsRank, n = 20L) ``` We explain in detail the above steps. @@ -319,7 +319,7 @@ We explain in detail the above steps. More window specification methods include `rangeBetween`, which can define boundaries of the frame by value, and `rowsBetween`, which can define the boundaries by row indices. -* `withColumn` appends a Column called `"rank"` to the `SparkDataFrame`. `over` returns a windowing column. The first argument is usually a Column returned by window function(s) such as `rank()`, `lead(carsDF$wt)`. That calculates the corresponding values according to the partitioned-and-ordered table. +* `withColumn` appends a Column called `rank` to the `SparkDataFrame`. `over` returns a windowing column. The first argument is usually a Column returned by window function(s) such as `rank()`, `lead(carsDF$wt)`. That calculates the corresponding values according to the partitioned-and-ordered table. ### User-Defined Function @@ -516,7 +516,7 @@ titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5]) naiveBayesModel <- spark.naiveBayes(titanicDF, Survived ~ Class + Sex + Age) summary(naiveBayesModel) naiveBayesPrediction <- predict(naiveBayesModel, titanicDF) -showDF(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "prediction")) +head(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "prediction")) ``` #### k-Means Clustering @@ -527,7 +527,7 @@ showDF(select(naiveBayesPrediction, "Class", "Sex", "Age", "Survived", "predicti kmeansModel <- spark.kmeans(carsDF, ~ mpg + hp + wt, k = 3) summary(kmeansModel) kmeansPredictions <- predict(kmeansModel, carsDF) -showDF(select(kmeansPredictions, "model", "mpg", "hp", "wt", "prediction")) +head(select(kmeansPredictions, "model", "mpg", "hp", "wt", "prediction"), n = 20L) ``` #### AFT Survival Model @@ -558,7 +558,7 @@ df <- createDataFrame(data) gmmModel <- spark.gaussianMixture(df, ~ V1 + V2, k = 2) summary(gmmModel) gmmFitted <- predict(gmmModel, df) -showDF(select(gmmFitted, "V1", "V2", "prediction")) +head(select(gmmFitted, "V1", "V2", "prediction")) ``` @@ -700,7 +700,7 @@ Make predictions. ```{r} predicted <- predict(model, df) -showDF(predicted) +head(predicted) ``` #### Isotonic Regression Model @@ -767,7 +767,7 @@ summary(gaussianGLM2) # Check model prediction gaussianPredictions <- predict(gaussianGLM2, irisDF) -showDF(gaussianPredictions) +head(gaussianPredictions) unlink(modelPath) ``` @@ -795,7 +795,7 @@ This is often an intermediate object with group information and followed up by a ### Architecture -A complete description of architecture can be seen in paper [SparkR: Scaling R Programs with Spark](https://people.csail.mit.edu/matei/papers/2016/sigmod_sparkr.pdf), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. +A complete description of architecture can be seen in reference, in particular the paper *SparkR: Scaling R Programs with Spark*. Under the hood of SparkR is Spark SQL engine. This avoids the overheads of running interpreted R code, and the optimized SQL execution engine in Spark uses structural information about data and computation flow to perform a bunch of optimizations to speed up the computation. @@ -803,13 +803,13 @@ The main method calls of actual computation happen in the Spark JVM of the drive Two kinds of RPCs are supported in the SparkR JVM backend: method invocation and creating new objects. Method invocation can be done in two ways. -* `invokeJMethod` takes a reference to an existing Java object and a list of arguments to be passed on to the method. +* `sparkR.invokeJMethod` takes a reference to an existing Java object and a list of arguments to be passed on to the method. -* `invokeJStatic` takes a class name for static method and a list of arguments to be passed on to the method. +* `sparkR.invokeJStatic` takes a class name for static method and a list of arguments to be passed on to the method. The arguments are serialized using our custom wire format which is then deserialized on the JVM side. We then use Java reflection to invoke the appropriate method. -To create objects, a special method name `init` is used and then similarly the appropriate constructor is invoked with provided arguments. +To create objects, `sparkR.newJObject` is used and then similarly the appropriate constructor is invoked with provided arguments. Finally, we use a new R class `jobj` that refers to a Java object existing in the backend. These references are tracked on the Java side and are automatically garbage collected when they go out of scope on the R side. From 82b61676c6fad283dea55ea5f4a73888f4869c0a Mon Sep 17 00:00:00 2001 From: junyangq Date: Thu, 8 Sep 2016 15:02:48 +0800 Subject: [PATCH 07/13] Minor updates of markdown and create-docs script. --- R/create-docs.sh | 2 +- R/pkg/vignettes/sparkr-vignettes.Rmd | 10 ++++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index 4707f2727433e..e7f181c8541d1 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -46,6 +46,6 @@ Rscript -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); library(knit popd # render creates SparkR vignettes -Rscript -e 'library(rmarkdown); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' +Rscript -e 'library(rmarkdown); .libPaths(c("../../lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' popd diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index af73f3324a679..439adbbb77137 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -11,7 +11,7 @@ output: ## Overview -SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. With Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/). +SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. With Spark `r packageVersion("SparkR")`, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/). ## Getting Started @@ -58,7 +58,7 @@ carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) head(carsGPDF) ``` -The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. *Caution*: This can cause the driver to run out of memory, though, because `collect()` fetches the entire distributed `DataFrame` to a single machine; +The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. **Caution**: This can cause the driver to run out of memory, though, because `collect()` fetches the entire distributed `DataFrame` to a single machine; ```{r} carsGP <- collect(carsGPDF) class(carsGP) @@ -68,9 +68,11 @@ SparkR supports a number of commonly used machine learning algorithms. Under the SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example. ```{r} -model <- spark.glm(data = carsDF, formula = mpg ~ wt + cyl) +model <- spark.glm(carsDF, mpg ~ wt + cyl) ``` +The result matches that returned by R `glm` function applied to the corresponding `data.frame` `mtcars` of `carsDF`. In fact, for Generalized Linear Model, we specifically expose `glm` for `SparkDataFrame` as well so that the above is equivalent to `model <- glm(mpg ~ wt + cyl, data = carsDF)`. + ```{r} summary(model) ``` @@ -106,7 +108,7 @@ sparkR.session(sparkHome = "/HOME/spark") ### Spark Session {#SetupSparkSession} -In addition to `sparkHome`, many other options can be specified in `sparkR.session`. For a complete list, see the [SparkR API doc](http://spark.apache.org/docs/latest/api/R/sparkR.session.html). +In addition to `sparkHome`, many other options can be specified in `sparkR.session`. For a complete list, see [Starting up: SparkSession](http://spark.apache.org/docs/latest/sparkr.html#starting-up-sparksession) and [SparkR API doc](http://spark.apache.org/docs/latest/api/R/sparkR.session.html). In particular, the following Spark driver properties can be set in `sparkConfig`. From 1e34a2589602dd1cee895e737ebffa9dd0772e8b Mon Sep 17 00:00:00 2001 From: junyangq Date: Thu, 8 Sep 2016 16:32:26 +0800 Subject: [PATCH 08/13] Fix typo. --- R/create-docs.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index e7f181c8541d1..f51dc63da438d 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -46,6 +46,6 @@ Rscript -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); library(knit popd # render creates SparkR vignettes -Rscript -e 'library(rmarkdown); .libPaths(c("../../lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' +Rscript -e 'library(rmarkdown); .libPaths(c("lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' popd From adabb2d6b4c8359b02bbfeacd36b6793c354274b Mon Sep 17 00:00:00 2001 From: junyangq Date: Fri, 9 Sep 2016 11:39:06 +0800 Subject: [PATCH 09/13] Clean-up temp files. --- R/create-docs.sh | 2 ++ R/pkg/vignettes/sparkr-vignettes.Rmd | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index f51dc63da438d..a92b1f4fdf48f 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -48,4 +48,6 @@ popd # render creates SparkR vignettes Rscript -e 'library(rmarkdown); .libPaths(c("lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' +find pkg/vignettes -not -name '.' -not -name '*.Rmd' -not -name '*.md' -not -name '*.pdf' -not -name '*.html' -delete + popd diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 439adbbb77137..e6ff2ac0c7ec8 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -848,8 +848,6 @@ env | map * [SparkR: Scaling R Programs with Spark](https://people.csail.mit.edu/matei/papers/2016/sigmod_sparkr.pdf), Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, and Matei Zaharia. SIGMOD 2016. June 2016. - - ```{r, echo=FALSE} sparkR.session.stop() ``` From 1142facf3f02ededdc57a006ea065b6014510eae Mon Sep 17 00:00:00 2001 From: junyangq Date: Fri, 9 Sep 2016 12:28:18 +0800 Subject: [PATCH 10/13] Fix file cleaning. --- R/create-docs.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index a92b1f4fdf48f..8c4d3d946e4f3 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -48,6 +48,6 @@ popd # render creates SparkR vignettes Rscript -e 'library(rmarkdown); .libPaths(c("lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' -find pkg/vignettes -not -name '.' -not -name '*.Rmd' -not -name '*.md' -not -name '*.pdf' -not -name '*.html' -delete +find pkg/vignettes/. -not -name '.' -not -name '*.Rmd' -not -name '*.md' -not -name '*.pdf' -not -name '*.html' -delete popd From 7b552557a0fdbfbac6fa11ae578171ac42516cd6 Mon Sep 17 00:00:00 2001 From: junyangq Date: Fri, 9 Sep 2016 12:42:47 +0800 Subject: [PATCH 11/13] Restore Search Paths after creating vignette. --- R/create-docs.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/create-docs.sh b/R/create-docs.sh index 8c4d3d946e4f3..0dfba22463396 100755 --- a/R/create-docs.sh +++ b/R/create-docs.sh @@ -46,7 +46,7 @@ Rscript -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); library(knit popd # render creates SparkR vignettes -Rscript -e 'library(rmarkdown); .libPaths(c("lib", .libPaths())); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd")' +Rscript -e 'library(rmarkdown); paths <- .libPaths(); .libPaths(c("lib", paths)); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd"); .libPaths(paths)' find pkg/vignettes/. -not -name '.' -not -name '*.Rmd' -not -name '*.md' -not -name '*.pdf' -not -name '*.html' -delete From d2ae42a50b2d155e1378b658072d5e9e8323fbd3 Mon Sep 17 00:00:00 2001 From: junyangq Date: Wed, 14 Sep 2016 00:33:46 +0800 Subject: [PATCH 12/13] Update example of spark.lapply and slightly change the doc. --- R/pkg/vignettes/sparkr-vignettes.Rmd | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index e6ff2ac0c7ec8..a07bdc68eceb8 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -58,7 +58,7 @@ carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear)) head(carsGPDF) ``` -The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. **Caution**: This can cause the driver to run out of memory, though, because `collect()` fetches the entire distributed `DataFrame` to a single machine; +The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`. **Caution**: This can cause your interactive environment to run out of memory, though, because `collect()` fetches the entire distributed `DataFrame` to your client, which is acting as a Spark driver. ```{r} carsGP <- collect(carsGPDF) class(carsGP) @@ -385,22 +385,29 @@ head(result[order(result$max_mpg, decreasing = TRUE), ]) Similar to `lapply` in native R, `spark.lapply` runs a function over a list of elements and distributes the computations with Spark. `spark.lapply` works in a manner that is similar to `doParallel` or `lapply` to elements of a list. The results of all the computations should fit in a single machine. If that is not the case you can do something like `df <- createDataFrame(list)` and then use `dapply`. +We use `svm` in package `e1071` as an example. We use all default settings except for varying costs of constraints violation. `spark.lapply` can train those different models in parallel. + ```{r} -families <- c("gaussian", "poisson") -train <- function(family) { - model <- glm(mpg ~ hp, mtcars, family = family) +costs <- exp(seq(from = log(1), to = log(1000), length.out = 5)) +train <- function(cost) { + model <- e1071::svm(Species ~ ., data = iris, cost = cost) summary(model) } ``` Return a list of model's summaries. ```{r} -model.summaries <- spark.lapply(families, train) +model.summaries <- spark.lapply(costs, train) ``` -Print the summary of each model. ```{r} -print(model.summaries) +class(model.summaries) +``` + + +To avoid lengthy display, we only present the result of the second fitted model. You are free to inspect other models as well. +```{r} +print(model.summaries[[2]]) ``` From aa3f6a46fd27d7ad68973cb2426d06e20b6f0b32 Mon Sep 17 00:00:00 2001 From: junyangq Date: Wed, 14 Sep 2016 10:51:20 +0800 Subject: [PATCH 13/13] Add check of e1071 package in spark.lapply example. --- R/pkg/vignettes/sparkr-vignettes.Rmd | 1 + 1 file changed, 1 insertion(+) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index a07bdc68eceb8..aea52db8b8556 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -390,6 +390,7 @@ We use `svm` in package `e1071` as an example. We use all default settings excep ```{r} costs <- exp(seq(from = log(1), to = log(1000), length.out = 5)) train <- function(cost) { + stopifnot(requireNamespace("e1071", quietly = TRUE)) model <- e1071::svm(Species ~ ., data = iris, cost = cost) summary(model) }