[SPARK-8807][SparkR] Add between operator in SparkR

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-8807

Add between operator in SparkR.

[SparkQA]

Test build #37091 has finished for PR 7356 at commit c6a25c5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

@sun-rui could you also take a look at this ?

[shivaram]

Would we also need to check the types of bound here ? Or to put it another way, does this work for all types or just numbers ?

[viirya]

Because in Column.scala, the types of lowerBound and upperBound in between function are Any, so I have not checked the types of bound here too.

[viirya]

@shivaram Any more comments?

[SparkQA]

Test build #37463 has finished for PR 7356 at commit 7f51b44.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

Thanks @viirya -- LGTM.

[sun-rui]

@viirya,

1. Why pass the lower bound and upper bound in a vector instead of pass separately? They are passed separately in Scala API. and A vector can not hold complex types (at least a list is expected).
2. Instead of writing R wrapper functions individually for each method of Scala Column, currently Column.R dynamically creates wrapper function using template. Is it possible to use the same mechanism for supporting between? something like:

column_functions3 <- c("between")

createColumnFunction3 <- function(name) {
setMethod(name,
signature(x = "Column"),
function(x, arg1, arg2) {
... # some check of arg1/arg2
jc <- callJMethod(x@jc, name, arg1, arg2)
column(jc)
})
}

[viirya]

@sun-rui I use the API described in the JIRA ticket. Actually I used two parameters for the lower bound and upper bounds at the first. I am not sure if the between operation supports complex types?

I think it is possibly to have createColumnFunction3 to support it. If there are enough methods with 3 parameters, I think we can create it.

[shivaram]

Hmm I think the scala API having two arguments makes an interesting case -- I did not notice that. What does the Python API do ?

One thing about the createColumnFunction things is that they are hard to document with roxygen. We should create a separate JIRA to discuss this.