[SPARK-12645] [SparkR] SparkR support hash function

[yanboliang]

Add hash function for SparkR DataFrame.

[SparkQA]

Test build #48757 has finished for PR 10597 at commit c41eb1f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

cc @sun-rui

[SparkQA]

Test build #48820 has finished for PR 10597 at commit 995fd06.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

LGTM

[felixcheung]

looks good. no conflict with base/stats

[shivaram]

LGTM. Thanks @yanboliang - Merging this to master and branch-1.6

[yhuai]

It breaks hadoop 1 test (https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-1.6-test-sbt-hadoop-1.0/30/console). Can you take a look?

[shivaram]

Hmm is hash not a supported function in 1.6 by any chance ? Are the other hadoop versions in branch-1.6 fine ?

[yhuai]

looks like 2.2 tests are broken as well. How about we revert it from 1.6 for now? If it is good, can you do the revert? Thanks!

[shivaram]

sure - sounds good to me. Can you open a JIRA and cc me and @yanboliang on it ?

[yhuai]

let's just reuse the original jira. I will reopen it.

[yhuai]

Actually, I am not sure if we should merge this to branch 1.6 since it is a new feature.

[shivaram]

The SparkR module is still considered alpha, so we do include new features in minor updates when it is appropriate. In this case it was a small new function, so it seemed fine to me.

[felixcheung]

it looks like hash is new in 2.0

  /**
   * Calculates the hash code of given columns, and returns the result as a int column.
   *
   * @group misc_funcs
   * @since 2.0
   */
  @scala.annotation.varargs
  def hash(cols: Column*): Column = withExpr {
    new Murmur3Hash(cols.map(_.expr))
  }

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L1823

so shouldn't be in 1.6.0

[yhuai]

@shivaram I will revert it from 1.6 branch.

[yhuai]

reverted from 1.6

[shivaram]

@yanboliang Could you test why this doesn't with branch 1.6 ?

[felixcheung]

@shivaram in my comment above, hash was added only in 2.0.0.

[yanboliang]

@shivaram Just like @felixcheung commented, the hash function was added only in 2.0.0. So revert it from branch 1.6 will fix the broken test.

[shivaram]

Ah I didn't notice @felixcheung earlier comment. Thanks for clarifying. I guess there is nothing to do here as the JIRA rightly says this feature is fixed in 2.0.0