[SPARK-12158] [SparkR] [SQL] Fix 'sample' functions that break R unit test cases #10160

gatorsmile · 2015-12-05T18:06:30Z

The existing sample functions miss the parameter seed, however, the corresponding function interface in generics has such a parameter. Thus, although the function caller can call the function with the 'seed', we are not using the value.

This could cause SparkR unit tests failed. For example, I hit it in another PR:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47213/consoleFull

SparkQA · 2015-12-05T18:34:24Z

Test build #47230 has finished for PR 10160 at commit ec77010.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2015-12-05T19:55:34Z

@davies Could you take a look at this PR? Thank you!

shivaram · 2015-12-05T20:15:46Z

cc @sun-rui

felixcheung · 2015-12-06T05:20:45Z

weird - when I added the seed param it actually was harder to fail. (See #9549)
anyway, thanks for fixing this.

felixcheung · 2015-12-06T05:23:45Z

R/pkg/R/DataFrame.R

+setMethod("sample",
+          # we can send seed as an argument through callJMethod
+          signature(x = "DataFrame", withReplacement = "logical",
+                    fraction = "numeric", seed = "numeric"),


you could in fact merge these overload/variant into one. Please see this for an example:

spark/R/pkg/R/DataFrame.R

Line 1177 in 5011f26

if (!missing(j)) {

if (!missing(seed)) { sdf <- callJMethod(x@sdf, "sample", withReplacement, fraction, as.integer(seed)) } else { sdf <- callJMethod(x@sdf, "sample", withReplacement, fraction) }

gatorsmile · 2015-12-06T08:23:55Z

@felixcheung @sun-rui Thank you! Based on your comments, I did the changes. Please review the changes. : )

gatorsmile · 2015-12-06T08:24:43Z

ok to test

felixcheung · 2015-12-06T09:00:02Z

looks good! could you think of the best way to add a test for not setting seed?
hmm.. perhaps the loop I use in #9549?

SparkQA · 2015-12-06T13:21:34Z

Test build #47236 has finished for PR 10160 at commit 2ab89e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2015-12-06T16:45:49Z

@felixcheung I am not sure if we need to add a test case for sample. Normally, using a specific seed is the common way to verify the result of sample. The existing test case may be enough?

  sampled <- sample(df, FALSE, 1.0)
  expect_equal(nrow(collect(sampled)), count(df))

If needed, maybe we can add something like the below:

repeat {  if (count(sample(df, FALSE, 0.1)) != count(sample(df, FALSE, 0.1))) { break } }

felixcheung · 2015-12-07T04:49:58Z

@gatorsmile Sure - I guess the main thing is to ensure the seed is getting set. How about:

count1 <-  count(sample(df, FALSE, 0.1, 0))
count2 <-  count(sample(df, FALSE, 0.1, 0))
expect_equal(count1, count2)

?

shivaram · 2015-12-07T05:06:26Z

Yeah thats a good idea @felixcheung

gatorsmile · 2015-12-07T05:25:46Z

@felixcheung @shivaram Sure, just added that test case. Please review it. Thank you! : )

shivaram · 2015-12-07T05:30:36Z

LGTM. I'll wait for @felixcheung and Jenkins before merging

sun-rui · 2015-12-07T05:33:34Z

R/pkg/R/DataFrame.R

-          function(x, withReplacement, fraction) {
-            sample(x, withReplacement, fraction)
+          function(x, withReplacement, fraction, seed) {
+            if (!missing(seed)) {


why not call directly: sample(x, withReplacement, fraction, seed)?

gatorsmile · 2015-12-07T05:42:43Z

: ) @sun-rui Done. Thank you!

SparkQA · 2015-12-07T05:56:24Z

Test build #47253 has finished for PR 10160 at commit 34d0118.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sun-rui · 2015-12-07T05:59:34Z

R/pkg/R/DataFrame.R

-          function(x, withReplacement, fraction) {
-            sample(x, withReplacement, fraction)
+          function(x, withReplacement, fraction, seed) {
+              sample(x, withReplacement, fraction, as.integer(seed))


2 space ident. directly pass seed is ok instead of as.integer(seed)?

Yeah, done! This is my first time to read and write R. : ) Thank you!

SparkQA · 2015-12-07T06:11:43Z

Test build #47254 has finished for PR 10160 at commit 4337c35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-07T06:48:27Z

Test build #47255 has finished for PR 10160 at commit a78109e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2015-12-07T06:51:13Z

@shivaram @felixcheung @sun-rui Thank you everyone! Hopefully, my code changes resolve all your concerns. I learned a lot from you! : )

sun-rui · 2015-12-07T06:59:14Z

LGTM

shivaram · 2015-12-07T18:40:46Z

@gatorsmile This will need to be rebased to master as we moved test file locations in #10030 -- Let me know once thats done and I'll merge

felixcheung · 2015-12-07T19:14:23Z

looks good.

gatorsmile · 2015-12-07T19:50:49Z

retest this please

SparkQA · 2015-12-07T20:11:37Z

Test build #47279 has finished for PR 10160 at commit ad3ea31.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-07T20:21:35Z

Test build #47280 has finished for PR 10160 at commit ad3ea31.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shivaram · 2015-12-07T21:12:04Z

R/pkg/R/DataFrame.R

          signature(x = "DataFrame", withReplacement = "logical",
                    fraction = "numeric"),
-          function(x, withReplacement, fraction) {
+          function(x, withReplacement, fraction, seed) {


@felixcheung Shouldn't we document this param in the roxygen doc above ? Otherwise how would anybody know we support a seed ?

yes we should add a @param seed above, thanks for catching it

felixcheung · 2015-12-07T21:50:35Z

looks good thanks @gatorsmile

gatorsmile · 2015-12-07T21:50:53Z

Thank you, @shivaram @felixcheung !

SparkQA · 2015-12-07T22:20:17Z

Test build #47287 has finished for PR 10160 at commit 493b368.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2015-12-11T23:48:08Z

@shivaram Will it be merged before the release of 1.6? Thanks!

shivaram · 2015-12-12T04:54:30Z

Sorry @gatorsmile -- I missed this PR for a couple of days. LGTM. Merging this to master and branch-1.6. Regarding whether this will make 1.6 release, it depends on which RC becomes a release -- I think RC2 was cut earlier today, but I'm not sure.

…est cases The existing sample functions miss the parameter `seed`, however, the corresponding function interface in `generics` has such a parameter. Thus, although the function caller can call the function with the 'seed', we are not using the value. This could cause SparkR unit tests failed. For example, I hit it in another PR: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47213/consoleFull Author: gatorsmile <gatorsmile@gmail.com> Closes #10160 from gatorsmile/sampleR. (cherry picked from commit 1e3526c) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

gatorsmile · 2015-12-12T07:40:17Z

Thank you, everyone! : )

add sample functions with seeds

ec77010

felixcheung reviewed Dec 6, 2015
View reviewed changes

combine two variants

2ab89e8

gatorsmile changed the title ~~[SPARK-12158] [R] [SQL] Fix 'sample' functions that break R unit test cases~~ [SPARK-12158] [SparkR] [SQL] Fix 'sample' functions that break R unit test cases Dec 6, 2015

added a test case

34d0118

sun-rui reviewed Dec 7, 2015
View reviewed changes

simplified the impl of sample_frac

4337c35

sun-rui reviewed Dec 7, 2015
View reviewed changes

simplified the impl of sample_frac.

a78109e

Merge remote-tracking branch 'upstream/master' into sampleR

ad3ea31

shivaram reviewed Dec 7, 2015
View reviewed changes

added a new parm into the comment

493b368

asfgit closed this in 1e3526c Dec 12, 2015

gatorsmile deleted the sampleR branch December 30, 2015 17:14

[SPARK-12158] [SparkR] [SQL] Fix 'sample' functions that break R unit test cases #10160

[SPARK-12158] [SparkR] [SQL] Fix 'sample' functions that break R unit test cases #10160

Conversation

gatorsmile commented Dec 5, 2015

SparkQA commented Dec 5, 2015

gatorsmile commented Dec 5, 2015

shivaram commented Dec 5, 2015

felixcheung commented Dec 6, 2015

felixcheung Dec 6, 2015

Choose a reason for hiding this comment

sun-rui Dec 6, 2015

Choose a reason for hiding this comment

gatorsmile commented Dec 6, 2015

gatorsmile commented Dec 6, 2015

felixcheung commented Dec 6, 2015

SparkQA commented Dec 6, 2015

gatorsmile commented Dec 6, 2015

felixcheung commented Dec 7, 2015

shivaram commented Dec 7, 2015

gatorsmile commented Dec 7, 2015

shivaram commented Dec 7, 2015

sun-rui Dec 7, 2015

Choose a reason for hiding this comment

gatorsmile commented Dec 7, 2015

SparkQA commented Dec 7, 2015

sun-rui Dec 7, 2015

Choose a reason for hiding this comment

gatorsmile Dec 7, 2015

Choose a reason for hiding this comment

SparkQA commented Dec 7, 2015

SparkQA commented Dec 7, 2015

gatorsmile commented Dec 7, 2015

sun-rui commented Dec 7, 2015

shivaram commented Dec 7, 2015

felixcheung commented Dec 7, 2015

gatorsmile commented Dec 7, 2015

SparkQA commented Dec 7, 2015

SparkQA commented Dec 7, 2015

shivaram Dec 7, 2015

Choose a reason for hiding this comment

felixcheung Dec 7, 2015

Choose a reason for hiding this comment

felixcheung commented Dec 7, 2015

gatorsmile commented Dec 7, 2015

SparkQA commented Dec 7, 2015

gatorsmile commented Dec 11, 2015

shivaram commented Dec 12, 2015

gatorsmile commented Dec 12, 2015