[SPARK-20889][SparkR] Grouped documentation for NONAGGREGATE column methods #18422

actuaryzhang · 2017-06-26T18:19:35Z

What changes were proposed in this pull request?

Grouped documentation for nonaggregate column methods.

SparkQA · 2017-06-26T18:52:24Z

Test build #78643 has finished for PR 18422 at commit d2a7ca8.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-26T19:51:18Z

jenkins, retest this please

SparkQA · 2017-06-26T20:37:21Z

Test build #78646 has finished for PR 18422 at commit d2a7ca8.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-26T22:22:10Z

Test build #78655 has finished for PR 18422 at commit 3057856.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-26T23:12:22Z

jenkins, retest this please

SparkQA · 2017-06-26T23:59:11Z

Test build #78662 has finished for PR 18422 at commit 3057856.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-27T00:28:21Z

Test build #78665 has finished for PR 18422 at commit 4e77d7b.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-27T00:42:15Z

jenkins, retest this please

SparkQA · 2017-06-27T01:25:03Z

Test build #78668 has finished for PR 18422 at commit 4e77d7b.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-27T02:34:48Z

jenkins, retest this please

actuaryzhang · 2017-06-27T02:55:21Z

jenkins, retest this please

felixcheung · 2017-06-27T16:16:48Z

jenkins, retest this please

SparkQA · 2017-06-27T16:54:02Z

Test build #78713 has finished for PR 18422 at commit 4e77d7b.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-28T18:27:57Z

Test build #78819 has finished for PR 18422 at commit b72bf9c.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-28T20:10:42Z

Test build #78817 has finished for PR 18422 at commit 83ecb98.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-28T21:42:38Z

jenkins, retest this please

SparkQA · 2017-06-28T22:08:20Z

Test build #78824 has finished for PR 18422 at commit b72bf9c.

This patch fails due to an unknown error code, -10.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-06-29T03:24:16Z

R/pkg/R/functions.R

+NULL
+
+#' @details
+#' \code{lit}: A new \linkS4class{Column} is created to represent the literal value.


this format is actually kinda weird. let's fix it? I don't think we need to link to Column
(yes, I think I added this...)

felixcheung · 2017-06-29T03:26:17Z

R/pkg/R/functions.R

-#' head(select(df, input_file_name()))
-#' }
+#' \dontrun{
+#' tmp <- read.text("README.md")


why rename to tmp though?

To avoid overwriting the dataframe example df used throughout the doc.

felixcheung · 2017-06-29T03:32:02Z

R/pkg/R/functions.R

-#'
-#' @param x Column to compute on.
+#' @details
+#' \code{is.nan}: Alias for \link{isnan}.


roxygen does this by text order, I think - doesn't it make this go first, before isnan? perhaps we swap the order of code?

OK, swapped the order.

felixcheung · 2017-06-29T03:33:29Z

R/pkg/R/functions.R

+#' tmp <- mutate(df, v1 = lit(df$mpg), v2 = lit("x"), v3 = lit("2015-01-01"),
+#'                   v4 = negate(df$mpg), v5 = expr('length(model)'),
+#'                   v6 = greatest(df$vs, df$am), v7 = least(df$vs, df$am),
+#'                   v8 = column("mpg"))


is there example for

nanvl(df$c, x) coalesce(df$c, df$d, df$e)

that I've missed?

felixcheung · 2017-06-29T03:35:01Z

R/pkg/R/functions.R

+#' @param x Column to compute on. In \code{lit}, it is a literal value or a Column.
+#'          In \code{monotonically_increasing_id}, it should be empty.
+#' @param y Column to compute on.
+#' @param ... additional argument(s). In \code{expr}, it contains an expression character


In \code{expr}, it contains an expression character - this isn't quite right actually - it's in x in expr, not as ... parameter

and so in all other cases in this group, ... is expected for other columns. perhaps we can say additional Columns

Right, thanks for catching this.

felixcheung · 2017-06-29T03:36:46Z

R/pkg/R/functions.R

 #'
-#' @param x a literal value or a Column.
+#' @param x Column to compute on. In \code{lit}, it is a literal value or a Column.
+#'          In \code{monotonically_increasing_id}, it should be empty.


and same for input_file_name - btw, should be empty might be a bit confusing? how about In ......., Should be used with no argument.? or omitted?

Yes, I was just copying from the old doc.
I now remove this and add The method should be used with no argument. to the two individual methods.

SparkQA · 2017-06-29T04:05:33Z

Test build #78855 has finished for PR 18422 at commit aff832e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

actuaryzhang · 2017-06-29T06:14:09Z

R/pkg/R/functions.R

+#' head(tmp)
+#' tmp <- mutate(tmp, ind_na1 = is.nan(tmp$mpg_na), ind_na2 = isnan(tmp$mpg_na))
+#' head(select(tmp, coalesce(tmp$mpg_na, tmp$mpg)))
+#' head(select(tmp, nanvl(tmp$mpg_na, tmp$hp)))}


@felixcheung Examples for coalesce and nanvl are here.

SparkQA · 2017-06-29T07:02:22Z

Test build #78871 has finished for PR 18422 at commit 1d0989a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung

LGTM.

felixcheung · 2017-06-29T08:23:39Z

merged to master

…ethods ## What changes were proposed in this pull request? Grouped documentation for nonaggregate column methods. Author: actuaryzhang <actuaryzhang10@gmail.com> Author: Wayne Zhang <actuaryzhang10@gmail.com> Closes apache#18422 from actuaryzhang/sparkRDocNonAgg.

update doc for column nonaggregate functions

d2a7ca8

fix test error

4e77d7b

actuaryzhang force-pushed the sparkRDocNonAgg branch from 3057856 to 4e77d7b Compare June 27, 2017 00:06

actuaryzhang added 3 commits June 28, 2017 10:35

revert from_json and to_json

e32aace

add doc for monotonically_increasing_id

83ecb98

Merge branch 'master' into sparkRDocNonAgg

b72bf9c

Merge branch 'master' into sparkRDocNonAgg

aff832e

felixcheung reviewed Jun 29, 2017

View reviewed changes

actuaryzhang commented Jun 29, 2017

View reviewed changes

address comments

1d0989a

felixcheung approved these changes Jun 29, 2017

View reviewed changes

asfgit closed this in a2d5623 Jun 29, 2017

actuaryzhang deleted the sparkRDocNonAgg branch June 30, 2017 16:32

[SPARK-20889][SparkR] Grouped documentation for NONAGGREGATE column methods #18422

[SPARK-20889][SparkR] Grouped documentation for NONAGGREGATE column methods #18422

Conversation

actuaryzhang commented Jun 26, 2017

What changes were proposed in this pull request?

SparkQA commented Jun 26, 2017

actuaryzhang commented Jun 26, 2017

SparkQA commented Jun 26, 2017

SparkQA commented Jun 26, 2017

actuaryzhang commented Jun 26, 2017

SparkQA commented Jun 26, 2017

SparkQA commented Jun 27, 2017

actuaryzhang commented Jun 27, 2017

SparkQA commented Jun 27, 2017

actuaryzhang commented Jun 27, 2017

actuaryzhang commented Jun 27, 2017

felixcheung commented Jun 27, 2017

SparkQA commented Jun 27, 2017

SparkQA commented Jun 28, 2017

SparkQA commented Jun 28, 2017

actuaryzhang commented Jun 28, 2017

SparkQA commented Jun 28, 2017

felixcheung Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixcheung Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixcheung Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jun 29, 2017

Choose a reason for hiding this comment

SparkQA commented Jun 29, 2017

felixcheung left a comment

Choose a reason for hiding this comment

felixcheung commented Jun 29, 2017

felixcheung Jun 29, 2017 •

edited

felixcheung Jun 29, 2017 •

edited

felixcheung Jun 29, 2017 •

edited