Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24054][R] Add array_position function / element_at functions #21130

Closed

Conversation

HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Apr 23, 2018

What changes were proposed in this pull request?

This PR proposes to add array_position and element_at in R side too.

array_position:

df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))
mutated <- mutate(df, v1 = create_array(df$gear, df$am, df$carb))
head(select(mutated, array_position(mutated$v1, 1)))
  array_position(v1, 1.0)
1                       2
2                       2
3                       2
4                       3
5                       0
6                       3

element_at:

df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))
mutated <- mutate(df, v1 = create_array(df$mpg, df$cyl, df$hp))
head(select(mutated, element_at(mutated$v1, 1)))
  element_at(v1, 1.0)
1                21.0
2                21.0
3                22.8
4                21.4
5                18.7
6                18.1
df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))
mutated <- mutate(df, v1 = create_map(df$model, df$cyl))
head(select(mutated, element_at(mutated$v1, "Valiant")))
  element_at(v3, Valiant)
1                      NA
2                      NA
3                      NA
4                      NA
5                      NA
6                       6

How was this patch tested?

Unit tests were added in R/pkg/tests/fulltests/test_sparkSQL.R and manually tested. Documentation was manually built and verified.

@HyukjinKwon
Copy link
Member Author

cc @felixcheung, can you take a look please when you are available?

@HyukjinKwon HyukjinKwon force-pushed the sparkr_array_position_element_at branch from 443034a to 7f75f5e Compare April 23, 2018 15:19
@SparkQA
Copy link

SparkQA commented Apr 23, 2018

Test build #89729 has finished for PR 21130 at commit 443034a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 23, 2018

Test build #89731 has finished for PR 21130 at commit 7f75f5e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

#' @param value A value to compute on.
#' \itemize{
#' \item \code{array_contains}: a value to be checked if contained in the column.
#' \item \code{array_position}: a value to to locate in the given array.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: to to

@viirya
Copy link
Member

viirya commented Apr 24, 2018

LGTM except for a tiny typo.

@SparkQA
Copy link

SparkQA commented Apr 24, 2018

Test build #89765 has finished for PR 21130 at commit d1d1bb4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

Merged to master.

@asfgit asfgit closed this in 87e8a57 Apr 24, 2018
@HyukjinKwon
Copy link
Member Author

Thank you for reviewing this @felixcheung and @viirya.

@HyukjinKwon HyukjinKwon deleted the sparkr_array_position_element_at branch October 16, 2018 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants