[SPARK-12393] [SparkR] Add read.text and write.text for SparkR #10348
Conversation
yanboliang
changed the title
|
Test build #47904 has finished for PR 10348 at commit
|
| #' @rdname write.text | ||
| #' @export | ||
| setGeneric("write.text", function(x, path) { standardGeneric("write.text") }) | ||
|
|
There was a problem hiding this comment.
shouldn't read.text be added too?
There was a problem hiding this comment.
I found all functions in SQLContext.R did not use setGeneric, is this on purpose or a bug?
If it's a bug, I can fix it.
There was a problem hiding this comment.
This is historical legacy:) (you can find same cases in context.R.) I guess that because we did not implement SparkContext and SQLContext as S4 classes. If there is no strong reason, we can keep it as is.
cc @shivaram.
|
Test build #48000 has finished for PR 10348 at commit
|
| #' sc <- sparkR.init() | ||
| #' sqlContext <- sparkRSQL.init(sc) | ||
| #' path <- "path/to/file.json" | ||
| #' df <- read.json(sqlContext, path) |
There was a problem hiding this comment.
is this a bit odd as an example? it's loading a DF as json and writing it out as text (which requires one string column?
There was a problem hiding this comment.
I will update it with read.text.
|
Test build #48185 has finished for PR 10348 at commit
|
|
LGTM |
| @@ -661,6 +661,34 @@ setMethod("saveAsParquetFile", | |||
| write.parquet(x, path) | |||
| }) | |||
|
|
|||
| #' write.text | |||
| #' | |||
| #' Saves the content of the DataFrame in a text file at the specified path. | |||
There was a problem hiding this comment.
might want to add the DataFrame must have a column with the name "value" I recall there is a similar doc clarification in Scala recently.
|
Test build #48813 has finished for PR 10348 at commit
|
|
looks good |
|
LGTM. Thanks @yanboliang for the patch an @sun-rui @felixcheung for reviewing. Merging this to master and |
Add ```read.text``` and ```write.text``` for SparkR. cc sun-rui felixcheung shivaram Author: Yanbo Liang <ybliang8@gmail.com> Closes #10348 from yanboliang/spark-12393. (cherry picked from commit d1fea41) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Add
read.textandwrite.textfor SparkR.cc @sun-rui @felixcheung @shivaram