[SPARK-6806] [SPARKR] [DOCS] Add a new SparkR programming guide #6490
Conversation
Also update write.df, read.df to handle defaults better
docs/sparkr.md
Outdated
There was a problem hiding this comment.
"supports operations similar to R data frames, [dplyr]"
add "etc"/"e.g."?
|
Test build #33721 has finished for PR 6490 at commit
|
There was a problem hiding this comment.
Can we hide sparkR.init() and call it in sparkRSQL.init() internally?
There was a problem hiding this comment.
Yeah I was thinking about the fact that we have these two init calls being wasteful. But longer term when we say want to introduce ML stuff which requires the SparkContext it might be good to familiarize users with the idea of having a SparkContext around ?
We can definitely do an implicit sparkR.init though if we find that no spark context exists (something like the logic we use in
Line 105 in e7b6177
There was a problem hiding this comment.
Yes, it seems reasonable since we only support one SparkContext at a time.
|
This looks good to me. Can we have a section about Hive? hiveContext and load a table from hive. |
|
Thanks @davies for taking a look. Added the section on HiveContext now |
|
BTW @davies I think we can add the automatic SparkContext creation in a separate PR ? I'd like to keep the amount of code change in this PR small |
|
Test build #33759 has finished for PR 6490 at commit
|
|
@shivaram LGTM, merging this into master and 1.4. |
This PR adds a new SparkR programming guide at the top-level. This will be useful for R users as our APIs don't directly match the Scala/Python APIs and as we need to explain SparkR without using RDDs as examples etc. cc rxin davies pwendell cc cafreeman -- Would be great if you could also take a look at this ! Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu> Closes #6490 from shivaram/sparkr-guide and squashes the following commits: d5ff360 [Shivaram Venkataraman] Add a section on HiveContext, HQL queries 408dce5 [Shivaram Venkataraman] Fix link dbb86e3 [Shivaram Venkataraman] Fix minor typo 9aff5e0 [Shivaram Venkataraman] Address comments, use dplyr-like syntax in example d09703c [Shivaram Venkataraman] Fix default argument in read.df ea816a1 [Shivaram Venkataraman] Add a new SparkR programming guide Also update write.df, read.df to handle defaults better (cherry picked from commit 5f48e5c) Signed-off-by: Davies Liu <davies@databricks.com>
|
Test build #33768 timed out for PR 6490 at commit |
This PR adds a new SparkR programming guide at the top-level. This will be useful for R users as our APIs don't directly match the Scala/Python APIs and as we need to explain SparkR without using RDDs as examples etc. cc rxin davies pwendell cc cafreeman -- Would be great if you could also take a look at this ! Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu> Closes apache#6490 from shivaram/sparkr-guide and squashes the following commits: d5ff360 [Shivaram Venkataraman] Add a section on HiveContext, HQL queries 408dce5 [Shivaram Venkataraman] Fix link dbb86e3 [Shivaram Venkataraman] Fix minor typo 9aff5e0 [Shivaram Venkataraman] Address comments, use dplyr-like syntax in example d09703c [Shivaram Venkataraman] Fix default argument in read.df ea816a1 [Shivaram Venkataraman] Add a new SparkR programming guide Also update write.df, read.df to handle defaults better
This PR adds a new SparkR programming guide at the top-level. This will be useful for R users as our APIs don't directly match the Scala/Python APIs and as we need to explain SparkR without using RDDs as examples etc. cc rxin davies pwendell cc cafreeman -- Would be great if you could also take a look at this ! Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu> Closes apache#6490 from shivaram/sparkr-guide and squashes the following commits: d5ff360 [Shivaram Venkataraman] Add a section on HiveContext, HQL queries 408dce5 [Shivaram Venkataraman] Fix link dbb86e3 [Shivaram Venkataraman] Fix minor typo 9aff5e0 [Shivaram Venkataraman] Address comments, use dplyr-like syntax in example d09703c [Shivaram Venkataraman] Fix default argument in read.df ea816a1 [Shivaram Venkataraman] Add a new SparkR programming guide Also update write.df, read.df to handle defaults better
4 participants
This PR adds a new SparkR programming guide at the top-level. This will be useful for R users as our APIs don't directly match the Scala/Python APIs and as we need to explain SparkR without using RDDs as examples etc.
cc @rxin @davies @pwendell
cc @cafreeman -- Would be great if you could also take a look at this !