Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
21080af
added "empty seq" in explain
gatorsmile Dec 28, 2015
25b6908
address the comments.
gatorsmile Dec 29, 2015
c60d7f0
Merge remote-tracking branch 'upstream/master' into explainImprove
gatorsmile Dec 29, 2015
1a91be8
[SPARK-12547][SQL] Tighten scala style checker enforcement for UDF re…
rxin Dec 29, 2015
73862a1
[SPARK-11394][SQL] Throw IllegalArgumentException for unsupported typ…
maropu Dec 29, 2015
d80cc90
[SPARK-12526][SPARKR] ifelse`, `when`, `otherwise` unable to take Col…
saurfang Dec 29, 2015
8e629b1
[SPARK-12530][BUILD] Fix build break at Spark-Master-Maven-Snapshots …
kiszk Dec 29, 2015
f6ecf14
[SPARK-11199][SPARKR] Improve R context management story and add getO…
falaki Dec 29, 2015
be86268
[SPARK-12349][SPARK-12349][ML] Fix typo in Spark version regex introd…
srowen Dec 30, 2015
270a659
[SPARK-12549][SQL] Take Option[Seq[DataType]] in UDF input type speci…
rxin Dec 30, 2015
b600bcc
[SPARK-12362][SQL][WIP] Inline Hive Parser
nongli Dec 30, 2015
7ab0e22
[SPARK-12490][CORE] Limit the css style scope to fix the Streaming UI
zsxwing Dec 30, 2015
4f75f78
[SPARK-12564][SQL] Improve missing column AnalysisException
gatorsmile Dec 30, 2015
27af615
Revert "[SPARK-12362][SQL][WIP] Inline Hive Parser"
rxin Dec 30, 2015
932cf44
[SPARK-12263][DOCS] IllegalStateException: Memory can't be 0 for SPAR…
Dec 30, 2015
aa48164
[SPARK-12495][SQL] use true as default value for propagateNull in New…
cloud-fan Dec 30, 2015
d1ca634
[SPARK-12300] [SQL] [PYSPARK] fix schema inferance on local collections
holdenk Dec 30, 2015
27a42c7
[SPARK-10359] Enumerate dependencies in a file and diff against it fo…
JoshRosen Dec 30, 2015
5c2682b
[SPARK-12409][SPARK-12387][SPARK-12391][SQL] Support AND/OR/IN/LIKE p…
maropu Dec 30, 2015
b244297
[SPARK-12399] Display correct error message when accessing REST API w…
carsonwang Dec 30, 2015
f76ee10
[SPARK-8641][SPARK-12455][SQL] Native Spark Window functions - Follow…
hvanhovell Dec 31, 2015
ee8f8d3
[SPARK-12588] Remove HttpBroadcast in Spark 2.0.
rxin Dec 31, 2015
fd33333
[SPARK-3873][YARN] Fix import ordering.
Dec 31, 2015
9140d90
[SPARK-3873][GRAPHX] Import order fixes.
Dec 31, 2015
be33a0c
[SPARK-12561] Remove JobLogger in Spark 2.0.
rxin Dec 31, 2015
7b4452b
House cleaning: close open pull requests created before June 1st, 2015
rxin Dec 31, 2015
c642c3a
Closes #10386 since it was superseded by #10468.
rxin Dec 31, 2015
93b52ab
House cleaning: close old pull requests.
rxin Dec 31, 2015
e6c7787
[SPARK-12585] [SQL] move numFields to constructor of UnsafeRow
Dec 31, 2015
4f5a24d
[SPARK-7995][SPARK-6280][CORE] Remove AkkaRpcEnv and remove systemNam…
zsxwing Dec 31, 2015
5cdecb1
[SPARK-12039][SQL] Re-enable HiveSparkSubmitSuite's SPARK-9757 Persis…
yhuai Dec 31, 2015
efb10cc
[SPARK-3873][STREAMING] Import order fixes for streaming.
Dec 31, 2015
5adec63
[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.…
JoshRosen Jan 1, 2016
c9dbfcc
[SPARK-11743][SQL] Move the test for arrayOfUDT
viirya Jan 1, 2016
a59a357
[SPARK-3873][MLLIB] Import order fixes.
Jan 1, 2016
ad5b7cf
[SPARK-12409][SPARK-12387][SPARK-12391][SQL] Refactor filter pushdown…
viirya Jan 1, 2016
01a2986
[SPARK-12592][SQL][TEST] Don't mute Spark loggers in TestHive.reset()
liancheng Jan 1, 2016
6c20b3c
Disable test-dependencies.sh.
rxin Jan 1, 2016
0da7bd5
[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][SQL] always outp…
Jan 1, 2016
44ee920
Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][SQL] alw…
rxin Jan 2, 2016
970635a
[SPARK-12362][SQL][WIP] Inline Hive Parser
hvanhovell Jan 2, 2016
94f7a12
[SPARK-10180][SQL] JDBC datasource are not processing EqualNullSafe f…
HyukjinKwon Jan 2, 2016
15bd736
[SPARK-12481][CORE][STREAMING][SQL] Remove usage of Hadoop deprecated…
srowen Jan 2, 2016
513e3b0
[SPARK-12599][MLLIB][SQL] Remove the use of callUDF in MLlib
rxin Jan 3, 2016
6c5bbd6
Revert "Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][…
rxin Jan 3, 2016
c3d5056
[SPARK-12327][SPARKR] fix code for lintr warning for commented code
felixcheung Jan 3, 2016
c82924d
[SPARK-12533][SQL] hiveContext.table() throws the wrong exception
thomastechs Jan 3, 2016
7b92922
Update MimaExcludes now Spark 1.6 is in Maven.
rxin Jan 4, 2016
b8410ff
[SPARK-12537][SQL] Add option to accept quoting of all character back…
Cazen Jan 4, 2016
13dab9c
[SPARK-12611][SQL][PYSPARK][TESTS] Fix test_infer_schema_to_local
holdenk Jan 4, 2016
84f8492
[SPARK-12562][SQL] DataFrame.write.format(text) requires the column n…
xguo27 Jan 4, 2016
0d165ec
[SPARK-12612][PROJECT-INFRA] Add missing Hadoop profiles to dev/run-t…
JoshRosen Jan 4, 2016
9fd7a2f
[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-depen…
JoshRosen Jan 4, 2016
962aac4
[SPARK-12513][STREAMING] SocketReceiver hang in Netcat example
powerLambda Jan 4, 2016
8f65939
[SPARK-12486] Worker should kill the executors more forcefully if pos…
nongli Jan 4, 2016
6c83d93
[SPARK-12579][SQL] Force user-specified JDBC driver to take precedence
JoshRosen Jan 4, 2016
b504b6a
[SPARK-12470] [SQL] Fix size reduction calculation
robbinspg Jan 4, 2016
43706bf
[SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob d…
zsxwing Jan 4, 2016
573ac55
[SPARK-12512][SQL] support column name with dot in withColumn()
xguo27 Jan 4, 2016
40d0396
[DOC] Adjust coverage for partitionBy()
tedyu Jan 4, 2016
0171b71
[SPARK-12421][SQL] Prevent Internal/External row from exposing state.
hvanhovell Jan 4, 2016
ba5f818
[SPARK-11259][ML] Params.validateParams() should be called automatically
yanboliang Jan 4, 2016
93ef9b6
[SPARK-9622][ML] DecisionTreeRegressor: provide variance of prediction
yanboliang Jan 4, 2016
d084a2d
[SPARK-12541] [SQL] support cube/rollup as function
Jan 4, 2016
34de24a
[SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set t…
nongli Jan 4, 2016
fdfac22
[SPARK-12509][SQL] Fixed error messages for DataFrame correlation and…
NarineK Jan 5, 2016
77ab49b
[SPARK-12600][SQL] Remove deprecated methods in Spark SQL
rxin Jan 5, 2016
b1a7712
[SPARK-12480][SQL] add Hash expression that can calculate hash value …
cloud-fan Jan 5, 2016
8896ec9
[SPARKR][DOC] minor doc update for version in migration guide
felixcheung Jan 5, 2016
b634901
[SPARK-12600][SQL] follow up: add range check for DecimalType
rxin Jan 5, 2016
cc4d522
[SPARK-12625][SPARKR][SQL] replace R usage of Spark SQL deprecated API
felixcheung Jan 5, 2016
7058dc1
[SPARK-3873][EXAMPLES] Import ordering fixes.
Jan 5, 2016
53beddc
[SPARK-12568][SQL] Add BINARY to Encoders
marmbrus Jan 5, 2016
8eb2dc7
[SPARK-12641] Remove unused code related to Hadoop 0.23
sarutak Jan 5, 2016
1cdc42d
[SPARK-12331][ML] R^2 for regression through the origin.
iyounus Jan 5, 2016
b3c48e3
[SPARK-12438][SQL] Add SQLUserDefinedType support for encoder
viirya Jan 5, 2016
9a6ba7e
[SPARK-12643][BUILD] Set lib directory for antlr
viirya Jan 5, 2016
7676833
[SPARK-12480][FOLLOW-UP] use a single column vararg for hash
cloud-fan Jan 5, 2016
8ce645d
[SPARK-12615] Remove some deprecated APIs in RDD/SparkContext
rxin Jan 5, 2016
d202ad2
[SPARK-12439][SQL] Fix toCatalystArray and MapObjects
viirya Jan 5, 2016
047a31b
[SPARK-12617] [PYSPARK] Clean up the leak sockets of Py4J
zsxwing Jan 5, 2016
13a3b63
[SPARK-6724][MLLIB] Support model save/load for FPGrowthModel
yanboliang Jan 5, 2016
c26d174
[SPARK-12636] [SQL] Update UnsafeRowParquetRecordReader to support re…
nongli Jan 5, 2016
6cfe341
[SPARK-12511] [PYSPARK] [STREAMING] Make sure PythonDStream.registerS…
zsxwing Jan 5, 2016
1c6cf1a
[SPARK-12570][ML][DOC] DecisionTreeRegressor: provide variance of pre…
yanboliang Jan 5, 2016
78015a8
[SPARK-12450][MLLIB] Un-persist broadcasted variables in KMeans
rnowling Jan 5, 2016
ff89975
[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk
BrianLondon Jan 5, 2016
1537e55
[SPARK-12041][ML][PYSPARK] Add columnSimilarities to IndexedRowMatrix
vectorijk Jan 5, 2016
df8bd97
[SPARK-3873][SQL] Import ordering fixes.
Jan 6, 2016
0d42292
[SPARK-12504][SQL] Masking credentials in the sql plan explain output…
sureshthalamati Jan 6, 2016
70fe6ce
[SPARK-12659] fix NPE in UnsafeExternalSorter (used by cartesian prod…
Jan 6, 2016
7a375bb
[SPARK-3873][CORE] Import ordering fixes.
Jan 6, 2016
b3ba1be
[SPARK-3873][TESTS] Import ordering fixes.
Jan 6, 2016
d1fea41
[SPARK-12393][SPARKR] Add read.text and write.text for SparkR
yanboliang Jan 6, 2016
b2467b3
[SPARK-12578][SQL] Distinct should not be silently ignored when used …
viirya Jan 6, 2016
5d871ea
[SPARK-12340][SQL] fix Int overflow in the SparkPlan.executeTake, RDD…
QiangCai Jan 6, 2016
94c202c
[SPARK-12665][CORE][GRAPHX] Remove Vector, VectorSuite and GraphKryoR…
sarutak Jan 6, 2016
9061e77
[SPARK-11878][SQL] Eliminate distribute by in case group by is presen…
Jan 6, 2016
3b29004
[SPARK-7675][ML][PYSPARK] sparkml params type conversion
holdenk Jan 6, 2016
007da1a
[SPARK-11531][ML] SparseVector error Msg
rekhajoshm Jan 6, 2016
95eb651
[SPARK-11945][ML][PYSPARK] Add computeCost to KMeansModel for PySpark…
yanboliang Jan 6, 2016
3aa3488
[SPARK-11815][ML][PYSPARK] PySpark DecisionTreeClassifier & DecisionT…
yanboliang Jan 6, 2016
ea489f1
[SPARK-12573][SPARK-12574][SQL] Move SQL Parser from Hive to Catalyst
hvanhovell Jan 6, 2016
fcd013c
[SPARK-12006][ML][PYTHON] Fix GMM failure if initialModel is not None
zero323 Jan 6, 2016
f82ebb1
[SPARK-12368][ML][DOC] Better doc for the binary classification evalu…
BenFradet Jan 6, 2016
1e6648d
[SPARK-12617][PYSPARK] Move Py4jCallbackConnectionCleaner to Streaming
zsxwing Jan 6, 2016
19e4e9f
[SPARK-12672][STREAMING][UI] Use the uiRoot function instead of defau…
SaintBacchus Jan 6, 2016
cbaea95
Revert "[SPARK-12672][STREAMING][UI] Use the uiRoot function instead …
zsxwing Jan 6, 2016
6f7ba64
[SPARK-12681] [SQL] split IdentifiersParser.g into two files
Jan 6, 2016
917d3fc
[SPARK-12539][SQL] support writing bucketed table
cloud-fan Jan 7, 2016
ac56cf6
[SPARK-12604][CORE] Java count(AprroxDistinct)ByKey methods return Sc…
srowen Jan 7, 2016
a74d743
[SPARK-12640][SQL] Add simple benchmarking utility class and add Parq…
nongli Jan 7, 2016
6b6d02b
[SPARK-12663][MLLIB] More informative error message in MLUtils.loadLi…
Jan 7, 2016
8e19c76
[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0
JoshRosen Jan 7, 2016
174e72c
[SPARK-12673][UI] Add missing uri prepending for job description
jerryshao Jan 7, 2016
b673852
[SPARK-12678][CORE] MapPartitionsRDD clearDependencies
gpoulin Jan 7, 2016
e5cde7a
Revert "[SPARK-12006][ML][PYTHON] Fix GMM failure if initialModel is …
yhuai Jan 7, 2016
84e77a1
[DOC] fix 'spark.memory.offHeap.enabled' default value to false
zzcclp Jan 7, 2016
6a1c864
[SPARK-12295] [SQL] external spilling for window functions
Jan 7, 2016
fd1dcfa
[SPARK-12542][SQL] support except/intersect in HiveQl
Jan 7, 2016
8113dbd
[STREAMING][DOCS][EXAMPLES] Minor fixes
jaceklaskowski Jan 7, 2016
aa6ba74
address comments.
gatorsmile Jan 7, 2016
517559c
address comments.
gatorsmile Jan 7, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .rat-excludes
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,4 @@ org.apache.spark.sql.sources.DataSourceRegister
org.apache.spark.scheduler.SparkHistoryListenerFactory
.*parquet
LZ4BlockInputStream.java
spark-deps-.*
2 changes: 1 addition & 1 deletion R/pkg/.lintr
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, open_curly_linter(allow_single_line = TRUE), closed_curly_linter(allow_single_line = TRUE), commented_code_linter = NULL)
linters: with_defaults(line_length_linter(100), camel_case_linter = NULL, open_curly_linter(allow_single_line = TRUE), closed_curly_linter(allow_single_line = TRUE))
exclusions: list("inst/profile/general.R" = 1, "inst/profile/shell.R")
4 changes: 3 additions & 1 deletion R/pkg/NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ exportMethods("arrange",
"withColumnRenamed",
"write.df",
"write.json",
"write.parquet")
"write.parquet",
"write.text")

exportClasses("Column")

Expand Down Expand Up @@ -274,6 +275,7 @@ export("as.DataFrame",
"parquetFile",
"read.df",
"read.parquet",
"read.text",
"sql",
"table",
"tableNames",
Expand Down
61 changes: 44 additions & 17 deletions R/pkg/R/DataFrame.R
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,10 @@ setMethod("registerTempTable",
setMethod("insertInto",
signature(x = "DataFrame", tableName = "character"),
function(x, tableName, overwrite = FALSE) {
callJMethod(x@sdf, "insertInto", tableName, overwrite)
jmode <- convertToJSaveMode(ifelse(overwrite, "overwrite", "append"))
write <- callJMethod(x@sdf, "write")
write <- callJMethod(write, "mode", jmode)
callJMethod(write, "insertInto", tableName)
})

#' Cache
Expand Down Expand Up @@ -661,6 +664,34 @@ setMethod("saveAsParquetFile",
write.parquet(x, path)
})

#' write.text
#'
#' Saves the content of the DataFrame in a text file at the specified path.
#' The DataFrame must have only one column of string type with the name "value".
#' Each row becomes a new line in the output file.
#'
#' @param x A SparkSQL DataFrame
#' @param path The directory where the file is saved
#'
#' @family DataFrame functions
#' @rdname write.text
#' @name write.text
#' @export
#' @examples
#'\dontrun{
#' sc <- sparkR.init()
#' sqlContext <- sparkRSQL.init(sc)
#' path <- "path/to/file.txt"
#' df <- read.text(sqlContext, path)
#' write.text(df, "/tmp/sparkr-tmp/")
#'}
setMethod("write.text",
signature(x = "DataFrame", path = "character"),
function(x, path) {
write <- callJMethod(x@sdf, "write")
invisible(callJMethod(write, "text", path))
})

#' Distinct
#'
#' Return a new DataFrame containing the distinct rows in this DataFrame.
Expand Down Expand Up @@ -1948,18 +1979,15 @@ setMethod("write.df",
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
"org.apache.spark.sql.parquet")
}
allModes <- c("append", "overwrite", "error", "ignore")
# nolint start
if (!(mode %in% allModes)) {
stop('mode should be one of "append", "overwrite", "error", "ignore"')
}
# nolint end
jmode <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "saveMode", mode)
jmode <- convertToJSaveMode(mode)
options <- varargsToEnv(...)
if (!is.null(path)) {
options[["path"]] <- path
}
callJMethod(df@sdf, "save", source, jmode, options)
write <- callJMethod(df@sdf, "write")
write <- callJMethod(write, "format", source)
write <- callJMethod(write, "mode", jmode)
write <- callJMethod(write, "save", path)
})

#' @rdname write.df
Expand Down Expand Up @@ -2013,15 +2041,14 @@ setMethod("saveAsTable",
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
"org.apache.spark.sql.parquet")
}
allModes <- c("append", "overwrite", "error", "ignore")
# nolint start
if (!(mode %in% allModes)) {
stop('mode should be one of "append", "overwrite", "error", "ignore"')
}
# nolint end
jmode <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "saveMode", mode)
jmode <- convertToJSaveMode(mode)
options <- varargsToEnv(...)
callJMethod(df@sdf, "saveAsTable", tableName, source, jmode, options)

write <- callJMethod(df@sdf, "write")
write <- callJMethod(write, "format", source)
write <- callJMethod(write, "mode", jmode)
write <- callJMethod(write, "options", options)
callJMethod(write, "saveAsTable", tableName)
})

#' summary
Expand Down
Loading