Skip to content

Commit

Permalink
[SPARK-22788][STREAMING] Use correct hadoop config for fs append supp…
Browse files Browse the repository at this point in the history
…ort.

Still look at the old one in case any Spark user is setting it
explicitly, though.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #19983 from vanzin/SPARK-22788.
  • Loading branch information
Marcelo Vanzin authored and squito committed Dec 20, 2017
1 parent 9962390 commit 7570eab
Showing 1 changed file with 3 additions and 1 deletion.
Expand Up @@ -29,7 +29,9 @@ private[streaming] object HdfsUtils {
// If the file exists and we have append support, append instead of creating a new file
val stream: FSDataOutputStream = {
if (dfs.isFile(dfsPath)) {
if (conf.getBoolean("hdfs.append.support", false) || dfs.isInstanceOf[RawLocalFileSystem]) {
if (conf.getBoolean("dfs.support.append", true) ||
conf.getBoolean("hdfs.append.support", false) ||
dfs.isInstanceOf[RawLocalFileSystem]) {
dfs.append(dfsPath)
} else {
throw new IllegalStateException("File exists and there is no append support!")
Expand Down

0 comments on commit 7570eab

Please sign in to comment.