Skip to content

Commit

Permalink
[SPARK-32172][CORE] Use createDirectory instead of mkdir
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Use Files.createDirectory() to create local directory instead of File.mkdir() in DiskBlockManager.
Many times, we will see such error log information like "Failed to create local dir in xxxxxx". But there is no clear information indicating why the directory creation failed.
When Files.createDirectory() fails to create a local directory, it can give specific error information for subsequent troubleshooting(also throws IOException).

### Why are the changes needed?

Throw clear error message when creating directory fails.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

`DiskBlockManagerSuite`

Closes #28997 from sidedoorleftroad/SPARK-32172.

Authored-by: sidedoorleftroad <sidedoorleftroad@163.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
sidedoorleftroad authored and dongjoon-hyun committed Jul 6, 2020
1 parent 1d18096 commit 3fe3365
Showing 1 changed file with 3 additions and 2 deletions.
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.storage

import java.io.{File, IOException}
import java.nio.file.Files
import java.util.UUID

import org.apache.spark.SparkConf
Expand Down Expand Up @@ -69,8 +70,8 @@ private[spark] class DiskBlockManager(conf: SparkConf, deleteFilesOnStop: Boolea
old
} else {
val newDir = new File(localDirs(dirId), "%02x".format(subDirId))
if (!newDir.exists() && !newDir.mkdir()) {
throw new IOException(s"Failed to create local dir in $newDir.")
if (!newDir.exists()) {
Files.createDirectory(newDir.toPath)
}
subDirs(dirId)(subDirId) = newDir
newDir
Expand Down

0 comments on commit 3fe3365

Please sign in to comment.