Skip to content

Commit

Permalink
[SPARK-31163][SQL] TruncateTableCommand with acl/permission should ha…
Browse files Browse the repository at this point in the history
…ndle non-existed path

### What changes were proposed in this pull request?

This fix #26956
Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of the path doesn't exist.

### Why are the changes needed?

`truncate table` may fail to re-create path in case of interruption or something else. As a result, next time we `truncate table` on the same table with acl/permission, it will fail due to `FileNotFoundException`. And it also brings behavior change compares to previous Spark version, which could still `truncate table` successfully even if the path doesn't exist.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Added UT.

Closes #27923 from Ngone51/fix_truncate.

Authored-by: yi.wu <yi.wu@databricks.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
Ngone51 authored and dongjoon-hyun committed Mar 16, 2020
1 parent 6704103 commit cb26f63
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -508,8 +508,8 @@ case class TruncateTableCommand(
var optPermission: Option[FsPermission] = None
var optAcls: Option[java.util.List[AclEntry]] = None
if (!ignorePermissionAcl) {
val fileStatus = fs.getFileStatus(path)
try {
val fileStatus = fs.getFileStatus(path)
optPermission = Some(fileStatus.getPermission())
} catch {
case NonFatal(_) => // do nothing
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2084,6 +2084,27 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}

test("SPARK-31163: acl/permission should handle non-existed path when truncating table") {
withSQLConf(SQLConf.TRUNCATE_TABLE_IGNORE_PERMISSION_ACL.key -> "false") {
withTable("tab1") {
sql("CREATE TABLE tab1 (col1 STRING, col2 INT) USING parquet PARTITIONED BY (col2)")
sql("INSERT INTO tab1 SELECT 'one', 1")
checkAnswer(spark.table("tab1"), Row("one", 1))
val part = spark.sessionState.catalog.listPartitions(TableIdentifier("tab1")).head
val path = new File(part.location.getPath)
sql("TRUNCATE TABLE tab1")
// simulate incomplete/unsuccessful truncate
assert(path.exists())
path.delete()
assert(!path.exists())
// execute without java.io.FileNotFoundException
sql("TRUNCATE TABLE tab1")
// partition path should be re-created
assert(path.exists())
}
}
}

test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
Expand Down

0 comments on commit cb26f63

Please sign in to comment.