Skip to content

Commit

Permalink
[SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from inva…
Browse files Browse the repository at this point in the history
…lid path throws exception'

### What changes were proposed in this pull request?
The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'.

### Why are the changes needed?
- As discussed and analyzed in [41271](#41271 (comment))

- There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

Closes #41289 from panbingkun/SPARK-43767.

Authored-by: panbingkun <pbk1982@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
panbingkun authored and dongjoon-hyun committed May 24, 2023
1 parent 1e17c86 commit a72d1aa
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1350,7 +1350,7 @@ abstract class AvroSuite
}

intercept[FileNotFoundException] {
withTempPath { dir =>
withTempDir { dir =>
FileUtils.touch(new File(dir, "test"))
withSQLConf(AvroFileFormat.IgnoreFilesWithoutExtensionProperty -> "true") {
spark.read.format("avro").load(dir.toString)
Expand All @@ -1359,7 +1359,7 @@ abstract class AvroSuite
}

intercept[FileNotFoundException] {
withTempPath { dir =>
withTempDir { dir =>
FileUtils.touch(new File(dir, "test"))

spark
Expand Down

0 comments on commit a72d1aa

Please sign in to comment.