-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from invalid path throws exception' #41289
Conversation
…lid path throws exception'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for making an independent PR. It seems that you want to use withTempDir
instead of withTempPath + mkdirs
. Did I understand correctly?
The root cause for the failure of this case is that the parent directory was not created.
spark/core/src/test/scala/org/apache/spark/SparkFunSuite.scala
Lines 237 to 242 in 1e17c86
protected def withTempDir(f: File => Unit): Unit = { | |
val dir = Utils.createTempDir() | |
try f(dir) finally { | |
Utils.deleteRecursively(dir) | |
} | |
} |
Yep, Perhaps using |
…lid path throws exception'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you for updates.
…lid path throws exception' ### What changes were proposed in this pull request? The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'. ### Why are the changes needed? - As discussed and analyzed in [41271](apache#41271 (comment)) - There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes apache#41289 from panbingkun/SPARK-43767. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'.
Why are the changes needed?
As discussed and analyzed in 41271
There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.