[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

panbingkun · 2023-05-23T02:16:05Z

What changes were proposed in this pull request?

The pr aims to upgrade common-io from 2.11.0 to 2.12.0.

Why are the changes needed?

common-io new version includes some improvement & bug fixed, eg

IO-790 Fix symbolic link file filter commons-io#450
[IO-776] Fix parameters to requireNonNull call commons-io#368
Add PathUtils.touch(Path) The error exception of the FileUtils.touch method has been changed from java.io.FileNotFoundException to java.nio.file.NoSuchFileException
common-io 2.11.0 VS 2.12.0
apache/commons-io@rel/commons-io-2.11.0...rel/commons-io-2.12.0

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

…t version

LuciferYang · 2023-05-23T02:21:02Z

Please don't upgrade commons-crypto, there is an existing issue for Apple Silicon.

#40082 (comment)

panbingkun · 2023-05-23T02:39:38Z

#40082 (comment)

Ok, Let me revert it.

dongjoon-hyun

+1, LGTM.

dongjoon-hyun

Could you check the avro failures and re-trigger the failed pipeline?

[info] *** 2 TESTS FAILED ***
[error] Failed: Total 294, Failed 2, Errors 0, Passed 292, Ignored 2
[error] Failed tests:
[error] 	org.apache.spark.sql.avro.AvroV1Suite
[error] 	org.apache.spark.sql.avro.AvroV2Suite

panbingkun · 2023-05-23T10:41:15Z

Could you check the avro failures and re-trigger the failed pipeline?


[info] *** 2 TESTS FAILED ***

[error] Failed: Total 294, Failed 2, Errors 0, Passed 292, Ignored 2

[error] Failed tests:

[error] 	org.apache.spark.sql.avro.AvroV1Suite

[error] 	org.apache.spark.sql.avro.AvroV2Suite

OK，let me check it

dongjoon-hyun · 2023-05-23T11:30:59Z

connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

@@ -1349,7 +1349,7 @@ abstract class AvroSuite
      spark.read.format("avro").load("*/*/*/*/*/*/*/something.avro")
    }

-    intercept[FileNotFoundException] {
+    intercept[IOException] {


This looks like a breaking change because this is a kind of information loss.
Could you mention this to the PR description?

Do you know what is the background of this change?

Please do not merge it until I have investigated it thoroughly.

Add PathUtils.touch(Path)

Suggested change

intercept[IOException] {

intercept[java.nio.file.NoSuchFileException] {

?

panbingkun · 2023-05-23T11:47:46Z

2.11.0 FileUtils.touch, Call stack as follows
1.FileUtils.touch
2.FileUtils.setLastModified
3.File.setLastModified(timeMillis)
4.FileSystem.setLastModifiedTime(this, time)
5.UnixFileSystem.setLastModifiedTime --> Found that the file does not exist, throw FileNotFoundException

2.12.0 FileUtils.touch, Call stack as follows
1.FileUtils.touch
2.PathUtils.touch
3.java.nio.file.Files.createFile
4.java.nio.file.Files.newByteChannel
5.sun.nio.fs.UnixFileSystemProvider.newByteChannel
6.sun.nio.fs.UnixChannelFactory.newFileChannel
7.sun.nio.fs.UnixChannelFactory.open
8.UnixNativeDispatcher.open
9.(Native)UnixNativeDispatcher.open0 -> throw UnixException
10.UnixException.rethrowAsIOException
10.UnixException.translateToIOException --> Convert to java.nio.file.NoSuchFileException throw it

dongjoon-hyun · 2023-05-23T15:39:40Z

Please put the analysis to the PR description.
+1 for NoSuchFileException because it's better than a general IOException.

panbingkun · 2023-05-24T01:29:49Z

Please put the analysis to the PR description.

+1 for NoSuchFileException because it's better than a general IOException.

This is done.

panbingkun · 2023-05-24T02:03:18Z

@dongjoon-hyun @HyukjinKwon @LuciferYang
In fact, there is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).

After this PR is completed, I will submit a new PR to fix this issue.

The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error.

…lid path throws exception' ### What changes were proposed in this pull request? The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'. ### Why are the changes needed? - As discussed and analyzed in [41271](#41271 (comment)) - There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes #41289 from panbingkun/SPARK-43767. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun

I merged #41289 . Please rebase this PR to the master branch.

panbingkun · 2023-05-24T04:54:58Z

I merged #41289 . Please rebase this PR to the master branch.

This is done.

LuciferYang · 2023-05-24T05:02:26Z

2.Note: The error exception of the FileUtils.touch method has been changed from java.io.FileNotFoundException to java.nio.file.NoSuchFileException, changes from PR: [Add PathUtils.touch(Path)](https://github.com/apache/commons-io/commit/fd7c8182d2117d01f43ccc9fe939105f834ba672). The analysis process is as follows:

still need this part in description? @panbingkun

dongjoon-hyun · 2023-05-24T05:18:55Z

Right, @LuciferYang . We can remove that from this PR description.

panbingkun · 2023-05-24T05:29:55Z

Right, @LuciferYang . We can remove that from this PR description.

Ok, Let me update it.

dongjoon-hyun · 2023-05-24T15:00:37Z

Thank you, @panbingkun and all! Merged to master.

…lid path throws exception' ### What changes were proposed in this pull request? The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'. ### Why are the changes needed? - As discussed and analyzed in [41271](apache#41271 (comment)) - There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes apache#41289 from panbingkun/SPARK-43767. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? The pr aims to upgrade common-io from 2.11.0 to 2.12.0. ### Why are the changes needed? common-io new version includes some improvement & bug fixed, eg - apache/commons-io#450 - apache/commons-io#368 - [Add PathUtils.touch(Path)](apache/commons-io@fd7c818) The error exception of the FileUtils.touch method has been changed from `java.io.FileNotFoundException` to `java.nio.file.NoSuchFileException` - common-io 2.11.0 VS 2.12.0 apache/commons-io@rel/commons-io-2.11.0...rel/commons-io-2.12.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes apache#41271 from panbingkun/SPARK-43739. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-43739][BUILD] Upgrade 'commons-io' & 'commons-crypto' to newes…

6c7beeb

…t version

github-actions bot added the BUILD label May 23, 2023

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0

177c083

panbingkun changed the title ~~[SPARK-43739][BUILD] Upgrade commons-io & commons-crypto to newest version~~ [SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 May 23, 2023

HyukjinKwon approved these changes May 23, 2023

View reviewed changes

dongjoon-hyun approved these changes May 23, 2023

View reviewed changes

LuciferYang approved these changes May 23, 2023

View reviewed changes

dongjoon-hyun reviewed May 23, 2023

View reviewed changes

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0

2e39540

github-actions bot added AVRO SQL labels May 23, 2023

dongjoon-hyun reviewed May 23, 2023

View reviewed changes

panbingkun changed the title ~~[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0~~ [WIP][SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 May 23, 2023

dongjoon-hyun marked this pull request as draft May 23, 2023 15:40

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0

fdd6f0d

panbingkun marked this pull request as ready for review May 24, 2023 01:29

panbingkun changed the title ~~[WIP][SPARK-43739][BUILD] Upgrade commons-io to 2.12.0~~ [SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 May 24, 2023

panbingkun mentioned this pull request May 24, 2023

[SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from invalid path throws exception' #41289

Closed

dongjoon-hyun reviewed May 24, 2023

View reviewed changes

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0

f5f82af

github-actions bot removed AVRO SQL labels May 24, 2023

dongjoon-hyun approved these changes May 24, 2023

View reviewed changes

LuciferYang approved these changes May 24, 2023

View reviewed changes

dongjoon-hyun closed this in ee27e85 May 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

panbingkun commented May 23, 2023 •

edited

Loading

LuciferYang commented May 23, 2023

panbingkun commented May 23, 2023

dongjoon-hyun left a comment

dongjoon-hyun left a comment

panbingkun commented May 23, 2023

dongjoon-hyun May 23, 2023

dongjoon-hyun May 23, 2023

panbingkun May 23, 2023

panbingkun May 23, 2023

LuciferYang May 23, 2023 •

edited

Loading

panbingkun May 24, 2023

panbingkun commented May 23, 2023 •

edited

Loading

dongjoon-hyun commented May 23, 2023

panbingkun commented May 24, 2023

panbingkun commented May 24, 2023 •

edited

Loading

dongjoon-hyun left a comment

panbingkun commented May 24, 2023

LuciferYang commented May 24, 2023 •

edited

Loading

dongjoon-hyun commented May 24, 2023

panbingkun commented May 24, 2023

dongjoon-hyun commented May 24, 2023

	intercept[IOException] {
	intercept[java.nio.file.NoSuchFileException] {

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

Conversation

panbingkun commented May 23, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

LuciferYang commented May 23, 2023

panbingkun commented May 23, 2023

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented May 23, 2023

dongjoon-hyun May 23, 2023

Choose a reason for hiding this comment

dongjoon-hyun May 23, 2023

Choose a reason for hiding this comment

panbingkun May 23, 2023

Choose a reason for hiding this comment

panbingkun May 23, 2023

Choose a reason for hiding this comment

LuciferYang May 23, 2023 • edited Loading

Choose a reason for hiding this comment

panbingkun May 24, 2023

Choose a reason for hiding this comment

panbingkun commented May 23, 2023 • edited Loading

dongjoon-hyun commented May 23, 2023

panbingkun commented May 24, 2023

panbingkun commented May 24, 2023 • edited Loading

dongjoon-hyun left a comment

Choose a reason for hiding this comment

panbingkun commented May 24, 2023

LuciferYang commented May 24, 2023 • edited Loading

dongjoon-hyun commented May 24, 2023

panbingkun commented May 24, 2023

dongjoon-hyun commented May 24, 2023

panbingkun commented May 23, 2023 •

edited

Loading

LuciferYang May 23, 2023 •

edited

Loading

panbingkun commented May 23, 2023 •

edited

Loading

panbingkun commented May 24, 2023 •

edited

Loading

LuciferYang commented May 24, 2023 •

edited

Loading