-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-41800][BUILD] Upgrade commons-compress to 1.22 #39326
Conversation
<dependency> | ||
<groupId>org.apache.commons</groupId> | ||
<artifactId>commons-compress</artifactId> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use CountingOutputStream
class, so direct dependency should be added
spark/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
Lines 23 to 24 in 5300028
import org.apache.commons.compress.utils.CountingOutputStream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you know why this is required by 1.22 only?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will investigate root cause.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this isn't just a breaking change in commons-compress, then I feel like we have two different versions of common-compress in play, which could explain why just adding the dependency explicitly changed behavior
Can one of the admins verify this patch? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. It seems like we did need the explicit dependency for correctness anyway
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for Sean's comment.
Merged to master |
What changes were proposed in this pull request?
The pr aims to upgrade commons-compress from 1.21 to 1.22
Why are the changes needed?
This will bring the latest improvements.
![image](https://user-images.githubusercontent.com/15246973/210133282-18ae7d50-9076-4c95-a9e3-9ac6a266cda0.png)
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.