-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12961][CORE][FOLLOW-UP] Remove wrapper code for SnappyOutputStream #18949
Conversation
Probably, we forgot to remove this when upgrading |
Also, we could safely remove the code (commit: 5936bf9) to avoid memory leak in |
Test build #80673 has finished for PR 18949 at commit
|
LGTM |
@maropu There is another reason we leave the workaround in place: #11524 (comment) |
LGTM |
@viirya aha, ok. thanks. (btw, since the comment is still important, we better keep it in code comment, maybe). |
ping |
Previously, the concern was that this workaround was still necessary because the runtime environment might have a different snappy version from the one Spark uses. I thought that was because Hadoop provides it, but, I am not sure that's true? I can't see a dependency on it in Hadoop, and I think this is likely OK, but wonder if I'm missing something @viirya as you commented on it? |
If I understand the previous comment #11524 (comment) correctly, seems it is happened before snappy-java get downgraded by user code when the user classpath takes precedence. I am not sure where the snappy-java comes from, I don't know if it is Hadoop, maybe other library in user classpath? If we don't shade this dep, maybe we still keep it for safety? |
Hm, if the risk is only that the user classpath might have an older version, then at some point that's an acceptable risk. There are many problems like that, and this version has been out for over a year. Still, if it's not causing us problems except for a bit of extra code, we could also just err on the side of caution and push this out another 6-12 months or something. |
Ok. It sounds reasonable to me. |
Sounds good to me ,too. |
Closes apache#18522 Closes apache#17722 Closes apache#18879 Closes apache#18891 Closes apache#18806 Closes apache#18948 Closes apache#18949 Closes apache#19070 Closes apache#19039 Closes apache#19142 Closes apache#18515 Closes apache#19154 Closes apache#19162 Closes apache#19187
What changes were proposed in this pull request?
This pr removed the wrapper code (commit: f2cc6b5) to avoid the bug in
SnappyOutputStream.close()
because the corresponding bug has been fixed insnappy-java-1.1.2.6
that Spark currently uses. This fix has been merged insnappy-java-1.1.2
and see xerial/snappy-java#107 (comment).How was this patch tested?
Checked
UnsafeShuffleWriterSuite
passed.