-
Notifications
You must be signed in to change notification settings - Fork 530
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Breeze dependency conflict in Anomaly Detection Spark 3.4+ #545
Conversation
Hi @rdsharma26, what do you think about updating Breeze version? I wonder if there are other workaround to make Anomaly Detection works on more modern version of spark? |
The change looks good. Let me get back to you after understanding how this change affects our internal Spark 3.3 / 3.1 branches. |
Hi @rdsharma26, I just want to check the status of this. Are there any things I can help with (testing 3.3, 3.1, etc.) |
@zeotuan Apologies for the delayed response. Would it be possible for you to check how this change works against the |
Hi @rdsharma26, breeze 2.1.0 is not compatible with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. We will keep this on the master branch, but change to a different version on the non 3.4 Spark release version branches.
I think this would fix PyDeequ's upgrade to PySpark 3.4 as well, see errors related to breeze here https://github.com/awslabs/python-deequ/actions/runs/8886301683/job/24399475419?pr=203
|
in scala spark 3.4.1, deequ 2.0.7-spark-3.5 works not spark-3.4. |
Update breeze version to 2.1 to match with current spark-mlib 3.4 and spark-mlib 3.5 breeze dependency version. This would allow people migrating to spark 3.4+ to use anomaly detection without dependency conflict issue that is mentioned in
#336
#393
#428
#428
Also Breeze 0.13.2 has several security vulnerabilities which was solve in breeze 2.1.0
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.