Add support for Spark 3-based builds #9524
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, I bumped into the Breeze bug a while back while trying to upgrade Hail to Spark 3 internally, and realized it'd be a blocker downstream. I've only seen issues on the Python side; for example:
Turns out I needed to pull in the transitive breeze dependencies; without these, the Python tests failed while the Scala tests passed. I've re-run the tests on Spark 3 now; let me know if there are any other changes you'd like to see. Unfortunately, there's a small bug I saw on the Spark 3 MLLib side; I'll make sure this gets addressed ASAP: https://issues.apache.org/jira/browse/SPARK-33043