Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[jvm-packages] cross-version spark support #4350
@hcho3 I am going to work on support spark 2.4.1 and have compatibility test over spark 2.3
my plan is to trigger two builds for spark 2.4/2.3 respectively and also have version-specific test to ensure the compatibility
shall I wait for java worker to be ready in jenkins or I should work on travis?
I would limit the definition of cross-version support to "support loading models trained in previous version" in XGBoost
I have done several experiments on running a spark-2.4-built xgboost with spark 2.3 or vice versa. The most significant problem is from the library which Spark depends on and brings some breaking changes by their own. In that way we cannot guarantee a spark 2.4 built version can be run with spark 2.3 runtimes
even to "support loading models trained in previous version", we need some code to handle (1) breaking changes in XGBoost parameters, e.g. reg:linear doesn't exist anymore; (2) breaking changes in Spark, e.g. vectorAssembler will fail with Float.NaN by default...