diff --git a/doc/jvm/xgboost4j_spark_tutorial.rst b/doc/jvm/xgboost4j_spark_tutorial.rst index 3315004a4b49..e5248d0591a2 100644 --- a/doc/jvm/xgboost4j_spark_tutorial.rst +++ b/doc/jvm/xgboost4j_spark_tutorial.rst @@ -40,6 +40,21 @@ in order to add XGBoost4J-Spark as a dependency for your project. We provide bot Also, make sure to install Spark directly from `Apache website `_. **Upstream XGBoost is not guaranteed to work with third-party distributions of Spark, such as Cloudera Spark.** Consult appropriate third parties to obtain their distribution of XGBoost. +.. warning:: Spark 4.0.0 is not compatible with XGBoost4J-Spark + + Apache Spark 4.0.0 introduced a breaking change to the ``org.apache.spark.ml.param.Param`` class + constructor (`SPARK-52259 `_), which causes a + ``NoSuchMethodError`` when instantiating any XGBoost estimator: + + .. code-block:: text + + java.lang.NoSuchMethodError: 'void org.apache.spark.ml.param.Param.( + org.apache.spark.ml.util.Identifiable, java.lang.String, java.lang.String, scala.Function1)' + + This issue affects all third-party ML libraries that use ``Param``. It was fixed in + **Spark 4.0.1** and **Spark 4.1.0**. If you are using Spark 4.x, please upgrade to + **Spark 4.0.1 or later**. + Data Preparation ================ diff --git a/doc/jvm/xgboost_spark_migration.rst b/doc/jvm/xgboost_spark_migration.rst index 976acd5bb6d4..49276c979570 100644 --- a/doc/jvm/xgboost_spark_migration.rst +++ b/doc/jvm/xgboost_spark_migration.rst @@ -113,3 +113,18 @@ Starting from xgboost4j-spark 3.0, below parameters are removed. - singlePrecisionHistogram - lambdaBias - objectiveType + +*************************** +Spark 4.0 Compatibility +*************************** + +XGBoost4J-Spark JARs built against Spark 3.5 are binary compatible with **Spark 4.0.1+** and +**Spark 4.1.0+**. No code changes are required. + +.. warning:: + + **Spark 4.0.0** is **not** compatible due to an upstream bug + (`SPARK-52259 `_) that changed the + constructor signature of ``org.apache.spark.ml.param.Param``. This causes a + ``NoSuchMethodError`` at runtime when instantiating any XGBoost estimator. The fix is included + in Spark 4.0.1 and later. Please avoid Spark 4.0.0 and upgrade to at least **Spark 4.0.1**.