xgboost: Caused by: java.lang.ClassNotFoundException: scala.Serializable

[toktarev]

Tried to switch my project on java11.

I use xgboost (https://github.com/dmlc/xgboost) and scala inside, however
it failed with

        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.ClassNotFoundException: scala.Serializable
        at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
        at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?]

Is this scala's bug or I need to add some configuration parameters for JVM args ?
i use scala 2.13.1

[SethTisue]

scala.Serializable doesn't exist in Scala 2.13 except as a compile-time type alias; but it did exist in earlier Scala versions:

scala 2.12.10> classOf[scala.Serializable]
res0: Class[Serializable] = interface scala.Serializable

scala 2.13.1> classOf[scala.Serializable]
res0: Class[Serializable] = interface java.io.Serializable

So this error you are getting appears to be caused by using Scala 2.13 to run code that was compiled by some earlier Scala version. Is that possible?

Also, checking: are you saying that exact same thing worked on Java 8, but then when you changed nothing except switching to JDK 11, it stopped working? Or do you just happen to be on JDK 11, and you have a general suspicion that's relevant? Offhand, I don't see any this way the JDK version could matter here (though perhaps I'll be proved wrong as we dig deeper).

[SethTisue]

at search.maven.org I see xgboost-jvm_2.11 artifacts but no xgboost-jvm_2.12 or xgboost-jvm_2.13 artifacts, so that tends to indicate that xgboost requires Scala 2.11 specifically

[toktarev]

What is correct replacement for such code in scala 2.13.1 ?

package ml.dmlc.xgboost4j

/**
 * Labeled training data point.
 *
 * @param label Label of this point.
 * @param indices Feature indices of this point or `null` if the data is dense.
 * @param values Feature values of this point.
 * @param weight Weight of this point.
 * @param group Group of this point (used for ranking) or -1.
 * @param baseMargin Initial prediction on this point or `Float.NaN`
 */
case class LabeledPoint(
    label: Float,
    indices: Array[Int],
    values: Array[Float],
    weight: Float = 1f,
    group: Int = -1,
    baseMargin: Float = Float.NaN) extends Serializable {
  require(indices == null || indices.length == values.length,
    "indices and values must have the same number of elements")

  def this(label: Float, indices: Array[Int], values: Array[Float]) = {
    // [[weight]] default duplicated to disambiguate the constructor call.
    this(label, indices, values, 1.0f)
  }
}

I can't use

extends Serializable

in last Scala version ?

[toktarev]

---So this error you are getting appears to be caused by using Scala 2.13 to run code that was compiled by some earlier Scala version. Is that possible?

I compile xgboost source code together with all project.
I don't use maven xgboost

[toktarev]

--- Also, checking: are you saying that exact same thing worked on Java 8,
Yes, but with previous scala verion

---- but then when you changed nothing except switching to JDK 11, it stopped working?

It stopped working after switching to JDK11
After that I changed scala version on 2.13 and obtained corresponding error.

[toktarev]

I see

which means that code that typechecks may incur linkage errors at runtime
Here:

JDK 11 compatibility notes
Although the table above jumps from 8 to 11, JDK 9 and 10 will probably also work wherever 11 does. But unlike 9 and 10, 11 is an LTS release, so 11 is what we actually test on and recommend.
As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete. Notably, scalac will not enforce the restrictions of the Java Platform Module System, which means that code that typechecks may incur linkage errors at runtime. Scala 2.13.x will eventually provide rudimentary support for this, but likely only in nightlies built on Java 11.
JDK 11 support requires minimum sbt version 1.1.0, or 0.13.17 in the 0.13.x series.
To track progress on JDK 11 compatibility, watch:
the "Support JDK 11" issue
the jdk11 label in scala/bug
To help with testing on JDK 11, see scala/scala-dev#559.

Do we observe described bug ?

[toktarev]

---that was compiled by some earlier Scala version

How can I know which scala version will be used during compilation.

I don't have scala command at all on my laptop

[SethTisue]

please use https://users.scala-lang.org for questions. I don't think we have any Scala bugs here — perhaps you'll eventually uncover one, but you're not there yet

[SethTisue]

continued at https://users.scala-lang.org/t/xgboost-caused-by-java-lang-classnotfoundexception-scala-serializable/5445/2

[olafurpg]

For what it's worth, I hit on the same java.lang.ClassNotFoundException: scala.Serializable exception by writing an sbt plugin with scalaVersion := "2.13.1". If you're writing an sbt plugin, the following changes fixed the problem for me

 lazy val mySbtPlugin = project
   .settings(
     sbtPlugin := true,
+    scalaVersion := "2.12.10",
+    crossScalaVersions := List("2.12.10")

[Yaswi]

Hi @SethTisue , @olafurpg

I'm trying to upgrade to Spark 4.0 and Scala 2.13.16, I'm running into the same error:

Caused by: org.apache.spark.SparkException: Task failed while writing rows
	at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:167) ~[spark-core_2.13-4.0.1.jar:4.0.1]
	at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:90) ~[spark-core_2.13-4.0.1.jar:4.0.1]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) ~[spark-core_2.13-4.0.1.jar:4.0.1]
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171) ~[spark-core_2.13-4.0.1.jar:4.0.1]

Caused by: java.lang.NoClassDefFoundError: scala.Serializable
	at java.base/java.lang.ClassLoader.defineClassImpl(Native Method)
	at java.base/java.lang.ClassLoader.defineClassInternal(Unknown Source)
	at java.base/java.lang.ClassLoader.defineClass(Unknown Source)

Can you please help me with this error? My project is on Java 17

[som-snytt]

@Yaswi there was a bug related to package object loading in scalac -- but I see (after glancing at the long thread) that this is a classloading or versioning issue. I'd suggest asking on the forum.