Binary incompatibility between 2.13.16 and 2.13.17: MurmurHash3.caseClassHash$default$2() missing in 2.13.16

[alicez-db]

Summary

When running Apache Spark 4.1.0 with Scala 2.13.16 on the runtime classpath, Spark fails with a NoSuchMethodError on scala.util.hashing.MurmurHash3$.caseClassHash$default$2().

Filing to ask whether this method is expected to exist in Scala 2.13.16, or whether it was introduced in a later 2.13.x patch.

Reproduction

1. Use Apache Spark 4.1.0
2. Use Scala 2.13.16 on the runtime classpath
3. Run any Spark code that exercises Catalyst (e.g. a basic DataFrame operation that triggers TreeNode.hashCode)

Observed error

java.lang.NoSuchMethodError: 'java.lang.String scala.util.hashing.MurmurHash3$.caseClassHash$default$2()'
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$_hashCode$1(TreeNode.scala:226)
  at org.apache.spark.util.BestEffortLazyVal.apply(BestEffortLazyVal.scala:53)
  at org.apache.spark.sql.catalyst.trees.TreeNode.hashCode(TreeNode.scala:227)

Questions

1. Does MurmurHash3.caseClassHash$default$2() exist in Scala 2.13.16, or was it introduced in a later 2.13.x patch?
2. If it does not exist in 2.13.16, is that an expected outcome of patch version differences, or unintentional?
3. Is there a recommended Scala 2.13.x patch version range for running Apache Spark 4.1.0?

Environment

• Scala 2.13.16 (runtime)
• Apache Spark 4.1.0
• JDK 17

Context

I work at Databricks. A customer hit this while running Spark 4.1.0 on a runtime that ships Scala 2.13.16. Filing here for clarity on the Scala side. Happy to provide a minimal reproducer if helpful.

[som-snytt]

The method was introduced in scala/scala#11023 to address a bug.

There is some documentation at
https://www.scala-lang.org/api/2.13.18/scala/util/hashing/MurmurHash3$.html#caseClassHash(x:Product,caseClassName:String):Int

The stable API productHash was merely deprecated; it is equivalent to using the "default arg" (default$2()) with the new method.

There are a few notes on compatibility at
#13136

I'm sure @lrytz will provide more words of guidance.

[lrytz]

Hm... scala/scala#11023 was a complicated PR in terms of compatibility considerations. The compiler generates code that calls into the standard library. We had to make sure that code generated by the compiler before and after would run correctly with both a standard library before and after.

The deprecation / rename of productHash to caseClassHash was a cleanup on the side. I believe the fact that this is not forwards binary compatible simply slipped. We should figure out why MiMa didn't catch it.

For most Scala projects this error will not manifest at runtime, because sbt actually implements the SIP-51 restriction. Given a build.sbt

ThisBuild / scalaVersion := "2.13.16"
libraryDependencies += "org.apache.spark" %% "spark-core" % "4.1.0"

we get

sbt:sbtproj> compile
[error] stack trace is suppressed; run last scalaInstance for the full output
[error] (scalaInstance) Expected `sbtproj / scalaVersion` to be 2.13.17 or later, but found 2.13.16.
[error] To support backwards-only binary compatibility (SIP-51), the Scala 2.13 compiler
[error] should not be older than scala-library on the dependency classpath.
[error]
[error] Upgrade the `scalaVersion` to fix the build. If upgrading the Scala compiler version is
[error] not possible (for example due to a regression in the compiler or a missing dependency),
[error] this error can be demoted by setting `allowUnsafeScalaLibUpgrade := true`.
[error]
[error] See `sbtproj / evicted` to know why scala-library 2.13.17 is getting pulled in.
[error] Total time: 0 s, completed 21 May 2026, 15:05:16

But on a hosted runtime that provides the 2.13.16 standard library, code compiled against 2.13.17 can fail in this way.

[eed3si9n]

So does this mean that SIP-51 (accepted), which proposed to drop forward bincompat requirement for Scala 2.13 patch releases, is now retroactively in effect since Scala 2.13.17?