Scala 3.3.2: class file ... is broken, reading aborted with class java.util.NoSuchElementException contextual$

[mkurz]

Asking @SethTisue @lrytz @som-snytt @sjrd @smarter for advice.

Just yesterday I made Play build with with 3.3.2:

• Upgrade to latest scala versions #12401

Now I run into a problem which IMHO is caused by

• scala/scala3@a10a271
• (which is cherry-picked from Fix overcompilation due to unstable context bound desugaring scala/scala3#18280)

How to reproduce:

1. Checkout Play locally with current main (3ace273)
2. Build Play locally: sbt +publishLocal
3. Clone the play-samples repo: https://github.com/playframework/play-samples/

   git clone git@github.com:playframework/play-samples.git
   # Make sure you are on the 3.0.x branch (default anyway)
   

4. Within the samples repo:

   cd play-java-chatroom-example
   

5. Make sure to apply the local Play snapshot:

   --- a/play-java-chatroom-example/project/plugins.sbt
   +++ b/play-java-chatroom-example/project/plugins.sbt
   @@ -1,2 +1,2 @@
    // The Play plugin
   -addSbtPlugin("org.playframework" % "sbt-plugin" % "3.0.1")
   +addSbtPlugin("org.playframework" % "sbt-plugin" % "3.1.0-SNAPSHOT")

6. Currently, the sample project is using Scala 3.3.1 (but Play is now build with 3.3.2). Now try to compile the project:

   $ sbt ++3.x compile
   [info] welcome to sbt 1.9.8 (Eclipse Adoptium Java 11.0.22)
   ...
   [info] Version 3.1.0-SNAPSHOT running Java 11.0.22
   ...
   [info] Setting Scala version to 3.3.1 on 1 projects.
   ...
   [info] compiling 5 Scala sources and 3 Java sources to ./play-samples/play-java-chatroom-example/target/scala-3.3.1/classes ...
   [error] error while loading PathBindableMacros,
   [error] class file play/api/mvc/PathBindableMacros.class is broken, reading aborted with class java.util.NoSuchElementException
   [error] contextual$
   [error] one error found
   [error] (Compile / compileIncremental) Compilation failed
   [error] Total time: 2 s, completed Feb 27, 2024, 4:10:59 PM
   

7. Changing the Scala version of the project to 3.3.2 makes compiliation work however (in build.sbt: crossScalaVersions := Seq("2.13.12", "3.3.2"),)

So this is the file which this problem is about: https://github.com/playframework/playframework/blob/main/core/play/src/main/scala-3/play/api/mvc/PathBindableMacros.scala

If I build Play with Scala 3.3.1 by reverting yesterdays commit (git revert -m 1 552f5a0ae1876804b06114a94b619712a22c46c4), I can run the samples project again with Scala 3.3.1 set in the project.

Now what I found out is that the tasty files generated by Play with Scala 3.3.1 vs Scala 3.3.2 differ:

• The file ./core/play/target/scala-3.3.1/classes/play/api/mvc/PathBindableMacros.tasty contains evidence$
• The file ./core/play/target/scala-3.3.2/classes/play/api/mvc/PathBindableMacros.tasty replaced evidence$ with contextual$

I also inspected the class files

• ./core/play/target/scala-3.3.1/classes/play/api/mvc/PathBindableMacros.class
• ./core/play/target/scala-3.3.2/classes/play/api/mvc/PathBindableMacros.class

with javap -v and it looks like they are identical (except some bytes in the tasty part):

--- /tmp/3.3.1_javap	2024-02-27 16:43:43.766644833 +0100
+++ /tmp/3.3.2_javap	2024-02-27 16:43:47.259650233 +0100
@@ -1,5 +1,5 @@
-Classfile ./core/play/target/scala-3.3.1/classes/play/api/mvc/PathBindableMacros.class
+Classfile ./core/play/target/scala-3.3.2/classes/play/api/mvc/PathBindableMacros.class
   Last modified Feb 27, 2024; size 175 bytes
-  MD5 checksum 604088e2d12cc976c4f58b7d109da0e7
+  MD5 checksum 144a574029f956604ac0ae298c323c5f
   Compiled from "PathBindableMacros.scala"
 public interface play.api.mvc.PathBindableMacros
@@ -23,5 +23,5 @@
 SourceFile: "PathBindableMacros.scala"
   TASTY: length = 0x10 (unknown attribute)
-   00 98 FE E0 E7 74 F5 C5 00 66 D7 56 61 0B CB 19
+   00 B6 6E 0F BB C2 58 C5 00 66 D7 56 61 0B CB 19
 
   Scala: length = 0x0 (unknown attribute)

[mkurz]

What I ask myself is: If I build Play with 3.3.2 will it be necessary to make people also upgrade to Scala 3.3.2 in their projects? I thought this would not be necessary (but please correct me if I am wrong).
Thanks for your help!

[smarter]

If I build Play with 3.3.2 will it be necessary to make people also upgrade to Scala 3.3.2 in their projects?

No, that should not be necessary. I'll investigate your issue.

[smarter]

Running with -Ydebug gives more info:

java.util.NoSuchElementException: contextual$
        at dotty.tools.dotc.util.ReadOnlyMap.apply(ReadOnlyMap.scala:29)
        at dotty.tools.dotc.core.tasty.TastyUnpickler.readNameContents(TastyUnpickler.scala:72)
        at dotty.tools.dotc.core.tasty.TastyUnpickler.$init$$$anonfun$1(TastyUnpickler.scala:94)
        at dotty.tools.tasty.TastyReader.until(TastyReader.scala:125)
        at dotty.tools.dotc.core.tasty.TastyUnpickler.<init>(TastyUnpickler.scala:94)
        at dotty.tools.dotc.core.tasty.TastyUnpickler.<init>(TastyUnpickler.scala:34)
        at dotty.tools.dotc.core.tasty.DottyUnpickler.<init>(DottyUnpickler.scala:48)

/cc @bishabosha

[mkurz]

I have my doubts right now that Scala 3.3.1 can handle the Tasty files Scala 3.3.2 generates just by looking (grep'ing) at the source code:

./dotty ((3.3.1))$ grep -ir 'contextual\$'
./dotty ((3.3.2))$
# No results

./dotty ((3.3.1))$ grep -ir 'ContextFunctionParamName'
./dotty ((3.3.1))$ 
# No results

./dotty ((3.3.2))$ grep -ir 'contextual\$'
tests/neg/i11350.check:  |                                       Could not infer type for parameter contextual$1 of anonymous function
tests/neg/i11350.check:  |                                       Could not infer type for parameter contextual$2 of anonymous function
tests/run-staging/quote-nested-5.check:  ((q2: scala.quoted.Quotes) ?=> ((contextual$2: scala.quoted.Quotes) ?=> a).apply(using q2))
tests/run-staging/quote-nested-2.check:  ((contextual$2: scala.quoted.Quotes) ?=> a).apply(using q)
tests/run-staging/multi-staging.check:  scala.quoted.runtime.Expr.quote[scala.Int](1.+(scala.quoted.runtime.Expr.splice[scala.Int](((contextual$5: scala.quoted.Quotes) ?=> scala.quoted.Expr.apply[scala.Int](x1)(scala.quoted.ToExpr.IntToExpr[scala.Int])(contextual$5))))).apply(using q1)
compiler/src/dotty/tools/dotc/core/NameKinds.scala:   *      val x: A ?=> B = (contextual$1: A) ?=> b
compiler/src/dotty/tools/dotc/core/NameKinds.scala:  val ContextFunctionParamName: UniqueNameKind = new UniqueNameKind("contextual$")
./dotty ((3.3.2))$

./dotty ((3.3.2))$ grep -ir 'ContextFunctionParamName'
compiler/src/dotty/tools/dotc/semanticdb/Scala3.scala:        && !name.is(NameKinds.ContextFunctionParamName)
compiler/src/dotty/tools/dotc/ast/Desugar.scala:import NameKinds.{UniqueName, ContextBoundParamName, ContextFunctionParamName, DefaultGetterName, WildcardParamName}
compiler/src/dotty/tools/dotc/ast/Desugar.scala:    val params = makeImplicitParameters(formals, mods, mkParamName = () => ContextFunctionParamName.fresh())
compiler/src/dotty/tools/dotc/core/NameKinds.scala:  val ContextFunctionParamName: UniqueNameKind = new UniqueNameKind("contextual$")
./dotty ((3.3.2))$

But maybe I am wrong and there are other mechanism I am not aware of which might make things work.

[mkurz]

A workaround might be to make PathBindableMacros.scala emit evidence$ again instead of contextual$ somehow even when build with Scala 3.3.2.

[bishabosha]

So while these name prefixes such as contextual$ are not special in TASTy format, it seems that the compiler's own implementation of name reading is inflexible in receiving prefixes it doesn't know about, basically baking them into compatibility by proxy

[dwijnand]

  @sharable private val simpleNameKinds = util.HashMap[Int, ClassifiedNameKind]()
  @sharable private val qualifiedNameKinds = util.HashMap[Int, QualifiedNameKind]()
  @sharable private val numberedNameKinds = util.HashMap[Int, NumberedNameKind]()
  @sharable private val uniqueNameKinds = util.HashMap[String, UniqueNameKind]()

unique names is the outlier, in using any user-defined string as the key, rather than tag int defined in TastyFormat.NameTags.

[mkurz]

I can confirm Scala 3.3.3 fixes the problem. Thanks!

[mkurz]

• Scala 3.3.3 #12441
• [3.0.x] Scala 3.3.3 #12442
• [2.9.x] Scala 3.3.3 #12443