Scala 2.11 / 2.12 classfile parser crashes on dotty-compiler_0.1-0.1.2-RC1.jar

[lrytz]

$ scala11 -version
Scala code runner version 2.11.11 -- Copyright 2002-2017, LAMP/EPFL
$ scala11 -cp /Users/luc/.ivy2/cache/ch.epfl.lamp/dotty-compiler_0.1/jars/dotty-compiler_0.1-0.1.2-RC1.jar
Welcome to Scala 2.11.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> error: error while loading package, class file '/Users/luc/.ivy2/cache/ch.epfl.lamp/dotty-compiler_0.1/jars/dotty-compiler_0.1-0.1.2-RC1.jar(scala/reflect/internal/pickling/package.class)' is broken
(class java.lang.ArrayIndexOutOfBoundsException/978)

It's either a bug in Scalac's classfile parser on in dotty's code gen.

[DarkDimius]

My guess: Dotty doesn't emit generic signatures.
Does scalac in any place assume they are there?

[lrytz]

Without checking, I don't think the classfile parser can assume presence of any signatures, as they are optional in the classfile format.

[smarter]

@lrytz Have you had a look at this? I'm tempted to close this here and move this issue to scala/bug

[retronym]

Dotty is emitting a zero-length ScalaSig classfile attribute.

⚡ javap -v -cp /Users/jz/Downloads/dotty-0.1.2-RC1-bin-SNAPSHOT/lib/dotty-compiler_0.1-0.1.2-RC1-bin-SNAPSHOT.jar scala/reflect/internal/pickling/package | cat -b
     1	Classfile jar:file:/Users/jz/Downloads/dotty-0.1.2-RC1-bin-SNAPSHOT/lib/dotty-compiler_0.1-0.1.2-RC1-bin-SNAPSHOT.jar!/scala/reflect/internal/pickling/package.class
     2	  Last modified 31/05/2017; size 969 bytes
     3	  MD5 checksum aea8697b90db08e868a09b6874c1949b
     4	  Compiled from "package.scala"
     5	public final class scala.reflect.internal.pickling.package
     6	  minor version: 0
     7	  major version: 52
     8	  flags: ACC_PUBLIC, ACC_FINAL, ACC_SUPER
     9	Constant pool:
    10	   #1 = Utf8               scala/reflect/internal/pickling/package
    11	   #2 = Class              #1             // scala/reflect/internal/pickling/package
    12	   #3 = Utf8               java/lang/Object
    13	   #4 = Class              #3             // java/lang/Object
    14	   #5 = Utf8               package.scala
    15	   #6 = Utf8               PickleFormat
    16	   #7 = Utf8               ()Ldotty/tools/dotc/core/unpickleScala2/PickleFormat$;
    17	   #8 = Utf8               scala/reflect/internal/pickling/package$
    18	   #9 = Class              #8             // scala/reflect/internal/pickling/package$
    19	  #10 = Utf8               MODULE$
    20	  #11 = Utf8               Lscala/reflect/internal/pickling/package$;
    21	  #12 = NameAndType        #10:#11        // MODULE$:Lscala/reflect/internal/pickling/package$;
    22	  #13 = Fieldref           #9.#12         // scala/reflect/internal/pickling/package$.MODULE$:Lscala/reflect/internal/pickling/package$;
    23	  #14 = NameAndType        #6:#7          // PickleFormat:()Ldotty/tools/dotc/core/unpickleScala2/PickleFormat$;
    24	  #15 = Methodref          #9.#14         // scala/reflect/internal/pickling/package$.PickleFormat:()Ldotty/tools/dotc/core/unpickleScala2/PickleFormat$;
    25	  #16 = Utf8               Code
    26	  #17 = Utf8               SourceFile
    27	  #18 = Utf8               TASTY
    28	  #19 = Utf8               ScalaSig
    29	{
    30	  public static dotty.tools.dotc.core.unpickleScala2.PickleFormat$ PickleFormat();
    31	    descriptor: ()Ldotty/tools/dotc/core/unpickleScala2/PickleFormat$;
    32	    flags: ACC_PUBLIC, ACC_STATIC
    33	    Code:
    34	      stack=1, locals=0, args_size=0
    35	         0: getstatic     #13                 // Field scala/reflect/internal/pickling/package$.MODULE$:Lscala/reflect/internal/pickling/package$;
    36	         3: invokevirtual #15                 // Method scala/reflect/internal/pickling/package$.PickleFormat:()Ldotty/tools/dotc/core/unpickleScala2/PickleFormat$;
    37	         6: areturn
    38	}
    39	SourceFile: "package.scala"
    40	Error: unknown attribute
    41	  TASTY: length = 0x241
    42	   5C A1 AB 1F 80 85 00 00 00 00 00 00 00 00 00 00
    43	   00 00 03 84 51 90 02 F6 01 84 41 53 54 73 01 85
    44	   73 63 61 6C 61 01 87 72 65 66 6C 65 63 74 02 82
    45	   81 82 01 88 69 6E 74 65 72 6E 61 6C 02 82 83 84
    46	   01 88 70 69 63 6B 6C 69 6E 67 02 82 85 86 01 87
    47	   70 61 63 6B 61 67 65 28 81 88 01 86 3C 69 6E 69
    48	   74 3E 02 82 87 88 28 81 8B 3F 82 8A 8C 01 84 6A
    49	   61 76 61 01 84 6C 61 6E 67 02 82 8E 8F 01 86 4F
    50	   62 6A 65 63 74 02 82 90 91 3F 82 8A 92 01 81 5F
    51	   01 80 3F 82 88 95 01 84 55 6E 69 74 01 8C 50 69
    52	   63 6B 6C 65 42 75 66 66 65 72 01 8E 75 6E 70 69
    53	   63 6B 6C 65 53 63 61 6C 61 32 01 84 63 6F 72 65
    54	   01 84 64 6F 74 63 01 85 74 6F 6F 6C 73 01 85 64
    55	   6F 74 74 79 01 8C 50 69 63 6B 6C 65 46 6F 72 6D
    56	   61 74 3F 82 9E 95 02 82 9D 9C 02 82 A0 9B 02 82
    57	   A1 9A 02 82 A2 99 01 8A 53 6F 75 72 63 65 46 69
    58	   6C 65 01 8A 61 6E 6E 6F 74 61 74 69 6F 6E 02 82
    59	   81 A5 02 82 A6 84 02 82 A7 A4 01 86 53 74 72 69
    60	   6E 67 02 82 90 A9 3F 83 8A A8 AA 01 D6 2F 55 73
    61	   65 72 73 2F 64 61 72 6B 2F 77 6F 72 6B 73 70 61
    62	   63 65 2F 64 6F 74 74 79 2F 63 6F 6D 70 69 6C 65
    63	   72 2F 73 72 63 2F 73 63 61 6C 61 2F 72 65 66 6C
    64	   65 63 74 2F 69 6E 74 65 72 6E 61 6C 2F 70 69 63
    65	   6B 6C 69 6E 67 2F 70 61 63 6B 61 67 65 2E 73 63
    66	   61 6C 61 01 89 50 6F 73 69 74 69 6F 6E 73 80 EA
    67	   80 E8 43 85 80 E4 43 87 81 90 88 71 89 77 89 43
    68	   87 89 85 72 8D 66 40 8B 12 1F 83 CE 89 9C BB 89
    69	   87 72 93 66 77 91 43 90 78 94 6B 75 96 40 8F 82
    70	   87 8A 86 80 77 97 43 81 83 8D 98 73 98 72 99 72
    71	   9A 72 9B 72 9C 43 9D 81 91 9E 75 9F 43 A3 72 9E
    72	   72 99 72 9A 72 9B 72 9C 40 C5 12 AF 8D 77 A4 43
    73	   A7 89 87 72 AB 66 40 DD 4D AC AD C5 82 01 CA A4
    74	   A0 A2 7E D6 98 80 00 FB 01 AA 8F 9E 9C FE 01 92
    75	   7E F4 A8 7E FE F3 FA 8A F9 BE 8D 8D 00 EB 00 C5
    76	   85 9C 94 92 F3 92 F1 92 FB 92 FB 92 FA 97 B4 00
    77	   F3 84 9E 93 CF A2 B1 92 F3 92 F1 92 FB 92 FB 92
    78	   FA
    79	Error: unknown attribute
    80	  ScalaSig: length = 0x0

Contrast with a scalac emitted file, in which that attribute has length 0x3.

⚡ javap -v -cp ~/scala/2.11/lib/scala-reflect.jar scala/reflect/api/package | cat -v
Classfile jar:file:/Users/jz/scala/2.11/lib/scala-reflect.jar!/scala/reflect/api/package.class
  Last modified 04/03/2016; size 627 bytes
  MD5 checksum 8ee2315087ae07cf8bb467d8e0b015a0
  Compiled from "package.scala"
public final class scala.reflect.api.package
  minor version: 0
  major version: 50
  flags: ACC_PUBLIC, ACC_FINAL, ACC_SUPER
Constant pool:
   #1 = Utf8               scala/reflect/api/package
   #2 = Class              #1             // scala/reflect/api/package
   #3 = Utf8               java/lang/Object
   #4 = Class              #3             // java/lang/Object
   #5 = Utf8               package.scala
   #6 = Utf8               Lscala/reflect/ScalaSignature;
   #7 = Utf8               bytes
   #8 = Utf8               ^F^Ai:Q!^A^B\t^B%\tq^A]1dW^F<WM^C^B^D\t^E^Y^Q\r]5^K^E^U1^Qa^B:fM2,7\r^^^F^B^O^E)1oY1mC^N^A^AC^A^F^L^[^E^Qa!^B^G^C^Q^Ci!a^B9bG.^Lw-Z\n^C^W9^A\"a^D\t^N^C^YI!!^E^D^C\r^Es^WPU3g^Q^U^Y2^B\"^A^U^C^Ya^TN\5u}Q\t^Q^BC^D^W^W\t%\tAB^L^B-5^LG/^Z:jC2L'0Z,fC.$^V^P]3UC^^,\"^A^G^T^U^Eea^Bc^A^N\"K9^Q1^D^H^G^A^Q^UiR^C1^A^_^C^E)^HC^A^F ^S\t^A#A^A^EV]&4XM]:f^S\t^Q3EA^FXK^F\G+^_9f)^F<^WB^A^S^C^E!!^V^P]3UC^^^\^HCA^N'\t^U9SC1^A)^E^E!^VCA^U-!\ty!&^C^B,\r\t9aj^\;iS:<^GCA^H.^S\tqcAA^BB]fDq^AM^F^C\n^C1^Q'^A\nnCR,'/[1mSj,G+^_9f)^F<WC^A^Z:)\t^YT^GE^B5mar!aG^[\t^Kuy^C^Y^A^P\n^E]^Z#a^B+za^V$^Vm^Z\t^C7e\"QaJ^XC^B!^B
   #9 = Utf8               SourceFile
  #10 = Utf8               RuntimeVisibleAnnotations
  #11 = Utf8               ScalaSig
{
}
SourceFile: "package.scala"
RuntimeVisibleAnnotations:
  0: #6(#7=s#8)
Error: unknown attribute
  ScalaSig: length = 0x3
   05 00 00

This is still treated as an "empty ScalaSig attribute":

      val attrName = readTypeName()
      val attrLen = u4
      attrName match {
        case tpnme.ScalaSignatureATTR =>
          isScala = true
          val pbuf = new PickleBuffer(in.buf, in.bp, in.bp + attrLen)
          pbuf.readNat(); pbuf.readNat()
          if (pbuf.readNat == 0) // a scala signature attribute with no entries means that the actual scala signature
            isScalaAnnot = true    // is in a ScalaSignature annotation.
          in.skip(attrLen)

Which implies that the actual Scala2 pickle should be found in the @ScalaSignature annotation on the classfile. This also appears to be absent from the dotty generated class, which is either a bug or an yet-to-be-implemented feature.

[retronym]

Some history: Scala pickles used to be stored in classfile attributes, but were moved to annotations for better interop with, e.g, runtime reflection. The ScalaSig attribute is now only really used as a marker to say: this was compiled by scalac, and has its type info stored in the annotation. We also have the Scala annotation, which is used for inner classes, anon classes, module classes, trait impl classes, etc. It says: this is compiled by scalac, and its type information is in the top-level enclosing / companion class.

[retronym]

The other question, is what are these package objects doing in dotty-compiler.jar in the first place?

⚡ jar tf /Users/jz/Downloads/dotty-0.1.2-RC1-bin-SNAPSHOT/lib/dotty-compiler_0.1-0.1.2-RC1-bin-SNAPSHOT.jar | grep 'reflect/internal'
scala/reflect/internal/
scala/reflect/internal/pickling/
scala/reflect/internal/util/
scala/reflect/internal/pickling/package$.class
scala/reflect/internal/util/package.hasTasty
scala/reflect/internal/util/package.class
scala/reflect/internal/pickling/package.hasTasty
scala/reflect/internal/pickling/package.class
scala/reflect/internal/util/package$.class

[retronym]

I see that the absence of the @ScalaSignature annotation will actually emit the "foreign pickle marker"

// BCodeSkelBuilder.initJClass
      val ssa = getAnnotPickle(thisName, claszSymbol)
      cnode.visitAttribute(if (ssa.isDefined) pickleMarkerLocal else pickleMarkerForeign)

// BCodeHelpers 2.11-dotty
    def pickleMarkerForeign = {
      createJAttribute(ScalaSignatureATTRName, new Array[Byte](0), 0, 0)
    }

This is a bug in BCode, the corresponding spot it was copy/pasted from in GenASM:

// GenASM 2.11
    def pickleMarkerForeign = {
      createJAttribute(tpnme.ScalaATTR.toString, new Array[Byte](0), 0, 0)
    }

[retronym]

I haven't untangled the history of this, but the code in 2.12 looks right, so I'd suggest an upgrade of your version of the compiler backend..

[retronym]

Looks like it is a bug introduced in lampepfl/scala@3d66b0d#diff-7a3aa9d1c9acd44da15dc536d4760e5aR179

[smarter]

@retronym Thanks for the investigation!

The other question, is what are these package objects doing in dotty-compiler.jar in the first place?

We try to keep our backend compiling with both scalac and dotty, to do that we introduced some dummy package objects that forward to real implementations in dotty with other names: https://github.com/lampepfl/dotty/tree/master/compiler/src/scala

[smarter]

Fix in lampepfl/scala#21, thanks again for looking into it @retronym !