Binding Kotlin itself crashes `generator --only-xml-adjuster`

[mattleibow]

I am trying to bind Kotlin so that it can be consumed by Kotlin-based libraries, and the generator is not happy.

This is the library kotlin-stdlib-1.3.41.jar

I just create a new binding project and build. Because at this point I don't want any actual types, I added <remove-node path="/api/package" /> to the transforms.

The generator crashes with this error:
https://gist.github.com/mattleibow/92dff72582ec972b214283d8fe739a9c

The api.xml.class-parse file is this:
https://gist.github.com/mattleibow/34af078b7aca818dd0c5381e075896d1

I can reproduce the crash at this point in the build:

mono /Library/Frameworks/Xamarin.Android.framework/Versions/10.0.0.4/lib/xamarin.android/xbuild/Xamarin/Android/generator.exe \
    api.xml.class-parse \
    --only-xml-adjuster \
    --xml-adjuster-output=api.xml \
    --csdir=src \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/mscorlib.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/Java.Interop.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v9.0/Mono.Android.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/mscorlib.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Core.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Net.Http.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/Facades/System.Runtime.dll \
    --ref=/Library/Frameworks/Mono.framework/External/xbuild-frameworks/MonoAndroid/v1.0/System.Xml.dll \
    --api-level=28 \
    --global \
    --public

[mattleibow]

More info can be found at: #464

[jonpryor]

I don't know what's going on. Time for some musings to collect later...

I thought that RuntimeVisibleAnnotations might be involved, so I added support for parsing RuntimeVisibleAnnotations and RuntimeInvisibleAnnotations blobs to class-parse: #467.

This might be overthinking it, though the commit message in #467 points out an important thought:

How/Why does kotlin/collections/AbstractCollection$toString$1''s invoke() method mention the type E, when class-parse-generated XML output doesn't mention E within a <typeParameters/> block?

<typeParameters/> in turn comes from the Signature annotation, which I should have looked at earlier. For comparison purposes, consider android.app.FragmentHostCallback. If we run class-parse --dump android/app/FragmentHostCallback.class, we get:

Attributes Count: 5
        0: Signature(<E:Ljava/lang/Object;>Landroid/app/FragmentContainer;)

The Signature annotation blob has a grammar for ClassSignature, which has FormalTypeParameters at the start of the string, between <...> brackets. This is what we see with android.app.FragmentHostCallback: a FormalTypeParameters specification of <E:Ljava/lang/Object;>, introducing the type parameter E which is constrained to java.lang.Object, followed by the signature of the super (base) class, android.app.FragmentContainer.

Now, what does kotlin/collections/AbstractCollection\$toString\$1.class have for it's Signature blob?

Attributes Count: 5
        0: InnerClasses(Count=1, InnerClass(InnerClass='kotlin/collections/AbstractCollection$toString$1', OuterClass='', InnerName='', InnerClassAccessFlags=Static, Final))
        1: Unknown[EnclosingMethod](4)
        2: Signature(Lkotlin/jvm/internal/Lambda;Lkotlin/jvm/functions/Function1<TE;Ljava/lang/CharSequence;>;)
        3: Unknown[SourceFile](2)
        4: RuntimeVisibleAnnotations(Annotation('Lkotlin/Metadata;', {mv: Array(Integer(1), Integer(1), Integer(15)), bv: Array(Integer(1), Integer(0), Integer(3)), k: Integer(3), d1: Array(Utf8("...")), d2: Array(Utf8("<anonymous>"), Utf8(""), Utf8("E"), Utf8("it"), Utf8("invoke"), Utf8("(Ljava/lang/Object;)Ljava/lang/CharSequence;"))}))

The Signature blob does not specify any type parameter information!

This leaves the question unanswered: where is E coming from? (I do see E in the RuntimeVisibleAnnotations for d2, but I don't want to begin to figure out what d1 and d2 are supposed to mean...)

Could the EnclosingMethod annotation be relevant?

[jonpryor]

Thus, it appears to be due to attempted processing of generic types and their constraints.

It would be useful to know which exact type & method causes things to break. @jpobst mentioned a signature of ([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object; on kotlin/collections/ArraysKt___ArraysKt.class, but those methods look reasonable:

      <method
        abstract="false"
        deprecated="not deprecated"
        final="true"
        name="reduce"
        native="false"
        return="S"
        jni-return="TS;"
        static="true"
        synchronized="false"
        visibility="public"
        bridge="false"
        synthetic="false"
        jni-signature="([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object;">
        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>
        <parameter
          name="$this$reduce"
          type="T[]"
          jni-type="[TT;" />
        <parameter
          name="operation"
          type="kotlin.jvm.functions.Function2&lt;? super S, ? super T, ? extends S&gt;"
          jni-type="Lkotlin/jvm/functions/Function2&lt;-TS;-TT;+TS;&gt;;" />
      </method>
      <method
        abstract="false"
        deprecated="not deprecated"
        final="true"
        name="reduceRight"
        native="false"
        return="S"
        jni-return="TS;"
        static="true"
        synchronized="false"
        visibility="public"
        bridge="false"
        synthetic="false"
        jni-signature="([Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)Ljava/lang/Object;">
        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>
        <parameter
          name="$this$reduceRight"
          type="T[]"
          jni-type="[TT;" />
        <parameter
          name="operation"
          type="kotlin.jvm.functions.Function2&lt;? super T, ? super S, ? extends S&gt;"
          jni-type="Lkotlin/jvm/functions/Function2&lt;-TT;-TS;+TS;&gt;;" />
      </method>

...although...perhaps it's having T have an interface bound on S which is causing the problem?

        <typeParameters>
          <typeParameter
            name="S"
            jni-classBound="Ljava/lang/Object;"
            classBound="java.lang.Object"
            interfaceBounds=""
            jni-interfaceBounds="" />
          <typeParameter
            name="T"
            jni-classBound=""
            classBound=""
            interfaceBounds="S"
            jni-interfaceBounds="TS;" />
        </typeParameters>

Which is, I believe, equivalent to:

<S, T implements S>
public static S reduceRight(T[] $this$reduceRight, Function2<? super T, ? super S, ? extends S> operation);

[jonpryor]

Revisiting #466 (comment)

How/Why does kotlin/collections/AbstractCollection$toString$1''s invoke() method mention the type E, when class-parse-generated XML output doesn't mention E within a <typeParameters/> block?

With PR #469 we now parse the EnclosingMethod attribute and export it from class-parse-generated XML. From it we see:

$ mono class-parse.exe kotlin/collections/AbstractCollection\$toString\$1.class
...
    <class
      abstract="false"
      deprecated="not deprecated"
      enclosing-method-jni-type="Lkotlin/collections/AbstractCollection;"
      enclosing-method-name="toString"
      enclosing-method-signature="()Ljava/lang/String;"
      jni-extends="Lkotlin/jvm/internal/Lambda;"
      extends="kotlin.jvm.internal.Lambda"
      extends-generic-aware="kotlin.jvm.internal.Lambda"
      final="true"
      name="AbstractCollection.toString.1"
      jni-signature="Lkotlin/collections/AbstractCollection$toString$1;"
      source-file-name="AbstractCollection.kt"
      static="true"
      visibility="">
...

The enclosing type is Lkotlin/collections/AbstractCollection;.

If we turn our attention to AbstractCollection, we see:

$ mono class-parse.exe kotlin/collections/AbstractCollection.class
...
    <class
      abstract="true"
      deprecated="not deprecated"
      jni-extends="Ljava/lang/Object;"
      extends="java.lang.Object"
      extends-generic-aware="java.lang.Object"
      final="false"
      name="AbstractCollection"
      jni-signature="Lkotlin/collections/AbstractCollection;"
      source-file-name="AbstractCollection.kt"
      static="false"
      visibility="public">
      <typeParameters>
        <typeParameter
          name="E"
          jni-classBound="Ljava/lang/Object;"
          classBound="java.lang.Object"
          interfaceBounds=""
          jni-interfaceBounds="" />

i.e. AbstractCollection does contain a declaration for the type parameter E, which is used/referenced by AbstractCollection$toString$1.

Thus, we can infer a possible fix: if a type has an enclosing-method-jni-type attribute, "lookup" the specified type (and optional method) to find any type parameters specified in the enclosing scope, and if necessary copy those <typeParameters/> into the resulting XML.

Alternatively, instead of having class-parse do that "copy <typeParameters/> to the referencing scope" logic, have generator perform that copy.

My concern over doing the copy at all is it alters the semantics of class-parse lib.jar vs. class-parse type.class: the former can perform "enclosing type" lookups; the latter can't. It may thus be preferable to have generator do the lookups, as it should have all necessary context.