AOT + System.Reflection.Emit + ConstructorInfo doesn't work as expected

[jonpryor]

Context: dotnet/android#3753
Context: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=3203657

Small repro to help investigate a regression we're seeing when trying
to use mono/2019-10 with Xamarin.Android.

Background: Constructors! (Am I right?!)

Xamarin.Android "binds" Java code so that it can be used as if it were
C# code. Part of this binding is involves "peer types": in order for
things to work as expected, when a C# type inherits a Java type, a
"Java Callable Wrapper" (JCW) is constructed for the C# type. This
JCW is a "peer type": all C# subclases of Java.Lang.Object hold a
handle to a corresponding "Java peer", which is stored in the
Object.handle field.

C# types can be created from Java by invoking the Java constructor
on the Java Callable Wrapper.

This is where things get weird: in a "Java started the constructor"
scenario, we have a handle to an existing Object instance which we
want to wrap with a new C# instance. However, we don't want all C#
subclasses to (manually!) write some MyType(IntPtr) constructor, so
how do we invoke the C# constructor when we already have the Java
instance?

By trickery, that's how: instead of directly calling the constructor
of the C# type, we instead create a new delegate -- using
DynamicMethod -- which:

1. Creates a new uninitialized instance of the C# type by using
   FormatterServices.GetUninitializedObject()

2. Set the Object.handle field on the uninitialized instance.

3. Call the C# constructor, using the uninitialized instance, to
   initialize the instance.

This way the C# type doesn't need to know or explicitly deal with JNI
handles. It Just Works.

This system has worked for as long as Mono for Android has existed.

(Aside: what's "funny" is that while this works with current
Xamarin.Android v10.1 and is only failing with mono/2019-10, this
desktop repro fails in the same way with 6.4.0 & mono/2019-06!)

...until mono/2019-10, when AOT is used.

Thus, this repro project:

• CoreLib.dll is analogous to Mono.Android.dll, and contains the
  ConstructorBuilder.CreateDelegate() type which does the
  "create an uninitialized instance, set the handle, call the ctor"
  dance. It also contains BaseClass, which is analogous to
  Java.Lang.Object, and contains the handle field.

• UserLib.dll, which would be analogous to a Xamarin.Android
  App.dll assembly, with e.g. a custom Activity subclass.

• AotSreCtorInfo.exe, the app which drives things.

Instructions

Build the project:

    make all

Run the project:

    make run

With normal JIT use, it works as expected:

    $ make run
    mono AotSreCtorInfo/bin/Debug/AotSreCtorInfo.exe
    # jonp: Inside activation constructor for type `UserLib.MyClass`: before invoke!!
    BaseClass..ctor(IntPtr): this.handle=0x2a; handle=0x00
    BaseClass..ctor(IntPtr): ACTIVATION SCENARIO
    BaseClass.SetHandle: handle=0x2a
    BaseClass..ctor()
    MyClass..ctor invoked
    # jonp: Inside activation constructor for type `UserLib.MyClass`: after invoke!!

In particular, note the ACTIVATION SCENARIO message, and that
this.handle=0x2a in the preceeding message.

AOT the assemblies with:

    make aot

This runs mono --aot on each assembly.

make run after make aot will use the AOT'd assemblies.

This is where things to off the rails:

    $ make run
    mono AotSreCtorInfo/bin/Debug/AotSreCtorInfo.exe
    # jonp: Inside activation constructor for type `UserLib.MyClass`: before invoke!!
    BaseClass..ctor(IntPtr): this.handle=0x00; handle=0x00
    BaseClass.SetHandle: handle=0x00
    BaseClass..ctor()
    MyClass..ctor invoked
    # jonp: Inside activation constructor for type `UserLib.MyClass`: after invoke!!

Note that this.handle=0x00, and NO ACTIVATION SCENARIO message is
printed.

This is a problem. Enabling AOT breaks expected app semantics.

Steps to Reproduce

1. Download and Unzip: AotSreCtorInfo.zip

2. Build it:

    make
   

3. Run it:

    make run
   

   This is normal & expected JIT behavior & output. Record it. (Or see above.)

4. Build for AOT

    make aot
   

5. Re-run using AOT:

    make run
   

   This is expected to be identical to (3). It isn't.

Current Behavior

When AOT is used, BaseClass.handle is not set to 42 before invoking the MyClass constructor.

When AOT is not used, BaseClass.handle is set to 42 before the MyClass constructor is invoked.

Expected Behavior

AOT behavior should match non-AOT behavior: BaseClass.handle should be set to 42 before the MyClass constructor is executed.

On which platforms did you notice this

• macOS
• Linux
• Windows

Version Used:

Mono JIT compiler version 6.4.0.208 (2019-06/07c23f2ca43 Wed Oct  2 04:52:23 EDT 2019)

[jonpryor]

After this point I'm in way over my head. That said, I found this "interesting".

Get "diagnostic" outputs, so we can see what the JIT is doing:

$ make clean all
$ mono -v -v -v -v -v -v -v -v -v -v -v -v AotSreCtorInfo/bin/Debug/AotSreCtorInfo.exe > jit.txt
$ make aot
$ mono -v -v -v -v -v -v -v -v -v -v -v -v AotSreCtorInfo/bin/Debug/AotSreCtorInfo.exe > aot.txt

Within jit.txt and aot.txt, search for UserLib.MyClass:1.

They both start with the same IL, which is a good sanity check:

converting method (wrapper dynamic-method) void CoreLib.ConstructorBuilder:#ctor UserLib.MyClass:1 (intptr,object[])
creating vars
created temp 0 (R16) of type System.IntPtr
created temp 1 (R17) of type System.Object
        arg [0]:  arg R16 <-
        arg [1]:  arg R17 <-
creating locals
        local [0]: created temp 2 (R18) of type System.Object
locals done
method to IR (wrapper dynamic-method) CoreLib.ConstructorBuilder:#ctor UserLib.MyClass:1 (intptr,object[])
converting (in B2: stack: 0) IL_0000: ldtoken   0x00000001
converting (in B2: stack: 1) IL_000a: call      0x00000005
converting (in B2: stack: 1) IL_000f: stloc.0   
converting (in B2: stack: 0) IL_0010: ldloc.0   
converting (in B2: stack: 1) IL_0011: ldarg.0   
converting (in B2: stack: 2) IL_0012: stfld     0x00000007
converting (in B2: stack: 0) IL_0017: ldloc.0   
converting (in B2: stack: 1) IL_0018: ldstr     
converting (in B2: stack: 2) IL_001d: call      0x0000000b
converting (in B2: stack: 1) IL_0022: call      0x0000000d
converting (in B2: stack: 0) IL_0027: ldstr     
converting (in B2: stack: 1) IL_002c: call      0x00000011
converting (in B2: stack: 0) IL_0031: ret       

Things get "weird" -- to me -- when we hit the AFTER METHOD-TO-IR 2 section.

JIT:

AFTER METHOD-TO-IR 2: [IN:  BB3(0), OUT:  BB1(0) ]
 il_seq_point il: 0x0
 i8const R19 <- [4467934544]
 move R21 <- R19
 call R20 <- [object System.Runtime.Serialization.FormatterServices:GetUninitializedObject (System.Type)] [%rdi <- R21] clobbers: c
 il_seq_point il: 0xf, nonempty-stack
 move R18 <- R20
 il_seq_point il: 0x10
 move R22 <- R18
 move R23 <- R16
 not_null R22
 store_membase_reg [R22 + 0x10] <- R23
 il_seq_point il: 0x17
 move R24 <- R18
 i8const R25 <- [4467687728]
 move R26 <- R25
 voidcall [void System.Console:WriteLine (string)] [%rdi <- R26] clobbers: c
 il_seq_point il: 0x22, nonempty-stack
 move R27 <- R24
 voidcall [void UserLib.MyClass:.ctor ()] [%rdi <- R27] clobbers: c
 il_seq_point il: 0x27, nonempty-stack
 il_seq_point il: 0x27
 i8const R28 <- [4467687928]
 move R29 <- R28
 voidcall [void System.Console:WriteLine (string)] [%rdi <- R29] clobbers: c
 il_seq_point il: 0x31, nonempty-stack
 il_seq_point il: 0x31
 br [B1]

AOT:

AFTER METHOD-TO-IR 2: [IN:  BB3(0), OUT:  BB1(0) ]
 il_seq_point il: 0x0
 i8const R19 <- [4392207696]
 move R21 <- R19
 call R20 <- [object System.Runtime.Serialization.FormatterServices:GetUninitializedObject (System.Type)] [%rdi <- R21] clobbers: c
 il_seq_point il: 0xf, nonempty-stack
 move R18 <- R20
 il_seq_point il: 0x10
 move R22 <- R18
 move R23 <- R16
 not_null R22
 store_membase_reg [R22] <- R23
 il_seq_point il: 0x17
 move R24 <- R18
 i8const R25 <- [4391960880]
 move R26 <- R25
 voidcall [void System.Console:WriteLine (string)] [%rdi <- R26] clobbers: c
 il_seq_point il: 0x22, nonempty-stack
 move R27 <- R24
 voidcall [void UserLib.MyClass:.ctor ()] [%rdi <- R27] clobbers: c
 il_seq_point il: 0x27, nonempty-stack
 il_seq_point il: 0x27
 i8const R28 <- [4391961080]
 move R29 <- R28
 voidcall [void System.Console:WriteLine (string)] [%rdi <- R29] clobbers: c
 il_seq_point il: 0x31, nonempty-stack
 il_seq_point il: 0x31
 br [B1]

In particular, store_membase_reg differs:

• JIT: store_membase_reg [R22 + 0x10] <- R23
• AOT: store_membase_reg [R22] <- R23

Is the lack of + 0x10 in AOT output a problem? ¯_(ツ)_/¯

I just found it "interesting".

[vargaz]

So the original problem is fixed, but another problem is still there.
What seems to happen is that a dynamic method calls Xamarin.Forms.Performance.Integration.Droid.MainActivity..ctor () which makes a chained call to its base ctors etc. but somehow the lowermost ctor code in Java.Lang.Object:.ctor () is not executed.
Based on some testing, MainActivity/FormsAppCompatActivity/AppCompatActivity ctors are called, but
FragmentActivity:.ctor () is not called.

[vargaz]

Compiling with /p:AndroidLinkMode=None fixes the problem.

[vargaz]

ls -l tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/Mono.Android.dll 
-rw-rw-rw-  1 vargaz  staff  1643008 Nov 17 19:57 tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/Mono.Android.dll
ikdasm tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/Mono.Android.dll  | grep MVID
// MVID: {12600488-17B5-414C-9748-1E16C94F6621}
ls -l tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/shrunk/Mono.Android.dll 
-rw-rw-rw-  1 vargaz  staff  1459712 Nov 17 19:57 tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/shrunk/Mono.Android.dll
ikdasm tests//Xamarin.Forms-Performance-Integration/Droid/obj/Release/android/assets/shrunk/Mono.Android.dll  | grep MVID
// MVID: {12600488-17B5-414C-9748-1E16C94F6621}

So we have two different sized assemblies with the same MVID, some assemblies are AOTed against the first version, some are AOTed against the second version, which probably causes the problem. The AOT runtime can't detect this because the MVIDs are the same.

[vargaz]

There was a bug in cecil which caused MVID calculation to be incorrect, but it was fixed before 2019-10 was branched:
dotnet/linker#770
Maybe there is another problem somewhere.

[marek-safar]

@vargaz is this issue fixed on our side?

[marek-safar]

@vargaz ^ ?