Bug: Formatters error in IL2CPP build

[robotink]

Getting a crash in a build (Editor works fine) when setting up serializer using Unity IL2CPP and pre-generated formatter:

ExecutionEngineException: Attempting to call method 'Ceras.Formatters.CollectionFormatter`2[[System.Collections.Generic.List`1[[System.UInt32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]::.ctor' for which no ahead of time (AOT) code was generated.
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters, System.Exception& exc) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Object[] args) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Resolvers.CollectionFormatterResolver.GetFormatter (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.GetSpecificFormatter (System.Type type, Ceras.TypeMetaData meta) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.GetSpecificFormatter (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].GetOrCreateEntry (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.GeneratedFormatters.HMDDataFormatter.Serialize (System.Byte[]& buffer, System.Int32& offset, Ceras.GeneratedFormatters.HMDData value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.SerializeDelegate`1[T].Invoke (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.Serialize[T] (T obj, System.Byte[]& buffer, System.Int32 offset) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at TestCeras.Update () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Object[] args) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Resolvers.CollectionFormatterResolver.GetFormatter (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.GetSpecificFormatter (System.Type type, Ceras.TypeMetaData meta) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.GetSpecificFormatter (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].GetOrCreateEntry (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.GeneratedFormatters.HMDDataFormatter.Serialize (System.Byte[]& buffer, System.Int32& offset, Ceras.GeneratedFormatters.HMDData value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.SerializeDelegate`1[T].Invoke (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.Serialize[T] (T obj, System.Byte[]& buffer, System.Int32 offset) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasSerializer.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at TestCeras.Update () [0x00000] in <00000000000000000000000000000000>:0 

my serializer config:

var config = new SerializerConfig();
config.Advanced.AotMode = AotMode.Enabled;
GeneratedFormatters.UseFormatters(config);
//this line crashes in IL2CPP build - CerasUnityFormatters.ApplyToConfig(config); // Formatters from the UnityExtension for Vector3 etc
ceras = new CerasSerializer(config);

My data class:

namespace Ceras.GeneratedFormatters
{

    [GenerateFormatter]
    public class HMDData
    {
        //public ulong[] compressedPositions;
        public ulong compressedPosition;
        public uint compressedRotation;
        public ulong compressedPosition2;
        public uint compressedRotation2;
        public List<ulong> positions;
        public List<uint> rotations;
    }
}       

using generic List<> because auto-generator hangs on normal Array[]

Let me know if i can provide any more info.

Platform
Unity IL2CPP (UWP app). Unity 2019.2.7f

[rikimaru0345]

Hi,
it seems like we can probably fix all this 😄

Let's go through everything step by step:

1.) Aot Code

The message ...for which no ahead of time (AOT) code was generated. just means that the IL2CPP compiler didn't think it needed to generate any code for this method.

• Make sure your project uses the link file for unity:
  https://github.com/rikimaru0345/Ceras/blob/master/src/Ceras.UnityAddon/link.xml
• Also check out the official unity documentation on link files IL2CPP-BytecodeStripping

To actually tell the IL2CPP compiler that you are going to use a generic instantiation of a class you just have to use it somewhere.
In your specific case that would be this class:

Ceras/src/Ceras/Formatters/CollectionFormatter.cs

Line 269 in 454063a

public class CollectionFormatter<TCollection, TItem> : IFormatter<TCollection>

And you are (indirectly!) using it by having a field like this: public List<ulong> positions;

To directly use it, you can write a "mock usage" anywhere in your code where the analyzer will be able to find it:

var unused = new CerasSerializer();
new CollectionFormatter<List<ulong>, ulong>(unused); // for:  "public List<ulong> positions;"
new CollectionFormatter<List<uint>, uint>(unused); // for:  "public List<uint> rotations;"

What you could do is simply calling that code once from a Start() function, so the il2cpp static analyzer will see the call, follow it, see your usages of CollectionFormatter<...> and then descend into those calls as well.

As far as I know Unity requires you to write out an example use like that for every combination of (generic type) + (value type). If you are using reference-types, you don't have to specify anything multiple times.

Example where you must write out both:
new ArrayFormatter<float>(...); and new ArrayFormatter<DateTime>(...);
Both float and DateTime are value types, so their implementations can't be shared (they are a different size).

As for reference types, you don't need to list all the different ones:
new ArrayFormatter<object>(...);
It doesn't matter if the actual instantiation later on gets done with <object>, <Person>, <MyLoginPacket>, or whatever, because they're all classes/reference types, and their 'size' is the pointer size of the platform (4bytes, or 8bytes on x64 platforms).

2.) Crash in ApplyToConfig

You mentioned here that you're getting another crash:
//this line crashes in IL2CPP build - CerasUnityFormatters.ApplyToConfig(config); // Formatters from the UnityExtension for Vector3 etc
Is that related to the first problem?
Or is that an entirely different problem?

3.) Formatter generator hangs

Can you tell me more about that part?
using generic List<> because auto-generator hangs on normal Array[]

Do you have a call-stack of that? Where does it hang?
Can you make a screenshot of the thread window / expanded stack?
Pause the process with visual studio, then open the thread list, and click/expand the stack of the thread that's hanging in the formatter generator.

[robotink]

thanks for the help!

when i add the lines you suggested, creating a dummy serializer and CollectionFormatters, I get a nullref when deserializing in the Unity editor:

NullReferenceException: Object reference not set to an instance of an object
Ceras.Formatters.CollectionFormatter`2[TCollection,TItem].Deserialize (System.Byte[] buffer, System.Int32& offset, TCollection& value) (at Assets/Plugins/Ceras/Ceras/Formatters/CollectionFormatter.cs:339)
Ceras.Formatters.ReferenceFormatter`1[T].Deserialize (System.Byte[] buffer, System.Int32& offset, T& value) (at Assets/Plugins/Ceras/Ceras/Formatters/ReferenceFormatter.cs:288)
Ceras.GeneratedFormatters.HMDDataFormatter.Deserialize (System.Byte[] buffer, System.Int32& offset, Ceras.GeneratedFormatters.HMDData& value) (at Assets/Emerge/Core/Shared/NetworkMessaging/Serializer/CerasAutoGenerated/CerasAotFormattersGenerated.cs:34)
Ceras.Formatters.ReferenceFormatter`1[T].Deserialize (System.Byte[] buffer, System.Int32& offset, T& value) (at Assets/Plugins/Ceras/Ceras/Formatters/ReferenceFormatter.cs:288)
Ceras.CerasSerializer.Deserialize[T] (T& value, System.Byte[] buffer, System.Int32& offset, System.Int32 expectedReadLength) (at Assets/Plugins/Ceras/Ceras/CerasSerializer.cs:528)
Ceras.CerasSerializer.Deserialize[T] (System.Byte[] buffer) (at Assets/Plugins/Ceras/Ceras/CerasSerializer.cs:482)
TestCeras.Update () (at Assets/_Testing/TestCeras.cs:86)

2. Is an entirely different problem. The error has something to do with KnownTypes and a Vector2 which i am not using - i assume those are the standard Unity types its trying to setup? here is the error, again in a build only, works fine in editor:

ExecutionEngineException: Attempting to call method 'Ceras.TypeConfig`1[[UnityEngine.Vector2, UnityEngine.CoreModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]]::.ctor' for which no ahead of time (AOT) code was generated.
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters, System.Exception& exc) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.SerializerConfig.GetTypeConfigForConfiguration (System.Type type, System.Boolean isStatic) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.SerializerConfig.ConfigType (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasUnityFormatters.ApplyToConfig (Ceras.SerializerConfig config) [0x00000] in <00000000000000000000000000000000>:0 
  at TestCeras.Awake () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.SerializerConfig.GetTypeConfigForConfiguration (System.Type type, System.Boolean isStatic) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.SerializerConfig.ConfigType (System.Type type) [0x00000] in <00000000000000000000000000000000>:0 
  at Ceras.CerasUnityFormatters.ApplyToConfig (Ceras.SerializerConfig config) [0x00000] in <00000000000000000000000000000000>:0 
  at TestCeras.Awake () [0x00000] in <00000000000000000000000000000000>:0 

3. This is issue Bug: AOT autogenerator hangs Unity editor + other issues #66 I posted earlier. The hang happens in the Unity editor, as far as i know I can't profile the editor that deeply, so i'm not sure how to get more info on that.
   edit: I'm using the editor script to run the AOT generator in Unity, I guess i could run it from visual studio somehow?