Unity IL2CPP Guide outdated?

[Hurri04]

Hi again,

I've recently started implementing some new systems for my Unity game (custom sprite renderer/animator) and wanted to try building a first executable version. After receiving a blank screen in it I turned on "Development Build" and "Autoconnect Profiler" in the Build Settings and discovered a few exceptions being thrown related to Ceras.

So I started reading your guide for IL2CPP but I'm guessing some of the steps are a bit outdated?

In step 1 I found the setting in question under "config.Advanced.AotMode". So far so good.

However, in step 2 the [CerasAutoGenFormatter] attribute seems to have changed? If I add it to my Tile struct I get these errors:

Assets\Scripts\Level\Tile\Tile.cs(7,2): error CS0246: The type or namespace name 'CerasAutoGenFormatterAttribute' could not be found (are you missing a using directive or an assembly reference?)

Assets\Scripts\Level\Tile\Tile.cs(7,2): error CS0246: The type or namespace name 'CerasAutoGenFormatter' could not be found (are you missing a using directive or an assembly reference?)

Step 3 is a bit unclear: "Compile your assembly". Which assembly? Usually Unity compiles everything automatically so if there's anything extra needed in this case a more indepth step-by-step explanation would be helpful ;)
About the AotGenerator.exe: Is it included somewhere already or do I need to build this myself (is this what you mean by "Compile your assembly")?

In step 4 GeneratedFormatters.UseFormatters(config); doesn't seem to work. However, CerasUnityFormatters.ApplyToConfig(config); from here does work. Is this the new/renamed version of that method?
And is the Extension.cs file still needed? I think I remember you saying something about rebuilding some stuff to handle the Unity types automatically?

Including the link.xml from step 5 is easy enough.

Any help with this would be much appreciated!
Cheers :)

[rikimaru0345]

Good feedback, the guide is definitely in need of an update.
I'll see what I can do about that.

As for the attributes: are you using the newest version from the v5 branch? For all development for you own projects you should be using the master (v4) branch

[Hurri04]

I downloaded the packages from appveyor where it says 4.0.41. I've tried both the net4.5 and net4.7 versions.

I just saw that the compiled AotGenerator.exe is on appveyor as well (packaged with a bunch of dlls). Does this need to be placed in the same Assets/Plugins/Ceras/ folder? Or can it be placed anywhere? Would the link.xml prevent unused dlls from being stripped when placing it into the Plugin folder? At 28.5MB it would increase the build size quite a bit.

[rikimaru0345]

The aotgenerator only generates source code files (cs files).
You can place it in your unity project, if you want to... but it's not needed there, it can be placed anywhere.

[rcFMS]

Hey,

we're also (still) looking to integrate Ceras into our Unity project and I saw that you updated both wiki pages last month. The instructions on the "Usage with Unity" page seem to work now.

However, the "[CerasAutoGenFormatter]" attribute mentioned on the "Unity IL2CPP (iOS and AOT)" page still does not seem to exist, leading to the same problem OP described above.

Without it we get this error message when trying to serialize an instance of our MapTile class:

InvalidOperationException: No formatter for the Type 'MapTile' was found. Ceras is trying to fall back to the DynamicFormatter, but that formatter will never work in on AoT compiled platforms. Use the code generator tool to automatically generate a formatter for this type.

Without setting the "config.Advanced.AotMode = AotMode.Enabled;" from the first step the Unity Editor even crashes (without any error message) as soon as the program reaches the part where the "CerasSerializer.Serialize" method is called.

Since it may be relevant: As per the instructions on the first wiki page, we're on the latest commit 02c3dcc from the master branch.

[rikimaru0345]

This is the attribute you're looking for:
https://github.com/rikimaru0345/Ceras/blob/master/src/Ceras/Attributes/CerasAttributes.cs#L269

GenerateFormatterAttribute in the Ceras.Formatters.AotGenerator namespace.

I will fix the guide, thanks for the reminder!

edit: Guide was updated to mention the correct names to the attributes (they have been renamed recently)

[rikimaru0345]

Without setting the "config.Advanced.AotMode = AotMode.Enabled;" from the first step the Unity Editor even crashes (without any error message) as soon as the program reaches the part where the "CerasSerializer.Serialize" method is called.

That is strange. You should be getting an exception from the .NET runtime (well, the IL2CPP compiled parts of it...) telling you about not being able to create new/dynamic code at runtime. That could be related to the IL2CPP code stripper that removes unused code (at least what that tool perceives to be unused! 😉 )

I'll investigate that when I have some more time.
Let me know if the attribute(s) I linked to help and the AotGen works and if there are any other problems that I can help with 😄

[rcFMS]

GenerateFormatterAttribute in the Ceras.Formatters.AotGenerator namespace.

Thanks, this part seems to work now. It may be good to also mention the namespace on the wiki page though (as well as the "Ceras.GeneratedFormatters" namespace necessary for the "GeneratedFormatters.UseFormatters(config);" line in step 3) ;)

However, there appear to be some problems with the "SourceFormatterGenerator" class which generates the "CerasAotFormattersGenerated.cs" file:

1. The attributes it writes above the formatter classes are named "[GeneratedFormatterAttribute]" instead of "[GenerateFormatterAttribute]" (notice the extra "d" after "Generate"), leading to new errors.

2. Nested classes are referenced as "ClassA+ClassB" instead of "ClassA.ClassB", resulting in more errors. I fixed these by hand for now.

3. There are also some errors in the "Deserialize" methods where in the first lines of each block the type declaration for the temp variables was missing. I fixed this by writing "var" at the beginning of the line for now.

4. In the third lines of these same blocks the temp variables get assigned back to the properties from which they were created in the first lines. However (at least in my case) these properties only have public getters but either private setters or no setters at all, making it a read-only property. These accessors are meant to be set this way to ensure that the values of the backing fields of the properties can only be manipulated from within the class itself (to prevent accidentally or erroneously overwriting any values which are not meant to be set outside of certain methods).
   And since these are properties they also can't be passed by ref in into the "Deserialize".
   I could set the setters to public for now but I'd prefer not having to do so.
   Reflection would probably do the trick but iirc this would create some serious overhead, right?
   Are there any other workarounds?

[rikimaru0345]

Hey, thanks for the reports, I will take care of them when I get home today! :)

However (at least in my case) these properties only have public getters but either private setters or no setters at all, making it a read-only property. These accessors are meant to be set this way to ensure that the values of the backing fields of the properties can only be manipulated from within the class itself (to prevent accidentally or erroneously overwriting any values which are not meant to be set outside of certain methods).
And since these are properties they also can't be passed by ref in into the "Deserialize".

In your specific scenario - which is AOT - there is no way to do this, nor can there be any serializer that can solve this (as far as I'm aware).

Because hidden backing fields are an implementation detail, overwriting them is not possible in a reliable way.
Sure, one can make an demo that changes any prop, even if it doesn't have a setter, as long as it has a backing field (so computed props are excluded of course).

But for a serializer you need some guarantees. Stable and deterministic names, and/or order.
For those backing fields, that is not a given. Names and orders can (and actually will!) change in random ways.

I'll link an issue later that describes the issue with serializing lambda closures. The part about backing fields is closely related to the issue here.

My advice:

• maybe use a serialization constructor
• or you could use explicit backing fields (so they are not compiler generated, that's the important part). They can even be readonly then because ceras can overwrite readonly fields (will be much slower of course since one must use reflection for that).

I'll reply to the remaining things when I'm at home :)

[rcFMS]

or you could use explicit backing fields

Sorry for the confusion, this is what I meant; basically we have all member variables set to private and if another class needs access to them we expose them through properties which contain an explicit getter. We do not use any auto-properties.

[rikimaru0345]

I see.
Unfortunately I don't really see a way to do this.

Dynamically generated code has no access restrictions whatsoever for internal/private fields.
The thing is, in AoT ("Ahead-of-Time" compiled) mode we can't generate any dynamic code.

Since the AotGenerator generates C# source code, it has to obey the language rules :/
And normally there's no way to access private fields from the outside.

Well, there's reflection, but that's a special case.
It's probably(?) way too slow to be of any use, unless your use-case is really not performance sensitive at all (like for example saving/loading a settings file, save-game, ... ).

With reflection you're looking at a something like a ~100x slowdown to read/write a field/property plus the allocations that causes. For loading some application settings it won't matter, because the difference between 10 nanoseconds and even 1000 nanoseconds isn't perceivable by the user.

[rcFMS]

Would it maybe be possible to solve this through a constructor?

In our use case the objects we need to serialize / deserialize are rather large (because they contain various Lists and other Collections*) and have a lifetime of undefined length (so far they are needed through the entire time our tool runs and even when we add functionality in the future which would allow them to be replaced I'd much rather just delete them to free memory because of their size).

Because of this they are not really reusable anyway which is why we're already using the overload of the "Deserialize" method which does not take a ref object but constructs a new one instead.

*) We even have a few classes which just act as wrappers for some Lists or Collections which then get exposed through a property with an explicit getter; I suppose those could be turned into public readonly fields and be set by a constructor?

[rikimaru0345]

@rcFMS Hm, the construction methods are incredibly flexible. Chances are that something can be done that way.

I think it would help if we could come up a specific example.
You have something like this, right?

class BigObject 
{
   readonly int _someInt;
   public int SomeInt => _someInt;
}

And all of this is in AoT mode as well, right?

Take a look at this file, I marked one example here:
https://github.com/rikimaru0345/Ceras/blob/master/tests/Ceras.Test/Tests/ConstructionAndPooling.cs#L26-L28

There are other examples in the same file as well (just CTRL+F for ConstructBy).
In cases where the parameter-names can be matched with field-names Ceras should even do the mapping automatically.

Let me know if that example captures the main issues here. And also let me know if you think that any of the ConstructBy methods could be of any help. Maybe one of them works pretty well for you (or is getting close and only needs a few little changes) 😄

[rcFMS]

You have something like this, right?

Almost, we're using fully manual properties instead of expression-bodied members (since we're still on C# 6):

public int SomeInt
{
	get
	{
		return _someInt;
	}
}

And all of this is in AoT mode as well, right?

Yes, we're using the IL2CPP Scripting Backend (if that's the question) which is why I posted in this issue in the first place before we got a bit off-topic ;)

In cases where the parameter-names can be matched with field-names Ceras should even do the mapping automatically.

So if I create a constructor like below it will be used automatically? This would probably be the easiest way for me to do this then. I'm guessing if I'm not calling this constructor anywhere else I'll need to prevent code-stripping from removing it via a link.xml? I might give this a try if I have some time tomorrow.

public BigObject(int _someInt)
{
	this._someInt = someInt;
}

[rikimaru0345]

Yes that constructor should work, but I'm not sure if it will be used automatically.

You can either configure it manually like in the example,
Or you can use OnConfigNewType
That way you apply any setting by matching it against your filter.
For example you could check the namespace of a type, or it's attributes, or whether or not it has exactly one constructor with more than zero parameters.

Also, as long as ceras knows what function or constructor to use, the parameter matching is pretty lenient. It doesn't care about uppercase/lowercase. And you probably don't even have to add the _ either.

The aot code generator should write the exact same code the DynamicFormatter creates.

Let me know how it goes or if there is anything unclear :)

[rikimaru0345]

You shouldn't need to add anything to the link xml, as the constructor will be referenced by the generated code.

[rikimaru0345]

@rcFMS
While looking into the issues reported here #56 (comment)
I realized that many of the changes and fixes related to the aot gen (and some other things I've talked about) are actually part of of Ceras v5.

The old version (master branch) writes source code "by hand":
SourceFormatterGenerator (old)

The new version (Ceras-v5 branch) uses the actual expression tree and rewrites it to C#:
SourceFormatterGenerator (new)

Can you try the v5 beta instead of master?

[rcFMS]

Sorry for the delay, the last few days I was rather busy.

I tried v5 now but there seem to be 2 or 3 compiler errors in the current version:

[rikimaru0345]

@rcFMS
Those errors tell me that a needed nuget package / dll isn't available.
Unity can't resolve nuget packages on its own.
You need to restore the packages using visual studio or the nuget commandline.
With VS you can just click build and AgileObjects.ReadableExpressions.dll and its dependencies should end up in the build output.

I'm not sure how this can be solved without support from Unity.
If you have any idea please let me know!

[rcFMS]

Would it be not possible to just drop those dlls into Unity?
If you can give me some links where I can download them I can try it out next week (since I'm on vacation this week) ;)

[rikimaru0345]

Would it be not possible to just drop those dlls into Unity?
If you can give me some links where I can download them I can try it out next week (since I'm on vacation this week) ;)

Yes that's the idea of building it. It will copy the correct version to the output directory.
As for download links, I'm not sure if there are any. Maybe you could download the nuget package of https://github.com/agileobjects/ReadableExpressions but then you'd have to unpack it.
Or you could clone the repo and build it yourself.

But that's more work than just opening Ceras.sln and pressing F5 :)

I'm aware that it's a bad solution :( But I don't think there is much I can do.
Building the project and copying just the dependencies to your unity project is most likely the easiest way.

Hmm, maybe I can modify the CI build script to package all dependency .dlls into a new archive. That should make it a little easier.