Realm Obfuscation (Unity + Local Database)

[peachypixels]

Problem

Hello!

I've recently integrated obfuscation into an application and hit a number of issues, mostly regarding Realm objects (i.e., IRealmObject)

The Realms are encrypted (no issues there) but I am also wanting to obfuscate Realm object & property names.

Initially I tried to use the obfuscation tool to achieve this (so the obfuscator changed the woven Realm 'proxy' classes) but this resulted in the following issue...

https://www.mongodb.com/docs/atlas/device-sdks/sdk/dotnet/troubleshooting/#resolve-a--no-properties-in-class--exception

After much trial and error, the only way I can obfuscate the Realm aspect of the app is to...

1. Use MapTo attributes on Realm objects & properties
2. Tell the obfuscator to ignore all Realm objects

But the limitation\issue with this approach is that...

1. Mono builds (in Assembly-CSharp.dll) & IL2CPP builds (in global-metadata.dat) still contain the un-obfuscated names. I believe it's not too difficult (with the right knowledge\tools) to correlate the un-obfuscated & obfuscated names.
2. Calculated properties (i.e., non-persisted) on the Realm objects are also not obfuscated.

The obfuscation tool that I'm using obfuscates mostly by means of attributes. I had hoped these would carry across to their counterparts on the woven 'proxies' but this doesn't seem to be the case.

So I'm wondering if there is a known workaround? If not, could this be considered for a feature request please?

Solution

Couldn't the woven 'proxies' be generated using the mapped names? They currently do, but only as attributes with literal strings. Class & property names still use their un-mapped names.

Alternatives

I'm considering a brute force approach of using explicit obfuscated names (for Realm objects & properties) but this means uncompiled code becomes far less readable and is really an absolute last resort.

How important is this improvement for you?

Would be a major improvement

Feature would mainly be used with

Local Database only

[sync-by-unito]

➤ PM Bot commented:

Jira ticket: RNET-1134

[nirinchev]

What tool are you using for obfuscation? I guess the main question is if it's running before or after the Realm weaver, because if it's running after, I would expect things to work just fine. One thing we do use reflection for is the generated RealmSchema field - if you exclude just that from obfuscation, do you still experience the same problems?

[peachypixels]

Hi Nikola,

Many thanks for the reply. The tool (that I'm currently evaluating) is the Beebyte obfuscator. There are others under consideration, but let's stick with this one for now.

If I extract the woven 'proxy' from the Mono DLL, the class & property names are clearly obfuscated, which I had assumed to mean the obfuscation was occurring after weaving (during build)

If it helps, I can send the 'proxy' of a test Realm object before (i.e., weaved in the Unity editor) and after (i.e., weaved during build). If so, let me know where to E-mail it to.

As for RealmSchema, I don't believe that would be the issue. The Realm package is (from what I can tell) stored in a separate assembly that is currently un-obfuscated. I wanted to get the basics working before turning to obfuscating third party tools.

Looking at the obfuscation translation table, the only reference to Realm (other than my own classes) is RealmModuleInitializer

[nirinchev]

Yeah, it'd be helpful if you could share those dlls to nikola.irinchev@mongodb.com.

I'm assuming you're using the source generator based models - i.e. your models implement IRealmObject rather than inherit from RealmObject. If that's the case, then in the generated class, there'll be a RealmSchema static property, which we're getting using reflection:

realm-dotnet/Realm/Realm/Schema/ObjectSchema.cs

Line 261 in cc537e6

var schemaField = type.GetField("RealmSchema", BindingFlags.Public | BindingFlags.Static);

. I can't seem to find docs for Beebyte, but if it has the option to configure members that should be skipped, you could try and annotate the RealmSchema property. This is not a real solution as regenerating the file will remove your changes, but we could at least try and see if it works.

One thing I can see that won't work correctly is property change notifications. Those use strings generated at compile time, so if you take advantage of it, after obfuscation, the names in the property change event handler won't match the obfuscated names of the class.

[peachypixels]

OK, let me do some more research (with RealmSchema) and I'll message back in the next day or two.

In answer to your question, yes I'm using IRealmObject and not inheriting from RealmObject

I quickly looked at the Realm "C:\Users<user>\AppData\Local\Temp\VSGeneratedDocuments<id>\ClassName_generated.cs" proxy file and could not find the code you posted above.

I found the below though, is that what you're referring to?

public static Realms.Schema.ObjectSchema RealmSchema = new Realms.Schema.ObjectSchema.Builder("MappedClassName", ObjectSchema.ObjectType.RealmObject)
{
    Realms.Schema.Property.Primitive("MappedProperty1Name", Realms.RealmValueType.String, isPrimaryKey: true, indexType: IndexType.None, isNullable: true, managedName: "Property1Name"),
    Realms.Schema.Property.Primitive("MappedProperty2Nam>", Realms.RealmValueType.Data, isPrimaryKey: false, indexType: IndexType.None, isNullable: true, managedName: "Property2Name"),
}.Build();

That was from an IRealmObject decorated with MapTo attributes (for the obfuscated names)

As for the property change events, that may not be an issue here. I'm using mapped names that would otherwise match the obfuscated names. That said, I'm not using those events (yet) so haven't proved that theory.

Thanks again for the rapid response, it's much appreciated.

[peachypixels]

Hi Nikola,

I've looked a little deeper into the RealmSchema suggestion and I think you're right, this could be the issue.

Going on your code link above, Realm is looking for a field called "RealmSchema" by means of a magic string.

I rebuilt the app in question with Realm obfuscation enabled and the static property now looks like this (extracted from a Mono build using dotPeek)...

static PLHAHHKPCAB()
{
  ObjectSchema.Builder builder = new ObjectSchema.Builder("MappedClassName", (ObjectSchema.ObjectType) 0);
  builder.Add(Property.Primitive("MappedProperty1Name", (RealmValueType) 3, true, (IndexType) 0, true, nameof (NDNBBOCIAKN)));
  builder.Add(Property.Primitive("MappedProperty2Name", (RealmValueType) 4, false, (IndexType) 0, true, nameof (HHPHLHLODFC)));
  PLHAHHKPCAB.MPCIFIMONMC = builder.Build();
}

We can see the obfuscator has renamed "RealmSchema" to "MPCIFIMONMC" therefore triggering the exception at line 283 from the code link you posted above.

Ok, so to test the theory I need to compile the app and somehow exclude the "RealmSchema" field from obfuscation.

The obfuscator allows classes to be excluded by means of an editor setting or an attribute. Class members can only be excluded by means of an attribute. So the only way to exclude "RealmSchema" from obfuscation is by using an attribute. Unfortunately custom attributes don't carry across to the woven proxies and even if they did, the "RealmSchema" field is not something under control of the app.

Assuming the attribute could somehow be added, the obfuscator supports custom attributes [Skip] & [SkipRename] that do similar things. So one of those could be added, but obviously they couldn't exist in Realm production. It might be possible to inject one via code in the build pipeline, but I think that'll be a huge amount of work for a simple test. However the obfuscator also supports the standard dot net obfuscation attributes...

[System.Reflection.Obfuscation]
[System.Reflection.Obfuscation(ApplyToMembers=false)]
[System.Reflection.Obfuscation(Exclude=false)]

So Realm could theoretically decorate the "RealmSchema" field with a [System.Reflection.Obfuscation] attribute (that defaults to exclude true) and it'll work for everyone.

During the build process, Realm weaving certainly appears to run before obfuscation (according to the console). So even if I could add it before building, I expect it'll be overwritten during build weaving?

On-top of that, if I "Find All References" to MyRealmObject in Visual Studio, it shows the generated Realm proxy file, but it's marked as auto generated and is zero bytes. So it looks like there's some VS magic going on there and doesn't appear to be editable.

Other possibilities are maybe using nameof() instead of a magic string (to lookup the field) or allowing the magic string value to be customisable somehow.

Hopefully this all helps, but if there's anything else you need just say and I'll do what I can.

[nirinchev]

Hey sorry for the delay, I had a family thing end of last week. The problem with nameof is that we don't have access to the field - the code I linked to is in the Realm library, whereas the code we're looking up is in your library, so we cannot reference members directly (at the time we build Realm, we don't have access to everyone's codebases). Additionally, since RealmSchema is a static property, we cannot add it as abstract interface member since Realm needs to support .NET Standard 2.0 and there this is not supported.

I'll try and update the generator to add [System.Reflection.Obfuscation] to the generated RealmSchema member and will send you a test build to see if that will solve the problem for you, since we don't have access to the Beebyte obfuscator.

[peachypixels]

Hey @nirinchev

Many thanks for the update.

Understood about using nameof(). I dived deeper into the code yesterday and see what you mean. You're just dealing with Type at that point and trying to have even minimal strong typing, would likely involve substantial changes.

Thanks so much for taking the time to implement a potential workaround, it's really appreciated. I'm using the tarball package (V12.0.0) so not sure if it's possible to send an updated version of that? It might be easier to send the relevant updated DLL (i.e., in the library package cache) but have a feeling Unity will restore any cache files (from the package) if they detect hash deltas.

Either way, I'll send you a mail and include some extra obfuscator info that may be of help.

[nirinchev]

Hey, really sorry for the delay here - this fell a bit between the cracks. I've tried adding the attribute and built this package - can you give it a shot and see if that fixes the obfuscation problems:

https://drive.google.com/file/d/1qgu3YfUktWajHk_d9OWGkRYbmStPQF6J/view?usp=share_link

[peachypixels]

Hey @nirinchev

No problems at all, assumed this was low priority.

So I've just run a very quick test and it appears to have addressed the issue above...

1. The above package was installed via the UPM without issue.
2. I removed one Realm record type from the Obfuscators skip obfuscation list (so it would be go through the obfuscation process)
3. The weaving process ran without issue
4. The woven proxy (for the specified record type) now included the [System.Reflection.Obfuscation] attribute on the RealmSchema static property.
5. Performed a Mono build (without issue)
6. Extracted the relevant class from the assembly (using dotPeek) and can see that the proxy is now fully obfuscated
7. Performed an IL2CPP build (without issue)
8. Installed the APK on a device and ran it (without issue)

So far so good :-)

That said, I've only tested very limited functionality (e.g., reading, writing, deleting, compacting) for one record type, not multiple types and not property change events etc.

So I need to perform wider tests, but this is a step in the right direction.

Going forward (and assuming this does resolve the issue) is this something that Realm would be comfortable committing to production? At the very least, it would be nice if another external developer (either using the same or another Obfuscator) could confirm these findings.

Thanks so much for all your efforts so far, it's greatly appreciated.

PS: When I have time in the next week, I'll post an example of an obfuscated RealmSchema property.

[nirinchev]

Yes, we'd be happy to merge that PR - it's fairly trivial one and I don't see risks for regressions there.

[peachypixels]

Great, thanks!

I have a (full) Realm obfuscated build on a device now and will test it over the next week and report back.

[peachypixels]

Quick update... performed further tests on both Mono & IL2CPP (device) builds and so far so good. Zero issues to report.

Will test some more over the weekend, so by early next week I'll be able to report final results. But it's looking good.

[peachypixels]

Hey @nirinchev

Testing over the weekend uncovered zero issues with obfuscated Realms (Mono & IL2CPP builds) but with some caveats...

1. Only CRUD operations have been tested, along with compacting & copying databases, database encryption & MapTo attributes (no property change events, no database syncing etc)
2. Only basic data types were tested, strings, ints, floats, bools, byte[]
3. Only Android has been tested, not iOS or Windows etc

So with that in mind, this change gets the thumbs up from me. That said, I would personally label obfuscation support as experimental for the time being (if at all). But it's a great step in the right direction (and a very worthwhile feature addition to Realm imho)

Anything else you need, please just ask and I'll try and help out as best as possible.

Thanks again for all your time & effort with this, it's greatly appreciated :-)

[nirinchev]

Hey, thanks for the thorough testing done - I'll move forward and get that merged and ready for next official release. One thing I'm curious is if you're explicitly specifying the schema in your realm configuration - i.e. do you have something like:

var config = new RealmConfiguration
{
  schema = new[] { typeof(Foo), typeof(Bar) }
}

Based on some guesswork, I had assumed that without it, you would get an empty schema, but perhaps I've missed something.

As for the status of obfuscation support - we'll likely release as is and not explicitly document it as it's not a topic that has come up often enough. It's a best-effort type of support and we'd be happy to work with you and other members of the community to fix bugs and/or add annotations as long as they're general-purpose enough.

[peachypixels]

Yes that's exactly how I'm declaring Realm types. But once the codebase is obfuscated (during build) you'll end up with something like this...

var config = new RealmConfiguration
{
  schema = new[] { typeof(YWAKLCZP), typeof(QUVAPRTK) }
}

Realm proxy class method & property names will also be obfuscated (as above)

On-top of that, persisted properties are also decorated with MapTo attributes (set to equally obfuscated names)

Re obfuscation support, I think that's a very sensible way forward. To support the numerous obfuscation tools out there across all aspects of database management would be a big undertaking imo.

That's what I liked about the Obfuscation attribute approach as it's Dot Net generic, albeit obfuscators aren't obligated to support it.

Thanks again for everything and I'll post again if there's anything further to add.