Introduce _customRealmProperties() to all Realm Objects to allow for client-generated RLMProperties (via Swift Macros)

[rjchatfield]

Goals

Allow client code to generate RLMProperties for Objects.

Background

Late last year I started investigating performance problems in our app. Specifically, App Launch was taking ~700ms to perform the first read from a Realm: without migration, without Realm Sync. I isolated it to the creation of RLMSchema.shared(). I then attempted to use Swift Macros to try and generate the schema during compile time, but I had no luck.

Reading the GitHub issues and discussions for "macros", I found these:

• Initialisation/Migration Performance - What does good look like? #8194 (reply in thread)
• Build a MVP of macro-based model classes #8270 (comment)

@tgoyne was kind enough to reply and shared his experimental work from back in July to do exactly what I wanted. His branch also included some explorations into something similar to Apple's @Model, which I'm not interested in at this point. But it was a great jumping off point.

This PR

This PR introduces the minimum set of changes required for the Swift Macro to work.

• [RLMObjectBase] Add _customRealmProperties()
• [RealmSwift] Check _customRealmProperties() from within getProperties()
• [RLMProperty] Expose convenience init similar in shape to @Persisted property wrapper

More about the Macro

You can find the macro here: https://github.com/rjchatfield/realm-swift-macro

I'm pretty excited about this. I'm new to macros, and used a lot of what @tgoyne built last year. I stripped it down to just generating class func _customRealmProperties() -> [RLMProperties]. I tested this out in my own codebase and saw promising improvements.

Here is an example:

@CompileTimeSchema
open class FooObject: Object {
    @Persisted(primaryKey: true) var id: String
    @Persisted var name: String
    @Persisted(indexed: true) internal var key: String
    @Persisted public internal(set) var nestedObject: NestedObject?
    @Persisted private var embeddedObjects: List<NestedEmbeddedObject>

    var computed: String { "" }
    func method() {}

    @CompileTimeSchema
    @objc(ObjcNestedObject)
    public class NestedObject: Object {
        @Persisted(primaryKey: true) var id: String
        @Persisted var name2: String!
        @Persisted var nestedNeighbour: FooObject.NestedEmbeddedObject?
    }

    @CompileTimeSchema
    @objc(ObjcNestedEmbeddedObject)
    private final class NestedEmbeddedObject: EmbeddedObject {
        @Persisted var name3: String
    }
}

The macro's output:

open class FooObject: Object {
    @Persisted(primaryKey: true) var id: String
    @Persisted var name: String
    @Persisted(indexed: true) internal var key: String
    @Persisted public internal(set) var nestedObject: NestedObject?
    @Persisted private var embeddedObjects: List<NestedEmbeddedObject>

    var computed: String { "" }
    func method() {}
    @objc(ObjcNestedObject)
    public class NestedObject: Object {
        @Persisted(primaryKey: true) var id: String
        @Persisted var name2: String!
        @Persisted var nestedNeighbour: FooObject.NestedEmbeddedObject?

        public override class func _customRealmProperties() -> [RLMProperty]? {
            return [
                RLMProperty(name: "id", objectType: Self.self, valueType: String.self, primaryKey: true),
                RLMProperty(name: "name2", objectType: Self.self, valueType: String?.self),
                RLMProperty(name: "nestedNeighbour", objectType: Self.self, valueType: FooObject.NestedEmbeddedObject?.self)
            ]
        }
    }
    @objc(ObjcNestedEmbeddedObject)
    private final class NestedEmbeddedObject: EmbeddedObject {
        @Persisted var name3: String

        override class func _customRealmProperties() -> [RLMProperty]? {
            return [
                RLMProperty(name: "name3", objectType: Self.self, valueType: String.self)
            ]
        }
    }

    open override class func _customRealmProperties() -> [RLMProperty]? {
        return [
            RLMProperty(name: "id", objectType: Self.self, valueType: String.self, primaryKey: true),
            RLMProperty(name: "name", objectType: Self.self, valueType: String.self),
            RLMProperty(name: "key", objectType: Self.self, valueType: String.self, indexed: true),
            RLMProperty(name: "nestedObject", objectType: Self.self, valueType: NestedObject?.self),
            RLMProperty(name: "embeddedObjects", objectType: Self.self, valueType: List<NestedEmbeddedObject>.self)
        ]
    }
}

I then used this in my own codebase. I annotated the 467x Objects & EmbeddedObjects with @CompileTimeSchema (in 322 files). There is a bug in Swift which stopped me using it in 7 places, but that is already fixed in the next version of Swift.

The performance is great.

	Before	After
RLMSchema.shared()	612ms	233ms (45%)

This Macro SPM package depends on nothing else by Swift Syntax (which may one day be exposed through the Swift Toolchain and avoid re-compilation). I separated out all the tests into a separate module to avoid some SPM resolution issues I had in my codebase. https://github.com/rjchatfield/realm-swift-macro-tests/tree/main

Requested stats

ProductName:		macOS
ProductVersion:		14.3.1
BuildVersion:		23D60

/Applications/Xcode-15.1.0.app/Contents/Developer
Xcode 15.1
Build version 15C65

pod not found
(not in use here)

/bin/bash
GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)

/opt/homebrew/bin/carthage
0.39.0
(not in use here)

/usr/bin/git
git version 2.39.3 (Apple Git-145)

Thanks for reading

This is my first PR into Realm. I'm open to any and all feedback at this point. I also understand this change may not be wanted by Realm/Mongo, and I accept what ever you want to do with this PR. Thanks. :)

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Robert J Chatfield.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[nirinchev]

Hey, thanks for this. We'll have a meeting on Monday to go over it - the initial feedback from the team is that the approach seems reasonable.

A few things would be needed before we can accept this though:

1. We'd need you to sign a CLA (the form can be found here).
2. We'd like to have at least some minimal testing of the feature to make sure we don't accidentally break it in the future. You may want to hold off on this until we're able to give you more concrete feedback after Monday, though, to make sure you don't have to redo them in case the implementation needs bigger changes.

[rjchatfield]

Thanks for getting back to me.

1. I've now signed the CLA
2. I'll be happy to add tests after your round of feedback.

[rjchatfield]

You'll see here I've added a static method directly to the base class here. In the original code by @tgoyne, he defined a Swift Protocol and then added conformance via the Macro. I couldn't work with that solution for two main reasons:

1. Supporting Object subclassing was really tricky. There is no way to know from an Extension Macro if the class subclasses from a type that already conforms to this protocol - it just sees the exact syntax of this local type. So the Macro would naively add the conformance, then generate a compile time error that the type already conforms.
2. The purpose of this Macro is to improve runtime performance. In profiling my own use case, I noticed that adding this casting of the generic Base class to this Swift protocol type was not free. It added ~100ms to the schema generation. Which is almost equivalent to the time taken to generate all the actual properties for all my Objects. By defining the method on all Base Objects avoided this cast and that cost.

[rjchatfield]

Side note: This line is quite slow, particularly for the List Accessor which is generic over the list's Element. There is a runtime cast of the List Accessor to AnyObject? that takes a long time for the Swift runtime to lookup class conformance. I am no expert in this kind of optimisation, but I believe it's because the class type is defined in your module and the generic argument is defined in my module, and Swift has no way to reason about and inline it at compile time... so it just does it at runtime.

Fixing this somehow is one of the one last bottlenecks for an insanely fast schema initialisation. I'm happy to make a separate GitHub issue for that if desired.

[tgoyne]

This may be one of the things covered by the dyld cache on iOS, which means that it'll only be slow the first time an app is launched after each installation. If not, it is just frontloading work that'll need to happen at some point. Theoretically it could be done lazily, but right now we take advantage of that the schema is immutable after construction to share it between threads without any locking, and introducing locking could be a meaningful perf hit.

[tgoyne]

I think this approach is fine. I had hoped that overriding the existing sharedSchema class method would work, but it really wouldn't without significant changes.

[tgoyne]

I have not actually verified that this works, but if it does this should be @_spi(RealmSwiftPrivate) so that code using it has to do @_spi(RealmSwiftPrivate) import RealmSwift to reflect that this isn't a stable part of the API.

[rjchatfield]

Love this. In fact, I've pushed another commit to remove _customRealmProperties from the Objective-C files and done it as an extension in RealmSwift and annotated it with this @_spi. Since we only want to expose it to Swift Macros, makes sense yeah?

What do you think?

[rjchatfield]

Updates:

1. I've added tests, as requested. I hope they're good enough.
2. I've edited the author to match my GitHub email address.