Write a test for migrating RealmObject -> EmbeddedObject models

[nirinchev]

This was reported by a user in #2393. We should write a test migrating from RealmObject links to EmbeddedObject and confirm that there's actually a way to do it reliably and then work with @MongoCaleb to document it. We should also look into exposing the function to delete all orphaned backlinks mentioned in realm/realm-swift#7145.

[DominicFrei]

In the related discussion #2393 I've added an explanation about what's going on when migrating to EmbeddedObject.
Adding this here too since it is relevant:

#2393 (comment)

I have added three new tests for this as part of #2394:
https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L149-L343

@vardansargsyan92 As @nirinchev already mentioned, deleting objects in the oldRealm is not going to have any effect, since that's not going to be forwarded to the newRealm while the migration is already in progress. The oldRealm is basically just a reference to the database using the old schema and the newRealm is the view onto the database using the new schema.

You would have to iterate in the new realm therefore. The problem is now, looking at https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L272 specifically (which is the iteration we're talking about here - the one mentioned by Nikola) we see a crash.

It originates in https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Realm/Realm/Realm.cs#L1736 where the Metadata is used which was set earlier in https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Realm/Realm/Realm.cs#L281.

We check against the Metadata and find this to be set to isEmbedded already. This appears to be a bug to me at the moment since the EmbeddedObject is actually not set to embedded while the migration happens but after.

I've fixed a bug in core a while ago where this was actually done pre migration:
https://github.com/realm/realm-core/pull/4414/files#diff-1d2d2dee75a4c6f88551eafd9474f4a6bab23c83377ed2a7696ee694b0d6a753R729
This is fixed now which is why I assume this might be a problem with us checking against our already updated metadata.

I'm going to investigate further.

This will be enabled by adding functionality to take care of orphaned objects (EmbeddedObjects that aren't linked by other objects) and objects with too many backlinks (EmbeddedObjects that are linked by more than one other object) automatically.

[DominicFrei]

Update: This requires #2408 which itself requires realm/realm-core#4729.

[sync-by-unito]

➤ Dominic Frei commented:

Unassigned myself and set it to Parked since we really don't know if and when we will do that. It's just an enhance to automate something that users can do manually anyway and it depends on a Core change that would have to be done first.

[sipersso]

This was reported by a user in #2393. We should write a test migrating from RealmObject links to EmbeddedObject and confirm that there's actually a way to do it reliably and then work with @MongoCaleb to document it. We should also look into exposing the function to delete all orphaned backlinks mentioned in realm/realm-cocoa#7145.

@nirinchev When you say that we need to confirm there's actually a way to do it reliably, it gets me super worried. @DominicFrei @nirinchev Can you confirm that this is something that can be done reliably?

The migration samples doesn't contain any of the edge cases. I noticed that there were test cases done here for the dot net SDK https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L149-L343

Are these available on the iOS and Android SDK:s as well? Can I use this approach to test that my migration handles all the edge cases?

[nirinchev]

@sipersso what SDK are you using? If it's the .NET SDK, then yes, migrating from a RealmObject to EmbeddedObject is possible, but you'll need to manually ensure that there's exactly one link for each target object. It's a bit of a hassle, but should be doable.

[sipersso]

I am using both the iOS SDK(Swift) and The Android/Java SDK.

In Kotlin I am doing this to delete orphaned children

val orphanedChildren = realm.where("Child").findAll().filter { it.linkingObjects("Parent", "children").isEmpty() }
orphanedChildren.forEach {
         it.deleteFromRealm()
}

On iOS/Swift it is much more complicated since I can't access linking objects from the child during migration

//First find map all child identifiers to parent identifiers
var parentMap:[String:String] = [:]
migration.enumerateObjects(ofType: ParentObjectClass.className()) { (oldParentObject, newParentObject) in
        // Look for the child in the parent and increase counter if so found.
    let parentId = oldParentObject!["idProperty"] as! String

    oldParentObject!.dynamicList("chilren").forEach { oldChildObject, newChildObject in
    	let childId = oldChildObject!["idProperty"] as! String
    	parentMap[childId] = parentId
    }

}

//Now loop through the children and lookup their parent id and delete orphaned objects
migration.enumerateObjects(ofType: ChildObjectClass.className()) { (oldChildObject, newChildObject) in
    
    let childId = oldChildObject["idProperty"] as! String

    if(parentMap[childId] == null){
    	migration.deleteObject(newChildObject)
    }
}

This seems to work, but how can I be sure? Also, it doesn't handle the multiple links. While I don't think that this will happen, I want to make sure that I test for this possibility as well. The migration test example here https://github.com/realm/realm-dotnet/blob/df/embedded-object-migration-test/Tests/Realm.Tests/Database/MigrationTests.cs#L149-L343 seems to cover both cases. How can I write such a test on iOS and Android?

The only option I can think of right now is to branch the app off using the old realm configuration and try to create sample realms using that configuration that intentionally creates orphaned objects and multiple backlinks.

I need to be 100% certain that the migration doesn't cause crashes at startups for all my customers and now I don't know how to make sure that this will be a smooth transition. It is also a prerequisite for the app to be able to move to Atlas... and make you money ;)