AsyncOpen() with Flexible Sync: Progress Indication

[bdkjones]

Problem

If you use asyncOpen() to open a flexible-sync Realm of any non-trivial size, you'll find that it takes FOREVER. The download completes quickly, but Realm then takes an absolute geologic era to "bootstrap changesets", as shown here:

Unfortunately, Realm provides no progress API for this and requests to add it have been ignored for years...like many Realm issues. Mongo staff just tell you not to use {}asyncOpen(){}, but that's not an acceptable solution for many apps: users will start creating new model objects because they think those objects are missing when, in reality, they just haven't been "bootstrapped" yet. Once they are, we've got a mess of duplicate model objects: ugh.

The Correct Solution

Realm needs to add a progress API for asyncOpen() so that download and bootstrap progress can be reported to the user.

The Stupid Workaround

Until Realm fixes this, there is a fragile, no-good way to hack it ourselves. It's resource-intensive and dumb, but when it takes 5+ minutes to open a Realm, users NEED to see progress. My approach relies on the fact that Realm is very chatty in the logs:

private func setupBootstrapProgressTrackingTask()
{
    realmBootstrapProgressTask?.cancel()
    realmBootstrapProgressTask = nil
    
    realmBootstrapProgressTask = Task.detached 
    {
        do
        {
            let store: OSLogStore = try OSLogStore(scope: .currentProcessIdentifier)
            let predicate = NSPredicate(format: "composedMessage BEGINSWITH 'Info: Connection['")
            var totalChangesets: Double? = nil
            
            // "realmIsLoading" is an iVar on my @MainActor controller class where I'm opening the Realm.
            while await self.realmIsLoading && !Task.isCancelled
            {
                // NOTE: getEntries() appears to have a bug where it ignores the "with" and "at" parameters. No matter what I pass, this method always returns ALL entries that our process
                // has logged that match the predicate. I cannot get it to function any other way, so I've filed a Radar and I'm sure Apple will get around to fixing it right after the sun explodes.
                // I have tried the position(timeIntervalSinceEnd:) and position(date:) APIs. Neither works. I have tried the .reverse option hard-coded: it never works.
                // Without the predicate, there will be *thousands* of entries every three seconds. (Way more than you see in Console.app.)
                let entries: AnySequence<OSLogEntry> = try store.getEntries(matching: predicate)
            
                if totalChangesets == nil
                {
                    for entry: OSLogEntry in entries
                    {
                        // As of January 2024 and Realm 10.45.3, the first bootstrap message for a flexible sync realm is --> Info: Connection[1]: Session[1]: Begin processing pending FLX bootstrap for query version 1. (changesets: 7387, original total changeset size: 451327529)
                        // We need to pull out that total changeset number to calculate progress.
                        // Note: On first-run, Realm will post this log message for "query version 0" and there will be just 1 changeset. It finishes that nonsense, then starts syncing the real stuff a second or two later.
                        // I have no idea what a "query version" is or whether it will ever be != 1, but in observed practice, this log message is consistent each time.
                        // If the app is quit in the middle of the bootstrapping process, the first "query version 0" message does not appear and the app immediately begins with bootstrapping our changesets.
                        if let prefixRange: Range = entry.composedMessage.range(of: "Begin processing pending FLX bootstrap for query version 1. (changesets: "),
                           let commaIndex: String.Index = entry.composedMessage.firstIndex(of: ","),
                           commaIndex > prefixRange.upperBound,
                           let proposedTotal = Double(String(entry.composedMessage[prefixRange.upperBound ..< commaIndex])),
                           proposedTotal > 0                                                                                        // Don't divide by zero.
                        {
                            // The number we want is between these two bounds. Unsure if the new Regex stuff in macOS 12+ is faster than sniffing by hand.
                            totalChangesets = proposedTotal
                            print("Found total changesets: \(proposedTotal)")
                            
                            await MainActor.run {
                                self.realmSetupSpinnerItem.message = "Unpacking Database..."
                            }
                            break
                        }
                    }
                }
                
                guard let totalChangesets else {
                    try await Task.sleep(nanoseconds: 3_000_000_000)
                    continue
                }
                
                // As of January 2024 and Realm 10.45.3, the "regular update" bootstrap message for a flexible sync realm is --> Info: Connection[1]: Session[1]: Integrated 20 changesets from pending bootstrap for query version 1, producing client version 1146 in 633 ms. 7367 changesets remaining in bootstrap
                // We pull out the changesets remaining and use that to calculate the progress percentage that Mongo engineers SHOULD have provided, since bootstrapping a non-trivial Realm takes a freaking eternity (5+ minutes).
                // AnySequence does not have a concept of "last", so we have no choice but to reverse an array. Expensive. Sure wish Mongo would make all of this unnecessary.
                for entry: OSLogEntry in entries.reversed()
                {
                    if let suffixRange: Range = entry.composedMessage.range(of: " changesets remaining"),
                       let msRange: Range = entry.composedMessage.range(of: "ms. "),
                       suffixRange.lowerBound > msRange.upperBound,
                       let remainingChangesets = Double(String(entry.composedMessage[msRange.upperBound ..< suffixRange.lowerBound]))
                    {

                        let progress: Double = (1.0 - (remainingChangesets / totalChangesets))
                        print("remaining changesets: \(remainingChangesets), total: \(totalChangesets), progress: \(progress)")
                        print("Bootstrapping Progress: \(NumberFormatter.localizedString(from: NSNumber(value: progress), number: .percent))")

                        // Update whatever UI is showing a progress bar:
                        await MainActor.run {
                            self.realmSetupSpinnerItem.progressPercentage = progress
                            self.realmSetupSpinnerItem.message = "Unpacking Database — \(NumberFormatter.localizedString(from: NSNumber(value: progress), number: .percent))"
                        }

                        break
                    }
                }
                
                try await Task.sleep(nanoseconds: 3_000_000_000)
            }
        }
        catch
        {
            print("The bootstrap log message tracking task threw an error: \(error.localizedDescription)")
            
            // Whatever failed is likely going to keep failing, so we'll just revert back to indeterminate progress and cancel.
            await MainActor.run
            {
                self.realmSetupSpinnerItem.message = "Unpacking Database..."
                self.realmSetupSpinnerItem.progressPercentage = 0.0
                self.realmBootstrapProgressTask?.cancel()
                self.realmBootstrapProgressTask = nil
            }
        }
    }
}

Using The Workaround

In whatever controller where you're opening the Realm async, add an ivar:

final class SomeController
{
    private var bootstrapProgressTask: Task<Void, Never>? = nil
}

Then, just before you call {}asyncOpen(){}, call {}setupBootstrapProgressTrackingTask(){}. This task will check for Realm log messages every 3 seconds and calculate the current progress of the bootstrapping. You'll just need to swap in your own code to go back to the main thread and update whatever UI is appropriate (for me, that's a "SpinnerItem" object that holds a message and progress value that I display in my UI.)

When asyncOpen() completes or errors, cancel the Task. (Or have the Task look up some iVar on your controller object that indicates loading is happening and cancel itself if that value changes.)

Warnings:

1. This is obviously stupid and fragile. If the log messages change, it will break. If an error occurs, the Task cancels and just resets to an indeterminate progress indicator. This is also resource-intensive and wasteful. But it's worth it so that we don't leave users hanging with a 5-minute indeterminate spinner, wondering if the app is broken or hung.

2. This method does not show progress of the actual download from Atlas. For my app, that happens very quickly—about 15 seconds. The "bootstrapping" is what takes FOREVER to complete.

For Realm Engineers:

Progress APIs for long-running tasks are not optional. They are not "nice to have" things. They are expected and this problem should have been solved and closed years ago. Users can't be expected to let an app just sit and spin for 5+ minutes without force-quitting it and thinking it's broken.

How important is this improvement for you?

Dealbreaker

Feature would mainly be used with

Atlas Device Sync

[bdkjones]

I don't know much about how Realm's internals work. It seems like Realm is establishing a "history" of changes so that it can sync writes. If that's the case, it would be really great if asyncOpen() could at least load the latest "snapshot" of the Realm in a read-only state so that data can be displayed in the UI immediately while this restoration of changesets continues in the background. When the changesets are all integrated, then the Realm could be available for writes.

Unfortunately, I assume that this process works "from the bottom up", so that it's not possible to load the latest state of the Realm (even in a read-only state) until all 12,000+ changesets have been "integrated". That makes Realm Sync a really poor choice for any large, shared data store.

[tgoyne]

Each of the log messages are printed after committing a batch of changesets, and if you do a non-async open during bootstrap application you can read the data written so far. We don't release the write lock while this is happening so trying to write while bootstrap application is happening will block until it's complete.

I think our assumption around query bootstrapping would be that the download would be the slow part rather than the application, but that's clearly not the case here.

[Jaycyn]

Question: What is your use case for .asyncOpen? The code presented in the question doesn't show how/why it's implemented.

There is a property wrapper in SwiftUI @AsyncOpen that leverages Realm.asyncOpen() which, according to the docs does have a progress indicator

We've got a fairly large flexSync dataset with about 10,000 objects and never really experienced any kind of delays in accessing the data. We're trying to understand how that would be used so we can avoid crazy long delays going forward.

[bdkjones]

Sure, happy to add context!

1. This is a Mac app built with AppKit targeting macOS 13+.

2. It's an enterprise application for a company in Hollywood.

3. The closest analogy is iTunes: the app holds about 30 pieces of metadata (artist, album, etc.) for roughly 1 million audio files.

4. That master list of audio files is then used to track when a particular file appears in a project (and for how long it appears) so that the studio can determine licensing fees to be paid. There are about 10,000 "projects" right now and each links to about 800-1000 audio files.

5. There are about 20 employees who all use this app simultaneously and it is not possible for me to narrow down what data is loaded--ALL of the audio files have to be loaded because when a new project is added, we need to scan all of the existing files to see if we already have a record for a given file that's used in the project. All projects must be loaded because we display them in a giant outlineView on the left side of the window.

6. On iOS, where screens are small and we don't display much at a time, I gather we'd load only a subset of the data--grab only projects to list until a user taps something, then slide over and load more data for whatever he tapped. Unfortunately, on Mac, that's not how UIs work so there is less ability to "segment" the loads.

7. A design requirement for this app was real-time sync. It's supposed to be like a giant version of iTunes where 20 people are editing metadata on tracks or adding new playlists and all that work should show up live, in realtime, for everyone--like a Google Doc.

8. Once the database is loaded, Realm works great. It's just the initial bootstrapping that faces a long delay.

[tgoyne]

Progress notifications for flexible sync downloads are in progress. The reason we originally didn't have them is that the server is just streaming query results to the client, rather than buffering the full result set in memory on the server before sending them. The server has now implemented estimated progress information for this, and we're working on surfacing it in the client.

We're planning on eliminating the separate download and application steps for query bootstraps, at least in the async open case. This would make the whole thing quite a bit faster, and relevant to this issue, make it so that the download progress notifications cover everything you need. We probably need to keep the separate download and apply steps for changes to subscriptions while the app is running, so that won't completely remove the need for bootstrap application progress notifications.

[bdkjones]

@Jaycyn regarding asyncOpen(): I'm aware of the SwiftUI property wrapper (covered at https://www.mongodb.com/developer/products/realm/realm-asyncopen-autoopen/), but this is not a SwiftUI app. Moreover, I think that progress case applies only during data download, not during bootstrapping?

The end goal is to disallow users from editing data in the app unless the model is up-to-date with the latest changes. This is a desktop app where the Mac is virtually guaranteed to be online (unless a server is down, etc.).

Suppose a project named "Top Gun" exists. If we open the Realm with empty data and it takes a few minutes to sync everything down in the background, we don't want the user to go, "Oh, the Top Gun project is missing. I'll just create it again." Then, once sync finishes, we have two "Top Gun" projects that we now have to merge/de-dupe. This is my use-case for asyncOpen(): I want the full local Realm populated before the user starts making changes.

[Jaycyn]

I see and thanks for the explanation - we are a macOS developer and no SwiftUI so we understand - please bear with my question.

I am not really clear on the difference (speed wise) between the bootstrapping and downloading.

We've used await code (previously a closure) that shows a spinner while downloading data with a 'please wait' message which doesn't allow the user to create anything new while downloading, just browse existing data.

let realm = try await Realm(configuration: config, downloadBeforeOpen: .always)

once that completes, the data is fully downloaded and the user can then create objects. While not ideal for the user, it at least avoids them creating duplicate objects.

I am no longer finding references or examples to asyncOpen in the documentation, other than in the previously mentioned context of the SwiftUI wrapper and a blurb in the API.

That then begs the question; what about AsyncOpenTask? Which states

This task object can be used to observe the state of the download or to cancel it. This should be used instead of trying to observe the download via the sync session as the sync session itself is created asynchronously, and may not exist yet when Realm.asyncOpen() returns.

So... and this is more of a @tgoyne question; is AsycOpenTask not functional? Or will that be the implementation with asyncOpen in the future. Or is the await the way to go?

Asking questions as while the delays we experience are handled via the above, as the dataset and app size grows, so will the delays so what's the best practice here - the workaround provided in the original post?

[bdkjones]

@Jaycyn yea, in my case the download is pretty quick: 20 seconds or so. It's the bootstrapping that requires MINUTES to complete. I don't think that progresstask tracks the bootstrapping process after the actual bytes have finished downloading from Atlas.

@tgoyne mentioned these two steps are going to be combined. At that point, the progresstask might be useful.

[bdkjones]

@tgoyne In the meantime, what can I do to minimize the number of changesets that must be bootstrapped? I've lowered the client-reset window to 5 days on Atlas so the app isn't keeping 30 days' of sync events. Do many small write transactions (as opposed to fewer, larger write transactions) produce more changesets? If there's something I can do to optimize/reduce these, I'm game.

[tgoyne]

The changesets in the query bootstrapped phase are synthesized history from the backing mongodb data and nothing you do other than change how much data you're subscribing to will change how many of them there are. It's only after bootstrapping is complete that you start receiving changesets from other clients. The raw number of changesets is also not super meaningful, as we apply the changesets in batches to reduce the per-transaction overhead. The batch size is currently only 1 MB which is probably too small, but increasing that to a much larger number would probably be something like a 10-20% speedup and not something which makes it not painfully slow.

[nirinchev]

Hey, so I'll close this as a duplicate of #8476 for the progress notifications case. It's something we've been working on for a while server-side and hope to expose support for it on the client in the near future.

For the actual changeset application speedup, you can subscribe to realm/realm-core#7285 for updates. This is more of a medium-term project with a lot of moving parts though, so I imagine it'll take a bit longer to land.