Overhead of subscriptions makes them unusable in real-world scenarios

[peppy]

What happened?

This issue escalates concerns raised in #2775. We have since had users reporting slowdowns in our game as a result of subscription usage. I've been able to reproduce the performance issues, and have isolated to a test project.

In the previous discussion, it was diligently explained that some overhead is expected from having subscriptions, but as we scale up to actual user workloads it's looking more and more like we cannot use subscriptions altogether.

In providing this test project and data, I hope there can be some kind of optimisations / improvement at realm's end to make this work better. If not, I am interested in any suggestions on how we can reduce the overhead, or work around it altogether.

Repro steps

Test project: test project.zip

Note that for the realm models this test project has our game package included via nuget. If required i can pull the models out.

Output of test project shows the overhead of two example subscriptions:

------------
Running with 0 subscriptions active

Refresh started...
Refresh completed! (0ms)

Performing arbitrary write...
Write completed.

Refresh started...
Refresh completed! (0ms)

------------
Running with 1 subscriptions active

Refresh started...
Got callback (1986 beatmaps)
Refresh completed! (5ms)

Performing arbitrary write...
Write completed.

Refresh started...
Got callback (1986 beatmaps)
Refresh completed! (4387ms)

------------
Running with 2 subscriptions active

Refresh started...
Got callback (18231 beatmaps)
Refresh completed! (2ms)

Performing arbitrary write...
Write completed.

Refresh started...
Got callback (1986 beatmaps)
Got callback (18231 beatmaps)
Refresh completed! (74392ms)

The data structure of interest here is that one BeatmapSetInfo can have multiple BeatmapInfo children.

The run with one subscription has the subscription on the top-level collection (BeatmapSetInfo) and the overhead is around 4 seconds for a collection of 1982 realm objects.

The run with two subscriptions adds a subscription on the child collection (BeatmapInfo) with a total overhead of 74 seconds for a collection of 18215 realm objects.

(as an aside, the user-reported case in our deployed production build has 7 active subscriptions on various realm collections, and a total refresh delay of 85 seconds. i've extracted only the two most expensive subscriptions for this reproduction - the rest are filtered to GUIDs and add less overhead)

From my perspective, these object numbers are not large enough to cause concern.

We do have other linked realm models which could be adding hidden overheads due to how the data diff is done internally (BeatmapSetInfo has many Files and BeatmapInfo has attached BeatmapDifficulty, Ruleset and BeatmapMetadata). I'm not immediately sure whether these are adding overhead but could test further if required.

Looking at what is being done in this sample project, from a high level it feels like there has to be a way for this to be optimised at realm's end.

Version

10.11.2

What SDK flavour are you using?

Local Database only

What type of application is this?

Other

Client OS and version

OS agnostic

Code snippets

using System.Diagnostics;
using osu.Game.Beatmaps;
using Realms;

string filename = Path.Combine(Environment.CurrentDirectory, "client.realm");

var realmConfiguration = new RealmConfiguration(filename)
{
    SchemaVersion = 14,
};

List<IDisposable> subscriptions = new List<IDisposable>();

using (var realm = Realm.GetInstance(realmConfiguration))
{
    var beatmapSets = realm.All<BeatmapSetInfo>();
    var beatmaps = realm.All<BeatmapInfo>();

    RunTest(realm, beatmapSets);

    subscriptions.Add(beatmapSets.SubscribeForNotifications(Callback));

    RunTest(realm, beatmapSets);

    subscriptions.Add(beatmaps.SubscribeForNotifications(Callback2));

    RunTest(realm, beatmapSets);
}

void RunTest(Realm realm, IQueryable<BeatmapSetInfo> beatmapSets)
{
    Console.WriteLine();
    Console.WriteLine("------------");
    Console.WriteLine($"Running with {subscriptions.Count} subscriptions active");

    Refresh(realm);

    Console.WriteLine();
    Console.WriteLine("Performing arbitrary write...");
    realm.Write(() => beatmapSets.First().DeletePending = true);
    Console.WriteLine("Write completed.");

    Refresh(realm);
}

static void Refresh(Realm realm)
{
    var stopwatch = new Stopwatch();

    stopwatch.Start();
    Console.WriteLine();
    Console.WriteLine("Refresh started...");
    realm.Refresh();
    Console.WriteLine($"Refresh completed! ({stopwatch.ElapsedMilliseconds}ms)");
}

static void Callback(IRealmCollection<BeatmapSetInfo> sender, ChangeSet changes, Exception error)
{
    Console.WriteLine($"Got callback ({sender.Count} beatmaps)");
}

static void Callback2(IRealmCollection<BeatmapInfo> sender, ChangeSet changes, Exception error)
{
    Console.WriteLine($"Got callback ({sender.Count} beatmaps)");
}

Stacktrace of the exception/crash you're getting

No response

Relevant log output

No response

[papafe]

Hi @peppy, thanks a lot for the test project, it's really useful.

We will take a look and see if there is room for improvement.

[papafe]

@peppy Just to give you a very small update. We managed to reproduce the issue, and we're investigating how to alleviate it

[nirinchev]

As a first step we should add keypath filtering for our Property/CollectionChanged subscriptions as those are single-level only and it'd be a waste to compute child property diffs. As a follow-up we can expose that publicly and allow developers to define the keypaths they're interested in.

[nirinchev]

@peppy forgot to circle back here, but with Realm 10.20.0 we released some perf improvements related to notifications, though those may not address your use case. Notably, CollectionChanged notifications should be much cheaper/faster now as those will only report changes to the collection itself, not for the objects. It's not fixing the issue with the repro project where you're observing object modifications, but perhaps it's something you can leverage in other parts of your codebase. One thing to note is that this only affects the CollectionChanged event, not SubscribeForNotifications.

[peppy]

Thanks for the mention. I have been following along and did notice the merged changes! And did also confirm that they didn't fix the case in this thread.

We will definitely be able to leverage that for some use cases though, I've made a note of it 👍 .