.NET process crashes when accessing a Realm that is also open in Studio

[nirinchev]

A user has reported that using .NET 11.4.0 and Studio 14.0.3, they get crashes when opening a synchronized Realm in both .NET app and Studio. Platform they're using is Windows. See slack thread for more details.

[jakobmollas]

Hi!
"User" here - I have tested the test application (RealmStudioCrash.sln) on my local machine and can recreate both issues, i.e.

1. Opening a Realm instance and then opening the db in Realm Studio -> "History type not consistent"
2. Opening a Realm db in Realm Studio and then trying to open a Realm instance for the same db via test app -> application terminates

Neither one of these are 100% consistent but happens fairly often.
It should be noted that:

• My test app (Atlas App) already contains an anonymous user with some existing schemas - those were added to the local db (but without any downloaded data)
• The Atlas App does not allow access to any data without having a valid JWT (custom JWT auth) with a claim that matches a field in the data model (we call it a TenantId) - this means the subscription will work but adding any data via Realm.Write will fail in the backend -> compensating local write etc.

I have attached my local db (from the test app that also includes some additional empty collections).
studio.zip

[jakobmollas]

The test app crashes in DEBUG builds, with/without a debugger attached.
It does not however seem to crash in RELEASE builds, at least I have not been able to trigger a crash (after 50 or so test runs).

That may be a clue.

[nirinchev]

Thanks for getting back - I have a fix for the history type issue: realm/realm-studio#1594 but I'm more interested in the app crashing - can you clarify the sequence of actions you take to make it crash in debug? Notably, do you first open the realm file in Studio, then run the app? Do you add some data/does some data get synchronized from your App Services application or does it crash on startup? I've tried a bunch of permutations I could think of but haven't been able to get it to crash.

[jakobmollas]

I first ran the test app to create a local realm db/file.
I tried to add some stuff but as expected it failed (due to security rules).
After that I terminated the app.
Then opened it up in Realm Studio by double-clicking the .realm file (I use Realm Studio via the windows installer/.msi).
That works fine and I can see 3 collections with no data.
I do not mess with any data via Realm Studio, simply let it stay open.
Then start the app again, via Visual Studio (debug build, both with debugger attached and then also without debugger).
After a few tries it fails (see previous screenshot).
After that it continues to fail intermittently on successive restarts (still just keeping the file open in Realm Studio, no errors there).

Running the app in Release mode works fine, switching back to debug mode again fails intermittently and so on.

It still fails intermittently after closing the file in Realm Studio and reopening again (in Studio) + then running the app, debug build.

[jakobmollas]

I have tried it out on two separate computers, both with similar result (it is not 100% consistent so it seems to be a timing bug of sorts). Both running Win 11 and the same VS + Realm Studio version.

[nirinchev]

And just to be clear - once you close the realm file in Studio, your app works just fine in debug mode again? I'll try to play a bit more with this and see if I can get it to reproduce.

[nirinchev]

I was able to reproduce this with a native debugger attached - seems to be due to MultipleSyncAgents getting thrown and bringing the sync client thread down. I'll continue investigating:

1. Why is this exception being thrown at all considering we're the only client there.
2. Why is it bringing the process down rather than throwing something catchable.

I'll report back once we have more info, but this is likely a bug in the Core database/sync client, so it may take a bit longer to resolve.

[jakobmollas]

Correct!

However, I am stupid regarding debug/release - I did not switch .realm file in Realm Studio.
If fails just as easily in release as long as I actually open the corresponding .realm file :-)

This is an example output - the first run it worked fine, the next run failed:

C:\Users\jakob\Downloads\realm-test\RealmStudioCrash\bin\Release\net7.0> .\RealmStudioCrash.exe
2023-09-07 16:45:12.200 Info: Realm sync client ([realm-core-13.17.1])
2023-09-07 16:45:12.206 Info: Platform: Windows Win32 unknown 10.0.22621.2070 x86_64
2023-09-07 16:45:12.215 Info: Connection[1]: Session[1]: Binding 'C:\Users\jakob\Downloads\realm-test\RealmStudioCrash\bin\Release\net7.0\studio.realm' to ''
2023-09-07 16:45:12.215 Info: Connection[1]: Session[1]: client_reset_config = false, Realm exists = true, client reset = false
2023-09-07 16:45:12.216 Info: Connection[1]: Connecting to 'wss://ws.eu-west-1.aws.realm.mongodb.com:443/api/client/v2.0/app/basicsync-emfqp/realm-sync'
2023-09-07 16:45:12.293 Info: Connected to endpoint '34.241.208.56:443' (from '127.0.0.1:60852')
2023-09-07 16:45:12.356 Info: Verifying server SSL certificate using 155 root certificates
2023-09-07 16:45:12.525 Info: Connection[1]: Connected to app services with request id: "64f9fe15850a35b601d62857"
Realm opened at C:\Users\jakob\Downloads\realm-test\RealmStudioCrash\bin\Release\net7.0\studio.realm. Type anything to add to Realm...
C:\Users\jakob\Downloads\realm-test\RealmStudioCrash\bin\Release\net7.0> .\RealmStudioCrash.exe
2023-09-07 16:45:13.804 Info: Realm sync client ([realm-core-13.17.1])
2023-09-07 16:45:13.809 Info: Platform: Windows Win32 unknown 10.0.22621.2070 x86_64

[nirinchev]

@jakobmollas we were able to narrow down the root causes of the issue you're seeing and I filed bug reports for the Core team to track and fix them. In the meantime, you can add the following (fairly hacky) code to your app that should allow you to keep Studio open while you debug/terminate your app:

var config = new FlexibleSyncConfiguration(...);

#if DEBUG
var lockFilePath = config.DatabasePath + ".lock";
if (File.Exists(lockFilePath))
{
    using var stream = new FileStream(lockFilePath, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
    stream.Seek(40, SeekOrigin.Begin);
    stream.WriteByte(0);
    stream.Flush();
}
#endif

var realm = Realm.GetInstance(config);

The root problem is that we're using a file called *your-realm-name*.lock to ensure there's always at most one sync process operating on a Realm file. When this process crashes or is otherwise terminated (e.g. when you stop a debug session in VS), the lock file remains in a state where it thinks there's a sync process already, so when your app restarts, it hits an exception in a background thread and terminates. The reason you're seeing it when Studio is open is because it keeps the .lock file in use, so when your app restarts, it thinks there is indeed someone using the Realm file, even though Studio itself does not have a sync agent running.

What the hacky workaround does is it opens the lock file and sets the sync agent flag to 0 before opening the Realm file. Since this is quite unsafe, we recommend you only do it in debug mode and only if you're certain there's only ever one process accessing the Realm file (i.e. you're not running multiple instances of your app at the same time). You should also only ever do it once when your app starts and before you open the Realm file.

I realize it's less than ideal and we'll try to prioritize fixing the Core bugs, but it should offer you a temporary path forward until we're ready. I'll close this issue now because the fixes are tracked in the linked tickets, but feel free to comment here or in the linked tickets if you have questions or concerns.

[jakobmollas]

Ah, ok I understand the dilemma here.
Thanks for the speedy investigation!

We can live with the workaround for a while but yeah, fixing the underlying issue would be really good.
Although we do not plan to keep the Realm open in multiple applications (Realm Studio will only be used for development/debugging) I guess there is always a risk that some lingering semi-crashed process could keep the lock file open in a similar way.