[Bug]: Unity Editor Crashing (Difficult to Repro)

[FreakTheMighty]

What happened?

We've been seeing hard crashes of the Unity Editor that fail to produce crash reports. We have flexible sync turned on and it seems possibly related to writing to the database. We keep thinking we have the crash fixed, initially by ensuring certain operation move onto the main thread, but then I make a small adjustment to the code, and the crash comes back. At that point, rolling the code back does not resolve the issue. One time I observed that restarting my computer resolved the crash with the same version of code.

Repro steps

I'm really struggling to identify clear steps. At this point the possible related elements are

• Flexible Sync
• WriteAsync
• I was trying to increment a RealmInteger, though removing this didn't solve the problem

Version

.Net Standard 2.0

What SDK flavour are you using?

MongoDB Atlas (i.e. Sync, auth, functions)

What type of application is this?

Unity

Client OS and version

Windows 11 Pro Build 22000.1098

Code snippets

await realmProp.Value.WriteAsync(
    () =>
    {
        syncedPointRecord = realmProp.Value.Add((SyncedPointRecord) pointRecord, true);
    });

Stacktrace of the exception/crash you're getting

If it does have to do with writing, this is the chunk of code that would be in the mix

#
await realmProp.Value.WriteAsync(
    () =>
    {
        syncedPointRecord = realmProp.Value.Add((SyncedPointRecord) pointRecord, true);
    });

Relevant log output

The last logs we see in Editor.log looks like this 


[Log] [] 2022-11-10 22:43:46.213 Debug: Connection[1]: Session[1]: Received: DOWNLOAD(download_server_version=38, download_client_version=22, latest_server_version=38, latest_server_version_salt=0, upload_client_version=38, upload_server_version=37, downloadable_bytes=0, last_in_batch=true, query_version=4, num_changesets=1, ...) []
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
Company.Core.CompanyLogger:Log_Internal (UnityEngine.LogType,string,object,UnityEngine.Object) (at Assets/Company/Core/CompanyLogger.cs:139)
Company.Core.CompanyLogger:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Assets/Company/Core/CompanyLogger.cs:215)
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
UnityUtils.UnityLogger:LogImpl (Realms.Logging.LogLevel,string)
Realms.Logging.Logger:Log (Realms.Logging.LogLevel,string)
Realms.Sync.AppHandle:HandleLogMessage (intptr,Realms.Native.PrimitiveValue,Realms.Logging.LogLevel)

[nirinchev]

When the unity editor crashes, it typically displays a dialog to report the error containing a bunch of native stacktraces - can you capture those the next time the crash occurs and attach them to the issue? That'll help quite a lot with the investigation.

[FreakTheMighty]

@nirinchev unfortunately that dialog is not showing up. And the crash logs also don't show anything.

[FreakTheMighty]

Here is a different traceback I see in the Editor logs.

[Log] [] 2022-11-10 23:30:25.374 Debug: User agent string: 'RealmSync/12.9.0 (Windows Win32 unknown 10.0.22000.708 x86_64)  ' []
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
Company.Core.CompanyLogger:Log_Internal (UnityEngine.LogType,string,object,UnityEngine.Object) (at Assets/Company/Core/CompanyLogger.cs:139)
Company.Core.CompanyLogger:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Assets/Company/Core/CompanyLogger.cs:215)
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
UnityUtils.UnityLogger:LogImpl (Realms.Logging.LogLevel,string)
Realms.Logging.Logger:Log (Realms.Logging.LogLevel,string)
Realms.Sync.AppHandle:HandleLogMessage (intptr,Realms.Native.PrimitiveValue,Realms.Logging.LogLevel)
Realms.SharedRealmHandle:OpenWithSyncAsync (Realms.Native.Configuration,Realms.Sync.Native.SyncConfiguration,Realms.Schema.RealmSchema,byte[],intptr)
Realms.Sync.SyncConfigurationBase/<CreateHandleAsync>d__18:MoveNext ()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<Realms.SharedRealmHandle>:Start<Realms.Sync.SyncConfigurationBase/<CreateHandleAsync>d__18> (Realms.Sync.SyncConfigurationBase/<CreateHandleAsync>d__18&)
Realms.Sync.SyncConfigurationBase:CreateHandleAsync (Realms.Schema.RealmSchema,System.Threading.CancellationToken)
Realms.RealmConfigurationBase/<CreateRealmAsync>d__44:MoveNext ()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<Realms.Realm>:Start<Realms.RealmConfigurationBase/<CreateRealmAsync>d__44> (Realms.RealmConfigurationBase/<CreateRealmAsync>d__44&)
Realms.RealmConfigurationBase:CreateRealmAsync (System.Threading.CancellationToken)
Realms.Sync.FlexibleSyncConfiguration:<>n__0 (System.Threading.CancellationToken)
Realms.Sync.FlexibleSyncConfiguration/<CreateRealmAsync>d__6:MoveNext ()
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<Realms.Realm>:Start<Realms.Sync.FlexibleSyncConfiguration/<CreateRealmAsync>d__6> (Realms.Sync.FlexibleSyncConfiguration/<CreateRealmAsync>d__6&)
Realms.Sync.FlexibleSyncConfiguration:CreateRealmAsync (System.Threading.CancellationToken)
Realms.Realm:GetInstanceAsync (Realms.RealmConfigurationBase,System.Threading.CancellationToken)
Company.Data.RealmStore.SyncedRealmDataSource/<ConnectSync>d__7:MoveNext () (at Assets/Company/Data/Realm/SyncedRealmDataSource.cs:46)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:Run ()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction (object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback (System.Threading.ContextCallback,object,System.Threading.Tasks.Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run (System.Threading.Tasks.Task,bool)
System.Threading.Tasks.Task:FinishContinuations ()
System.Threading.Tasks.Task:FinishStageThree ()
System.Threading.Tasks.Task`1<Realms.Sync.User>:TrySetResult (Realms.Sync.User)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<Realms.Sync.User>:SetResult (Realms.Sync.User)
Realms.Sync.App/<LogInAsync>d__14:MoveNext ()
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:Run ()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction (object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback (System.Threading.ContextCallback,object,System.Threading.Tasks.Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run (System.Threading.Tasks.Task,bool)
System.Threading.Tasks.Task:FinishContinuations ()
System.Threading.Tasks.Task:FinishStageThree ()
System.Threading.Tasks.Task`1<Realms.Sync.SyncUserHandle>:TrySetResult (Realms.Sync.SyncUserHandle)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<Realms.Sync.SyncUserHandle>:SetResult (Realms.Sync.SyncUserHandle)
Realms.Sync.AppHandle/<LogInAsync>d__13:MoveNext ()
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:Run ()
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation/<>c:<.cctor>b__7_0 (object)
UnityEngine.UnitySynchronizationContext/WorkRequest:Invoke ()
UnityEngine.UnitySynchronizationContext:Exec ()
UnityEngine.UnitySynchronizationContext:ExecuteTasks ()

[nirinchev]

Hm... so this seems to be happening on Windows and I don't have too much experience with how Unity behaves there. It's possible the crash report dialog only shows up on macOS. Do you do all your development on Windows or do you also have mac developer machines? The challenge here is that the stacktrace doesn't show anything unusual - it shows that the you're opening a synchronized realm, then you start getting log messages from sync, which we report with UnityEngine.Logger:Log. There's not even an error message that would point us to what the reason for the crash is.

[FreakTheMighty]

@nirinchev typically we do see the crash dialog, just not in this case. We are only on windows, so it wouldn’t be easy to test this on a Mac.

[nirinchev]

Do you have a project you can share that I can use to repro the issue locally? Next week @fealebenpae will be back from leave and he might have some tricks up his sleeve to get more info from the windows crash, but I'm a bit out of ideas at this point.

[FreakTheMighty]

I don’t at the moment, but I can try putting one together tomorrow. I had assumed it was something very specific to our setup, but it’s certainly worth seeing if this is easy to repo with Windows Unity and Flexible Sync.

[FreakTheMighty]

Two more observations, I’ve noticed that sometimes when it begins to crash other assembly dlls can become corrupted requiring me to rebuild the Unity Library folder. I wonder if there are some shenanigans happening with weaving?

I believe that we didn’t have this issue with partition based syncing.

[FreakTheMighty]

@nirinchev unfortunately I've been unable to reproduce this is a simple project. In our full project it does appear to crash on Realm.GetInstanceAsync, but in the sample project it does just fine.

[FreakTheMighty]

@nirinchev I may have a lead here!

I think that things go into a bad state if Unity crashes while a realm with sync is running. After that, it seems that only a restart of the computer (and possibly resetting the realm) gets things back into working order.

If I do something like this, forcing a crash, realm struggles to recover even after restarting Unity and removing the crash line. This isn't a perfect reproduction of the problem, as I sometimes see Unity just hang, and not crash, but I feel like I'm onto something with this. Unfortunately Unity crashes quite frequently for a variety of reasons, and I'm thinking that when it does something in Realm doesn't get cleaned up properly. Perhaps a port is left open or the state of the database is corrupted 🤷 .

  var config = new FlexibleSyncConfiguration(user)
  {
      ClientResetHandler = new RecoverOrDiscardUnsyncedChangesHandler(),
      Schema = schema
  };

  var realm = await Realm.GetInstanceAsync(config);

  realm.WriteAsync(() =>
  {
      realm.Add(new SyncedPointRecord { Id = "2552/004", ZoneId = "2552" }, true);
  });

  // Force a crash, then comment this out after restarting Unity
  UnityEngine.Diagnostics.Utils.ForceCrash(ForcedCrashCategory.Abort);

Thoughts?

---

Edit: Seems that its important to attempt a write while causing a crash.

[nirinchev]

That's interesting - when Unity crashes, does it always crash without capturing a crash report? While I've also seen occasional Unity Editor crashes, I don't feel like I've experienced a whole lot of those, so it might be interesting if it's something unrelated in your project/the plugins you're using or maybe some weird interaction between Realm and other components of your game. I'll try and force some crashes with the suggested code snippet to see if I'll be able to notice a pattern, but if you do manage to capture any crash reports, even if they don't seem to be directly caused by Realm, I'd be curious to take a look at those.

[FreakTheMighty]

Geez, I don't know how I missed this. That sure sounds close to what I'm seeing. The main thing is I don't think I'm running multiple instances (usually). Still, it makes me wonder if things aren't closing cleanly and then I get similar behavior.

[FreakTheMighty]

@nirinchev do you know if there would be an issue with having a local realm AND a synced realm running at the same time?

[nirinchev]

The main reason you could be getting crashes due to the above is if you're running the game in the editor and also run it as a standalone app. In such a case, they'll both attempt to open the same synchronized Realm file, which is not supported yet. If you're opening a local and a synced realm, they're unlikely to be sharing the same path, so that should be totally fine. One workaround for that would be to specify different base directories for editor and standalone apps. Something like:

var appConfig = new AppConfiguration("your-app-id");
if (Application.isEditor)
{
    appConfig.BaseFilePath = "C:\\xyz\\editor";
}
else
{
    appConfig.BaseFilePath = "C:\\xyz\\standalone";
}

Note that this will mean you're going to be downloading the Realm file twice - once for each of the base paths, but that way you can run your game as standalone and inside the editor at the same time and it has the added benefit you'll be able to pretend they're two different clients talking to one another.

But again, if you don't think you're doing something like that, the problem may be elsewhere.

[TigerLouck]

I'm from the same org as @FreakTheMighty, and I have been getting the same crashes with the same characteristics as others here, but for whatever reason, my crashes are going through the unity crash handler, rather than being abort or fastfail calls, so I do have a stack trace that might give you more information here.
The stack trace on the calls is the same on every crash, you can find it at the end of this log. I suspect that for whatever reason my instance is catching the exception before it propagates up out of unmanaged memory while others aren't, and that's why we're seeing unhandled crashes.
Editor.log

I'm reasonably certain that the mechanism behind my crashes and other crashes in our project is the same, and there is some variation even in the crashes I see (I have also seen crashes where the editor appears to instantly close and there is no message or handler, again in the same reproduction conditions)

[nirinchev]

I've been able to reproduce some sync-related crashes that occur when tearing down the appdomain. Unfortunately, I don't know if those are related to what you're seeing as the stacktrace Unity gives us doesn't offer much detail about the exact methods being executed. But I'll try and work with the Core team to get those fixed, after which we can see if that'll resolve the issues you're seeing.

[github-actions]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.