Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unity editor crash caused by Firestore lock issue #638

Closed
vyrin opened this issue Apr 7, 2020 · 17 comments
Closed

Unity editor crash caused by Firestore lock issue #638

vyrin opened this issue Apr 7, 2020 · 17 comments
Assignees

Comments

@vyrin
Copy link

vyrin commented Apr 7, 2020

Please fill in the following fields:

Unity editor version: 2019.3.7f1
Firebase Unity SDK version: 6.13.0
Source you installed the SDK (.unitypackage or Unity Package Manager): Unity Package Manager
Firebase plugins in use (Auth, Database, etc.): Auth, Firestore, App
Additional SDKs you are using (Facebook, AdMob, etc.): None (replicated in a clean new project)
Platform you are using the Unity editor on (Mac, Windows, or Linux): Mac
Platform you are targeting (iOS, Android, and/or desktop): Editor
Scripting Runtime (Mono, and/or IL2CPP): Mono

Please describe the issue here:

I created a brand new project and installed the Firebase SDK via the Unity Package Manager.

I created a single button that when pressed creates a single listener on a Firestore document.

public void Test() { var store = FirebaseFirestore.DefaultInstance; var doc = store.Document("/versions/1.0/people/foo"); doc.Listen(snapshot => { Debug.Log("Got snapshot"); }); }

I run the project (play button) in the editor, click the Test button, and everything works correctly.

I then stop the project (pause button), and then start the project again (play button).

When I click on the Test button the Unity Editor crashes.

This is the relevant portion of the crash report:

Application Specific Information: dyld: in dlopen() /Applications/Unity/Hub/Editor/2019.3.7f1/Unity.app/Contents/lib/libPackages/com.google.firebase.app/Firebase/Plugins/x86_64/FirebaseCppApp-6_13_0.bundle.so abort() called terminating with uncaught exception of type firebase::firestore::util::FirestoreInternalError: FIRESTORE INTERNAL ASSERTION FAILED: third_party/firebase/ios/Releases/FirebaseFirestore/core/src/firebase/firestore/core/firestore_client.cc(168) void firebase::firestore::core::FirestoreClient::Initialize(const firebase::firestore::auth::User &, const firebase::firestore::api::Settings &): Failed to open DB: Internal: Failed to open LevelDB database at /Users/colin/Library/Application Support/firestore/__FIRAPP_DEFAULT/com-quicksteplabs-wordz/main: LevelDB error: IO error: lock /Users/colin/Library/Application Support/firestore/__FIRAPP_DEFAULT/com-quicksteplabs-wordz/main/LOCK: already held by process (expected created.ok())

So, it appears the Editor is crashing because it attempts to lock the Firestore LOCK file, but it already locked it before (presumably the first time the project ran), and now it is crashing due to an assertion failure.

Please answer the following, if applicable:

This issue was replicated in a brand new project that simply had the Firestore SDK added via the Unity Package Manager.

This happens 100% of the time.

(I'm actually very surprised that I haven't seen anyone else report this issue. It is 100% consistent for me and has halted all of my progress integrating Firestore into my projects.)

@vyrin vyrin added the new New issue. label Apr 7, 2020
@wilhuff wilhuff self-assigned this Apr 8, 2020
@wilhuff
Copy link

wilhuff commented Apr 8, 2020

Hi! Sorry for the trouble.

The issue is essentially as you describe. The prior instance of Firestore hasn't completely shut down and it is still holding the lock on the underlying database. The next instance starts up and then fails to acquire the lock. The trouble comes from the fact that there's some nondeterminism in the way we're handling the shutdown of the AppDomain that represents the test run.

We had recently discovered this issue ourselves, but I'm pretty surprised this is happening every time you pause. We have seen this failure approximately 1 in 20 times.

So the good news is we have a plan for how to fix this (and a start on the implementation), but unfortunately this will take some time to land.

@vyrin
Copy link
Author

vyrin commented Apr 11, 2020

Thanks for the update. (And thank you so much for all the work on adding Firestore to Unity. I really really appreciate it!)

Do you have an estimate for when this issue will be fixed in an official release? Are we talking days, weeks, or months? (That will help me in my planning.)

And relatedly, is there any workaround that I could try to get it working beforehand?

@wilhuff
Copy link

wilhuff commented Apr 11, 2020

Given the non overlapping release schedules of the components that are required for this, it really couldn’t happen sooner than two weeks from now at the earliest.

@cynthiajoan cynthiajoan removed the new New issue. label Apr 13, 2020
@AThilenius
Copy link

AThilenius commented Apr 16, 2020

I also see this every time I reenter play-mode. Any recommendations on a quick (nasty) hack to release the lock until a patch comes out? Would save me lots of restarting-Unity time 😁

@wilhuff
Copy link

wilhuff commented Apr 16, 2020

Unfortunately, the lock is being held by a C++ object that has leaked so there's no way to get back to it. Normally I would suggest that you could disable offline persistence in settings, but we haven't shipped that API yet...

If you want to work around this you need to find some way to get the two instances to use different files for their storage. There are a few flavors of this:

  • On UNIX filesystems you also just remove the directory contents. The process will still have the file open, keeping the inode alive, but the filenames will become available for reuse. Once you exit the process the space will be reclaimed. This won't work on Windows.
  • An alternative to the above is to move the directory containing the database aside (renaming the directory or similar).
  • An alternative that's guaranteed to work but requires a code change is to create a new Firebase App (with a random name) for each run and create your Firestore instance from that App. The app name is part of the path Firestore uses, and a random app name will prompt the SDK to create a new database.

In all of these cases note that the files will still be open and the editor process will eventually run out of files/memory, but it will at least stop you from needing to restart every time.

@Thaina
Copy link
Contributor

Thaina commented Apr 17, 2020

Not sure it was related. My editor also eventually keep crashing after switching play mode and edit mode in editor. I could test run my app 2-5 times before the editor will crash again

I don't know where to find unity editor crash log but I have file crash report to unity, the unity team have response back to me as this below

From the crash logs, I can see that this crash happens in FirebaseCppApp-6.0.0.dll

My unity report case is Case 1238725

Is it related or I should open new issue?

@wilhuff
Copy link

wilhuff commented Apr 17, 2020

The Firestore lock issue is very clearly about Firestore failing to open its local database and the crash will be in the Firestore DLL. This sounds like a different issue.

@Thaina
Copy link
Contributor

Thaina commented Apr 20, 2020

@wilhuff Thanks, I have file new issue #653

@kirilogan
Copy link

kirilogan commented Apr 29, 2020

Seems the lock occurs when the code is changed while in play mode and the editor does not dispose properly. I've been able to stop it from crashing by making sure that I stop play mode before saving any code changes. Hope this helps for the time being.

@ijaureguialzo
Copy link

ijaureguialzo commented May 12, 2020

Following @wilhuff 's advice, I used this as temporary workaround and Unity has stopped crashing on every run:

db = FirebaseFirestore.GetInstance(FirebaseApp.Create());   // was FirebaseApp.DefaultInstance

Hope it helps.

wilhuff added a commit to firebase/firebase-ios-sdk that referenced this issue May 18, 2020
In particular:

  * All tasks are guaranteed to either run to completion or not run at all;
  * Executors can now be destroyed from the threads they own (if applicable); and
  * Many common concepts between the Executor implementations have been harmonized and shared.

Note:

  * This change does not yet implement `Executor::Dispose` or clean up any of the higher level componentry to take advantage of these changes.
  * The executors implement shutdown by canceling any tasks that haven't started yet. This diverges from the plan (which called for a separate `UserCallbackExecutor` to handle this), but in the end it turns out to be simpler to just cancel everything and make cancel handle tasks-in-progress in a reasonable way. LMK if you have grave objections to this, in which case I can rework this.

This is the foundation for addressing firebase/quickstart-unity#638.
wilhuff added a commit to firebase/firebase-ios-sdk that referenced this issue May 18, 2020
Now that the Executor makes strong guarantees that tasks submitted to it will either be completely run or not start at all once destruction starts (#5547), build a Dispose protocol on top of this to allow Firestore to shut itself down leaf first.

Now that Firestore calls the Dispose chain down to the Executor, it's no longer possible for tasks to run while the instance is partially destructed nor can user callbacks execute after the instance has been destroyed.

This is the other half of the iOS changes required to address firebase/quickstart-unity#638.

There are a few follow-on tasks that this PR does not address:

  * Ownership of FirestoreClient and AsyncQueue are still shared though they no longer need to be.
  * The tests are still overly defensive about keeping different test cases isolated from each other. Much of this can be removed now that we're guaranteed to release resources by the time the test case shuts down (at least in GoogleTest; XCTest remains to be seen).
@summitsteven
Copy link

summitsteven commented May 28, 2020

Hi, I see PRs from @wilhuff pulled into the ios SDK in this thread 10 days ago. Does that mean the fix has made it in to the Unity SDK (which also released 10 days ago)? Or will that come in the next release? Is there a way we can tell how up to date the Unity SDK is regarding fixes/updates to other repos?

Thanks!

@wilhuff
Copy link

wilhuff commented May 28, 2020

Sorry for the delay. The PRs that landed last week will go out in the next iOS release. After that we can make a Unity release that includes the fix.

@maximeLong
Copy link

maximeLong commented Jun 3, 2020

Hey @wilhuff do you have an estimate on when the next Unity release will make it out? I'm getting this error as well and it has halted development on using Firestore for our iOS Unity app. Should we roll a realtime DB solution in the intermit, or will a hotfix be released soon for Unity Firestore SDK? Thanks.

@wilhuff
Copy link

wilhuff commented Jun 5, 2020

As a general policy we can't really make promises about when things will go out. There are a variety of reasons why things may be delayed that are unrelated to these specific changes.

With that said, if all goes well, this fix should be in the next release. It's possible that might not be the case though: for example the next release might be a patch for the prior one for changes unrelated to Firestore.

@maximeLong
Copy link

maximeLong commented Jun 5, 2020

With the addition of the methods TerminateAsync() and ClearPersistenceAsync() in the 6.15 release of Firebase Unity SDK, this issue has been resolved for me.

If I add these methods to the OnApplicationQuit event in my class, this issue seems to be resolved:

FirebaseFirestore db;

void Awake()
{
    Firebase.FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(task => 
    {
        var dependencyStatus = task.Result;
        if (dependencyStatus == Firebase.DependencyStatus.Available) 
        {
            db = FirebaseFirestore.GetInstance(FirebaseApp.Create());
        }
    });
}

void OnApplicationQuit() {
    db.TerminateAsync();
    db.ClearPersistenceAsync();
}

Note: Unity documentation states iOS applications are usually suspended and do not quit. So for this method to work you might need to tick "Exit on Suspend" in Player settings to cause the game to quit and not suspend. I have not tested this thoroughly yet.

@wilhuff
Copy link

wilhuff commented Jun 5, 2020

Right, that's a workaround you can use for now. The fix that's pending should make it so that an explicit shutdown/clear like this should be unnecessary.

@wilhuff
Copy link

wilhuff commented Jun 30, 2020

The fix for this was released with the Firebase Unity SDK 6.15.1.

@wilhuff wilhuff closed this as completed Jun 30, 2020
@firebase firebase locked and limited conversation to collaborators Jul 31, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants