Firestore crash - NSInternalInconsistencyException "Cannot allocate memory"

[jdwilson11]

[REQUIRED] Step 1: Describe your environment

• Xcode version: Version 12.3 (12C33)
• Firebase SDK version: 6.34.0
• Installation method: CocoaPods
• Firebase Component: Firestore

Crashlytics Device Information:

• Model: iPad mini (5th generation)
• Orientation: Face Up
• RAM free: 95.63 MB
• Disk free: 44.79 GB

[REQUIRED] Step 2: Describe the problem

Crashlytics is reporting the following Cannot allocate memory crash on a regular basis. A root cause hasn't been found yet, and it hasn't been reproducible in local development environments.

Fatal Exception: NSInternalInconsistencyException
FIRESTORE INTERNAL ASSERTION FAILED: Failed to commit transaction: <LevelDbTransaction NotifyLocalViewChanges: 0 changes (0 bytes):> Failed: IO error: /var/mobile/Containers/Data/Application/152E2636-1CAA-410D-BDDF-E9F96ABEAACD/Library/Application Support/firestore/__FIRAPP_DEFAULT/app-name/main/045490.ldb: Cannot allocate memory (expected status.ok())

Sample stack trace:

Fatal Exception: NSInternalInconsistencyException
0  CoreFoundation                 0x1aab185ac __exceptionPreprocess
1  libobjc.A.dylib                0x1bec0642c objc_exception_throw
2  CoreFoundation                 0x1aaa211b4 -[CFPrefsSearchListSource addManagedSourceForIdentifier:user:]
3  Foundation                     0x1abdd5c80 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:]
4  app-name                       0x100d18624 firebase::firestore::util::ObjcThrowHandler(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 59 (exception_apple.mm:59)
5  app-name                       0x100d18174 firebase::firestore::util::Throw(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 91 (exception.cc:91)
6  app-name                       0x100d677dc firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) + 42 (hard_assert.cc:42)
7  app-name                       0x100d6787c firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) + 49 (hard_assert.cc:49)
8  app-name                       0x100d83e50 firebase::firestore::local::LevelDbTransaction::Commit() + 208 (leveldb_transaction.cc:208)
9  app-name                       0x100d795d8 firebase::firestore::local::LevelDbPersistence::RunInternal(absl::lts_2020_02_25::string_view, std::__1::function<void ()>) + 261 (leveldb_persistence.cc:261)
10 app-name                       0x100d8a5c0 firebase::firestore::local::LocalStore::NotifyLocalViewChanges(std::__1::vector<firebase::firestore::local::LocalViewChanges, std::__1::allocator<firebase::firestore::local::LocalViewChanges> > const&) + 1831 (functional:1831)
11 app-name                       0x100dd2974 firebase::firestore::core::SyncEngine::EmitNewSnapshotsAndNotifyLocalStore(firebase::firestore::immutable::SortedMap<firebase::firestore::model::DocumentKey, firebase::firestore::model::MaybeDocument, firebase::firestore::util::Comparator<firebase::firestore::model::DocumentKey> > const&, absl::lts_2020_02_25::optional<firebase::firestore::remote::RemoteEvent> const&) + 555 (vector:555)
12 app-name                       0x100dd3bf0 firebase::firestore::core::SyncEngine::ApplyRemoteEvent(firebase::firestore::remote::RemoteEvent const&) + 93 (optional.h:93)
13 app-name                       0x100dbe7b8 firebase::firestore::remote::RemoteStore::RaiseWatchSnapshot(firebase::firestore::model::SnapshotVersion const&) + 46 (model_fwd.h:46)
14 app-name                       0x100df0c4c firebase::firestore::remote::WatchStream::NotifyStreamResponse(grpc::ByteBuffer const&) + 2209 (memory:2209)
15 app-name                       0x100dcc814 firebase::firestore::remote::Stream::OnStreamRead(grpc::ByteBuffer const&) + 2608 (memory:2608)
16 app-name                       0x100d5aa1c std::__1::__function::__func<firebase::firestore::remote::GrpcCompletion::Complete(bool)::$_0, std::__1::allocator<firebase::firestore::remote::GrpcCompletion::Complete(bool)::$_0>, void ()>::operator()() + 1720 (functional:1720)
17 app-name                       0x100cf99e4 firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) + 957 (atomic:957)
18 app-name                       0x100ddb064 firebase::firestore::util::Task::ExecuteAndRelease() + 1859 (functional:1859)
19 libdispatch.dylib              0x1aa716ac8 _dispatch_client_callout
20 libdispatch.dylib              0x1aa71dc08 _dispatch_lane_serial_drain
21 libdispatch.dylib              0x1aa71e734 _dispatch_lane_invoke
22 libdispatch.dylib              0x1aa728528 _dispatch_workloop_worker_thread
23 libsystem_pthread.dylib        0x1f1d87908 _pthread_wqthread
24 libsystem_pthread.dylib        0x1f1d8e77c start_wqthread

Crashed: com.google.firebase.crashlytics.ios.exception
0  app-name                       0x100ce614c FIRCLSProcessRecordAllThreads + 392 (FIRCLSProcess.c:392)
1  app-name                       0x100ce6530 FIRCLSProcessRecordAllThreads + 423 (FIRCLSProcess.c:423)
2  app-name                       0x100cdce8c FIRCLSHandler + 34 (FIRCLSHandler.m:34)
3  app-name                       0x100cd8b90 __FIRCLSExceptionRecord_block_invoke + 218 (FIRCLSException.mm:218)
4  libdispatch.dylib              0x1aa716ac8 _dispatch_client_callout + 20
5  libdispatch.dylib              0x1aa724c8c _dispatch_lane_barrier_sync_invoke_and_complete + 60
6  app-name                       0x100cd8328 FIRCLSExceptionRecord + 225 (FIRCLSException.mm:225)
7  app-name                       0x100cd8620 FIRCLSExceptionRecordNSException + 111 (FIRCLSException.mm:111)
8  app-name                       0x100cd7f08 FIRCLSTerminateHandler() + 279 (FIRCLSException.mm:279)
9  libc++abi.dylib                0x1bed052a8 std::__terminate(void (*)()) + 20
10 libc++abi.dylib                0x1bed05234 std::terminate() + 44
11 libobjc.A.dylib                0x1bec0668c _objc_terminate() + 14
12 libdispatch.dylib              0x1aa716adc _dispatch_client_callout + 40
13 libdispatch.dylib              0x1aa71dc08 _dispatch_lane_serial_drain + 580
14 libdispatch.dylib              0x1aa71e734 _dispatch_lane_invoke + 408
15 libdispatch.dylib              0x1aa728528 _dispatch_workloop_worker_thread + 708
16 libsystem_pthread.dylib        0x1f1d87908 _pthread_wqthread + 276
17 libsystem_pthread.dylib        0x1f1d8e77c start_wqthread + 8

Steps to reproduce:

It hasn't been reproducible locally.

Relevant Code:

N/A

[morganchen12]

Hey @jwilsoncredera, thanks for filing this bug. Here's my thought process:

• In the crash description you've posted the device has less than 100 MB of unused RAM remaining, leading me to believe leveldb allocations are failing due to memory pressure.
• Assuming you're recording these crashes via Crashlytics, you could try capturing total app memory footprint via this snippet when a memory warning is triggered and logging it via crash breadcrumbs.
• If the crashes occur consistently after memory warnings indicating your app is using too much memory, then you can prevent these crashes from happening by reducing your app's memory footprint.
• Otherwise, something else is causing the crashes and we'll need to investigate further.

Please let me know if I've missed anything.

[jdwilson11]

Thank you for the quick response!

We'll look into reducing the app's memory footprint. I'll respond once we have more data so we can close the ticket if that resolves the issue.

[wilhuff]

One thing to note is that we open Firestore's underlying LevelDB essentially on the first operation we perform. If we're failing this way there has to be some external cause.

[jdwilson11]

Do you know what the license is for the Gist snippet for total app memory footprints?

Since the Gist doesn't mention a license but is a repository, it seems like it can only be used on GitHub projects if I'm interpreting the GitHub Terms of Service correctly.

[wilhuff]

Unfortunately, that's an area we specifically cannot help you with. To give you a definitive answer is equivalent to giving legal advice, which we're not in a position to do.

My best advice is to contact the author and ask them to explicitly license their work, either personally to you, or by amending the gist to include an explicit license.

[morganchen12]

@jwilsoncredera is this still an issue?

[jdwilson11]

Hi, we did confirm the crash was correlated with high memory usage after using the application for several days in a row without restarting it. To verify this, we set up metrics to log memory usage over time using a correlation ID unique to each application load.

I'll close the issue now that we know the root cause was likely related to application memory leaks, giving us the right direction to address it. Thank you for the pointer!