ForestDB crashes frequently in the background because of protection data level

[basememara]

We came from SQLite storageType and things seemed to be more under control while we sprinkle UIApplication.sharedApplication().protectedDataAvailable throughout our app so things don't crash while the protected data is unavailable in the background.

Switching to ForestDB, things were noticeably fast, but noticing more frequent crashes while in the background. My thought is because ForestDB tries to do a lot of unattended maintenance in the background like auto-compaction and other things automatically.

However, it never checks to compare the protection data availability with its protection level set on the file, which is NSFileProtectionCompleteUnlessOpen by default. This I believe is an oversight by Couchbase Lite for iOS in general, the core code should constantly be checking the protectedDataAvailable otherwise Couchbase crashes the app in the background for the majority of test cases.

Any ideas how to suspend ForestDB in the background or turn off auto-compaction? Anything else I can do to make Couchbase Lite more robust and conscious of the protect data state while in the background?

Here are some of the crash reports:

0  MyApp                        0x1001b7a04 CLSProcessRecordAllThreads + 1572
1  MyApp                        0x1001b7a04 CLSProcessRecordAllThreads + 1572
2  MyApp                        0x1001b78c0 CLSProcessRecordAllThreads + 1248
3  MyApp                        0x1001a8070 CLSHandler + 48
4  MyApp                        0x1001b5ac8 __CLSExceptionRecord_block_invoke + 92
5  libdispatch.dylib              0x18110947c _dispatch_client_callout
6  libdispatch.dylib              0x181114728 _dispatch_barrier_sync_f_invoke
7  MyApp                        0x1001b556c CLSExceptionRecord + 212
8  MyApp                        0x1001b509c CLSTerminateHandler() + 616
9  libc++abi.dylib                0x180d16f44 std::__terminate(void (*)()) + 16
10 libc++abi.dylib                0x180d16fd0 std::set_new_handler(void (*)()) + 90
11 MyApp                        0x10073d8a8 __clang_call_terminate + 14
12 MyApp                        0x10076eec0 c4View::c4View(c4Database*, cbforest::slice, cbforest::slice, fdb_config const&, cbforest::slice) + 432
13 MyApp                        0x10076d010 c4view_open + 156
14 MyApp                        0x10074443c -[CBL_ForestDBViewStorage openIndexWithOptions:status:] + 368
15 MyApp                        0x100744224 -[CBL_ForestDBViewStorage lastSequenceIndexed] + 32
16 MyApp                        0x10080738c -[CBLDatabase(Views) queryViewNamed:options:ifChangedSince:status:] + 124
17 MyApp                        0x1008060ac __46-[CBLQuery runAsyncIfChangedSince:onComplete:]_block_invoke + 56
18 MyApp                        0x1007c1668 __35-[CBL_Server tellDatabaseNamed:to:]_block_invoke + 80
19 Foundation                     0x18209402c __NSThreadPerformPerform + 340
20 CoreFoundation                 0x18167509c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
21 CoreFoundation                 0x181674b30 __CFRunLoopDoSources0 + 540
22 CoreFoundation                 0x181672830 __CFRunLoopRun + 724
23 CoreFoundation                 0x18159cc50 CFRunLoopRunSpecific + 384
24 Foundation                     0x181faccfc -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 308
25 MyApp                        0x1007c20b4 -[CBL_RunLoopServer runServerThread] + 316
26 Foundation                     0x182093e4c __NSThread__start__ + 1000
27 libsystem_pthread.dylib        0x181323b28 _pthread_body + 156
28 libsystem_pthread.dylib        0x181323a8c _pthread_body + 154
29 libsystem_pthread.dylib        0x181321028 thread_start + 4

--

Fatal Exception: cbforest::error: unknown error

1  MyApp                        0x100409c74 cbforest::Database::compactionCallback(_fdb_file_handle*, unsigned int, char const*, fdb_doc_struct*, unsigned long long, unsigned long long, void*) (Database.cc:383)
2  MyApp                        0x100456274 _fdb_compact_move_docs(_fdb_kvs_handle*, filemgr*, hbtrie*, btree*, btree*, btree*, docio_handle*, btreeblk_handle*, unsigned long*, bool) + 244
3  MyApp                        0x100454b8c _fdb_compact_file(_fdb_kvs_handle*, filemgr*, btreeblk_handle*, docio_handle*, hbtrie*, hbtrie*, btree*, btree*, unsigned long long, bool) + 1004
4  MyApp                        0x100454794 fdb_compact_file + 976
5  MyApp                        0x10043e354 compactor_thread(void*) + 256
6  libsystem_pthread.dylib        0x182503b28 _pthread_body + 156
7  libsystem_pthread.dylib        0x182503a8c _pthread_body + 154
8  libsystem_pthread.dylib        0x182501028 thread_start + 4

[FDB ERR] Error in OPEN on a database file '/var/mobile/Containers/Data/Application/.../Library/Application Support/CouchbaseLite/cbase.cblite2/MyModel.viewindex.0', errno = 1: 'Operation not permitted'
2016-08-29 07:28:04.392 MyApp[639:152025] WARNING: ForestDB error: A file operation is not permitted (-50)

---

• Version: 1.3
• Client OS: iOS 9
• Server: Sync Gateway

[pasin]

Thanks for reporting this. I don't think we have tested this case. I don't think we have an option to disable / enable auto compaction on ForestDB.

[pasin]

One workaround that I can think of now is closing the database when the app goes into background and reopen when the app is on the background. However it's also requires some additional code change such as recreating views and replicators.

[basememara]

Thanks for the suggested workaround, but is a painful one. It would be useful if there was an API to suspend the database instead of completely closing it down, that would be more reasonable.

Still though, every internal call in Couchbase should be guarded using the native flag UIApplication.sharedApplication().protectedDataAvailable to check if it is able to perform the action or back off until it can. This check should be run regardless of the storage type.

[basememara]

Any more thoughts given to this? It's a rather nasty bug for Couchbase Lite, not just ForestDB.

Couchbase needs to be more conscious and smarter about running in the background on iOS. Currently, it's a completely ignored, but is a common use case. It's probably crashing just about every app out there that's using the default settings.

@snej

[snej]

Even if file protection triggers errors, it shouldn't lead to a crash. Looks like this is exposing some bugs in our code — the first crash looks like the result of a missing C++ try...catch block. The second may be a bug in our compactionCallback method. The crashes reported in #1433 appear to be bugs in ForestDB (I've filed a bug report in their Jira tracker.) We need to fix these as well as adding checks to make sure we don't try to use a file that's unopenable.

[snej]

There's no public API for disabling auto-compaction for ForestDB; I think you'd have to modify some code inside CBL to do that. For now I think Pasin's suggestion of closing the database is best.

[snej]

11 MyApp                        0x10073d8a8 __clang_call_terminate + 14
12 MyApp                        0x10076eec0 c4View::c4View(c4Database*, cbforest::slice, cbforest::slice, fdb_config const&, cbforest::slice) + 432
13 MyApp                        0x10076d010 c4view_open + 156

This is weird … I'm not sure why the C++ runtime is aborting in c4View::c4View, since c4view_open is protected by a catch(...) clause that should handle any exception.

[snej]

Figured out the above crash; resolved it as couchbase/cbforest#105.

[basememara]

That's great news! Is there a way I can try it out?

Also out of curiousity, how's the Couchbase team testing for this, because I don't think there's a way to unit test this because the simulator/IDE doesn't have encryption, I've only found that this can be only tested on a real device (unless I'm missing something).

Thanks again for following up on this!

[snej]

Well, that's the problem: we don't have good tests for this, because it has to be tested manually. That's how it slipped through.

[snej]

The fix for couchbaselabs/cbforest#105 has been merged into the master branch.