Keeping file locks for database in shared app group after write violates Apple's recommendation and causes the system kill the app with 0xdead10cc

[tkafka]

How frequently does the bug occur?

All the time

Description

Here is the investigation: #5904 (comment) plus the previous posts in thread.

TLDR: Current design, where file locks are kept even outside write transactions, violates iOS rules when used for a database inside a shared app group (which is the only way to share data between app and its extensions, like WidgetKit, and is a major use case for Realm on iOS). When Realm database is opened in an extension process, Realm keeps the database and auxiliary files locked, and system kills the app extension with 0xdead10cc as a cause.

Here is a more info from Apple arguing why the design, which leaves locked files in the shared app group, is bad: https://developer.apple.com/forums/thread/655225?answerId=632433022#632433022 (linked from #7466 (comment))

It’s always unsafe for an app to hold a file lock while it’s suspended. If some other app [1] attempts to take that lock, it’ll end up waiting indefinitely. This is an annoying source of app hangs. Moreover, the user can’t clear the hang by terminating the just-launched app. They must either terminate the suspended app or restart the device.

...

Finally, some concrete advice:

• Avoid file locks where possible.
• For this sort of task, consider using Core Data, which is built in to the system and thus will do the right thing.
• If you must use a file lock, make sure to release it before you become eligible for suspension. For example, you might wrap this work in a UIApplication background task that prevents the app from being suspended while the work is in progress.
• If you’re using a third-party library, check that it follows these rules. If it doesn’t, raise this issue with the library’s vendor.

The library's vendor is Realm, and it is clearly leaving locked files after finished writes, which causes iOS to kill the app. So I am raising this as an issue.

This was written two years ago (Change in iOS 14 Beta 3 to trigger 0xdead10cc - https://developer.apple.com/forums/thread/655225?answerId=628197022#628197022)

If you unlock the file by calling flock(fd, LOCK_UN) before the app
enters suspended state, the app won't be killed by iOS.

Right, this is expected. Holding file locks while your app is suspended is exactly what triggers the 0xdead10cc termination. You must release file locks before coming eligible for suspension.

If Realm has communicated clearly that it doesn't support sharing data with the extensions (which it doesn't with the current design), I would have chosen a different solution (either SQLite or Core Data).

Related issues:

#6861 #5904 #6671

Stacktrace & log output

Incident Identifier: CEB6D0C5-3705-4036-8CF1-7AEF8717EDD3
Beta Identifier:     C7D3E2F4-B21A-4367-A94B-1877923362F4
Hardware Model:      iPhone14,2
Process:             <myappname> [1466]
Path:                /private/var/containers/Bundle/Application/CDD04A57-258B-4F40-922E-B00933ADC007/<myappname>.app/<myappname>
Identifier:          <my app id>
Version:             1.0.128 (935)
AppStoreTools:       14B44
AppVariant:          1:iPhone14,2:16
Beta:                YES
Code Type:           ARM-64 (Native)
Role:                unknown
Parent Process:      launchd [1]
Coalition:           <myappname> [571]

Date/Time:           2022-11-04 10:46:22.7880 +0100
Launch Time:         2022-11-04 08:27:09.6526 +0100
OS Version:          iPhone OS 16.1 (20B82)
Release Type:        User
Baseband Version:    2.12.02
Report Version:      104

Exception Type:  EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Termination Reason: RUNNINGBOARD 3735883980 

Triggered by Thread:  0

Thread 0 name:   Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   SwiftUI                       	       0x1b5e8af20 0x1b5d9f000 + 966432
1   SwiftUI                       	       0x1b5e31584 0x1b5d9f000 + 599428
2   AttributeGraph                	       0x1d51d76f4 AG::Graph::UpdateStack::update() + 520
3   AttributeGraph                	       0x1d51d6f44 AG::Graph::update_attribute(AG::data::ptr<AG::Node>, unsigned int) + 424
4   AttributeGraph                	       0x1d51d5c00 AG::Subgraph::update(unsigned int) + 844
5   SwiftUI                       	       0x1b5dbcf28 0x1b5d9f000 + 122664
6   SwiftUI                       	       0x1b5dbac24 0x1b5d9f000 + 113700
7   SwiftUI                       	       0x1b5e5fbf0 0x1b5d9f000 + 789488
8   SwiftUI                       	       0x1b65516a4 0x1b5d9f000 + 8070820
9   SwiftUI                       	       0x1b5df3ee0 0x1b5d9f000 + 347872
10  libdispatch.dylib             	       0x1b99dc4b4 _dispatch_call_block_and_release + 32
11  libdispatch.dylib             	       0x1b99ddfdc _dispatch_client_callout + 20
12  libdispatch.dylib             	       0x1b99ec7f4 _dispatch_main_queue_drain + 928
13  libdispatch.dylib             	       0x1b99ec444 _dispatch_main_queue_callback_4CF + 44
14  CoreFoundation                	       0x1b249e6f8 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
15  CoreFoundation                	       0x1b2480058 __CFRunLoopRun + 2036
16  CoreFoundation                	       0x1b2484ed4 CFRunLoopRunSpecific + 612
17  GraphicsServices              	       0x1eb786368 GSEventRunModal + 164
18  UIKitCore                     	       0x1b49633d0 -[UIApplication _run] + 888
19  UIKitCore                     	       0x1b4963034 UIApplicationMain + 340
20  SwiftUI                       	       0x1b5f70014 0x1b5d9f000 + 1904660
21  SwiftUI                       	       0x1b5ed116c 0x1b5d9f000 + 1253740
22  SwiftUI                       	       0x1b5eba4bc 0x1b5d9f000 + 1160380
23  <myappname>                  	       0x100b712b8 0x100b68000 + 37560
24  dyld                          	       0x1d0aec960 start + 2528

Thread 1 name:  Realm notification listener
Thread 1:
0   libsystem_kernel.dylib        	       0x1ef007480 kevent + 8
1   <myappname>                  	       0x100e5b11c 0x100b68000 + 3092764
2   <myappname>                  	       0x100e5b2fc 0x100b68000 + 3093244
3   libsystem_pthread.dylib       	       0x1ff4686cc _pthread_start + 148
4   libsystem_pthread.dylib       	       0x1ff467ba4 thread_start + 8

Thread 2 name:  com.apple.uikit.eventfetch-thread
Thread 2:
0   libsystem_kernel.dylib        	       0x1ef005b48 mach_msg2_trap + 8
1   libsystem_kernel.dylib        	       0x1ef018008 mach_msg2_internal + 80
2   libsystem_kernel.dylib        	       0x1ef018248 mach_msg_overwrite + 388
3   libsystem_kernel.dylib        	       0x1ef00608c mach_msg + 24
4   CoreFoundation                	       0x1b247eaf0 __CFRunLoopServiceMachPort + 160
5   CoreFoundation                	       0x1b247fd34 __CFRunLoopRun + 1232
6   CoreFoundation                	       0x1b2484ed4 CFRunLoopRunSpecific + 612
7   Foundation                    	       0x1ac822334 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 212
8   Foundation                    	       0x1ac82221c -[NSRunLoop(NSRunLoop) runUntilDate:] + 64
9   UIKitCore                     	       0x1b4a9833c -[UIEventFetcher threadMain] + 436
10  Foundation                    	       0x1ac83b808 __NSThread__start__ + 716
11  libsystem_pthread.dylib       	       0x1ff4686cc _pthread_start + 148
12  libsystem_pthread.dylib       	       0x1ff467ba4 thread_start + 8

Thread 3 name:  com.apple.NSURLConnectionLoader
Thread 3:
0   libsystem_kernel.dylib        	       0x1ef005b48 mach_msg2_trap + 8
1   libsystem_kernel.dylib        	       0x1ef018008 mach_msg2_internal + 80
2   libsystem_kernel.dylib        	       0x1ef018248 mach_msg_overwrite + 388
3   libsystem_kernel.dylib        	       0x1ef00608c mach_msg + 24
4   CoreFoundation                	       0x1b247eaf0 __CFRunLoopServiceMachPort + 160
5   CoreFoundation                	       0x1b247fd34 __CFRunLoopRun + 1232
6   CoreFoundation                	       0x1b2484ed4 CFRunLoopRunSpecific + 612
7   CFNetwork                     	       0x1b37dc5a8 0x1b3586000 + 2450856
8   Foundation                    	       0x1ac83b808 __NSThread__start__ + 716
9   libsystem_pthread.dylib       	       0x1ff4686cc _pthread_start + 148
10  libsystem_pthread.dylib       	       0x1ff467ba4 thread_start + 8

Thread 4:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 5:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 6:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 7:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 8:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 9:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 10:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 11:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0

Thread 12:
0   libsystem_pthread.dylib       	       0x1ff467b90 start_wqthread + 0


Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x000000011e991828   x1: 0x000000011e9919b0   x2: 0x000000011e9919d0   x3: 0x00000001b5e8af20
    x4: 0x000000016f2958e8   x5: 0x0000000209e10aa0   x6: 0x0000000000000000   x7: 0x0000000000000001
    x8: 0x000000011e991828   x9: 0x000000010526ae00  x10: 0x0000000000000008  x11: 0x0000002200000003
   x12: 0x0000002000000003  x13: 0xfffffffefe74ba9e  x14: 0x000000000029c945  x15: 0x0000000000000005
   x16: 0x9dc300011e9919d0  x17: 0x0000000000006990  x18: 0x0000000000000000  x19: 0x000000016f295b10
   x20: 0x0000000127b2068c  x21: 0x0000000208c447c0  x22: 0x0000000283c8dbd0  x23: 0x00000000000b0796
   x24: 0x0000000000000002  x25: 0x00000002825d85a8  x26: 0x0000000127b20670  x27: 0x00000002825d85ac
   x28: 0xffffffffffffffff   fp: 0x000000016f295a70   lr: 0x00000001b5e31584
    sp: 0x000000016f295a60   pc: 0x00000001b5e8af20 cpsr: 0x60001400
   far: 0x0000000104ca8210  esr: 0x56000080  Address size fault

Binary Images:
       0x1b5d9f000 -        0x1b76e2fff SwiftUI arm64e  <494a7d5790af3e9096237038275b8d87> /System/Library/Frameworks/SwiftUI.framework/SwiftUI
       0x1d51d3000 -        0x1d5215fff AttributeGraph arm64e  <b787692356a03db1a0d6a92a11291c14> /System/Library/PrivateFrameworks/AttributeGraph.framework/AttributeGraph
       0x1b99da000 -        0x1b9a20fff libdispatch.dylib arm64e  <fea36690a0003c55b7009120b05aa69b> /usr/lib/system/libdispatch.dylib
       0x1b2404000 -        0x1b27e9fff CoreFoundation arm64e  <5cdc5d9ae5063740b64ebb30867b4f1b> /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
       0x1eb785000 -        0x1eb78dfff GraphicsServices arm64e  <a633a095122639428f413877660185ee> /System/Library/PrivateFrameworks/GraphicsServices.framework/GraphicsServices
       0x1b45c1000 -        0x1b5d9efff UIKitCore arm64e  <179501b60fc2344ab969b4e3961ebe10> /System/Library/PrivateFrameworks/UIKitCore.framework/UIKitCore
       0x100b68000 -        0x1018a7fff <myappname> arm64  <3a6449d8502c32ef8886c08012682a43> /private/var/containers/Bundle/Application/CDD04A57-258B-4F40-922E-B00933ADC007/<myappname>.app/<myappname>
       0x1d0ad7000 -        0x1d0b5a00f dyld arm64e  <cb3ff411476234d286a4eca13f9fb6c3> /usr/lib/dyld
       0x1ef005000 -        0x1ef03bffb libsystem_kernel.dylib arm64e  <ff27fc8f90ba3332ab7ac5bc2d9ca7b1> /usr/lib/system/libsystem_kernel.dylib
       0x1ff467000 -        0x1ff472fff libsystem_pthread.dylib arm64e  <1aa3a4b6f9e730568c8b4e4dd81312a4> /usr/lib/system/libsystem_pthread.dylib
       0x1ac7e0000 -        0x1ad129fff Foundation arm64e  <c431acb6fe043d28b6774de6e1c7d81f> /System/Library/Frameworks/Foundation.framework/Foundation
       0x1b3586000 -        0x1b394dfff CFNetwork arm64e  <edb0559fc996327f9b3a6616e316f24d> /System/Library/Frameworks/CFNetwork.framework/CFNetwork

EOF

Can you reproduce the bug?

Yes, always

Reproduction Steps

Use Realm in App extension (eg. WidgetKit extension).

Version

10.32.2

What SDK flavour are you using?

Local Database only

Are you using encryption?

No, not using encryption

Platform OS and version(s)

iOS 16.1

Build environment

Xcode version: 14.1
Dependency manager and version: SPM - realm-swift = 10.32.2, realm-core = 12.11.0

[tkafka]

I have noticed there are many issues trying to solve the crash when Realm is used in app group container, many of them closed.

If Realm doesn't plan to address this, I think that swift-realm should have a huge warning that it is unsuitable to use in app group containers (= any instances where data sharing between app and extension, or multiple apps of a same vendor is needed), as since iOS 14 everyone using Realm for such case will run into this, and this issue seems to be unfixable by user with a current design of Realm.

[tgoyne]

On iOS (and watchOS) we do not hold a file lock the entire time the Realm file is open. We acquire one when opening the file, but release it before the initializer returns. Write transactions also hold a lock for the duration of the write, but release it once the write is complete.

Opening a Realm, reading and/or writing to it, and then closing it within the span of a single ProcessInfo.performExpiringActivity() should work in an extension without caveats (other than the usual one of that if you take too long the OS will kill your process). Opening the Realm while your extension is in the foreground and keeping it open, but only interacting with it while your extension is in the foreground also works, but with caveats (it means that we need to keep the version of the data that your extension is using alive even while the extension is suspended, which results in the Realm file growing if the main application is writing to it).

Your extension transitioning to the background while inside a write transaction or while in the middle of opening a Realm will result in the OS killing the process, so if you are not using performExpiringActivity() it is important that you structure your use of threads to ensure that you can cancel any ongoing work in response to being told that you're moving to the background.

[observableobject]

Any updates on this issue or workarounds to resolve it? We are seeing similar behaviour in our application.

[tgoyne]

As I wrote above, we believe that what we are doing is valid provided that you are making proper use of the OS APIs for background tasks, and we do not hold file locks outside of write transactions. There isn't really generic advice we can give beyond what I wrote above and that you need to be sure to use the applicable background task APIs. Anything more specific would be very app-specific and require a lot more information about what exactly you're doing and what isn't working.