iOS 14 + Xcode 12 (Beta 3, Beta 4, Beta 5 & Beta 6): When Realm is stored in a shared app group container, backgrounding the app triggers: Message from debugger: Terminated due to signal 9

[kunalsood]

Goals

Have my app continue to run normally, when backgrounded, with Realm stored in a shared app group container.

Expected Results

Same as "Goals" [above].

Actual Results

When, I run the app on an actual device, and swipe up while my app is in the foreground to send it to the background, the app quits with the following message in Xcode:-

Message from debugger: Terminated due to signal 9

Additionally: This only happens if I'm holding a reference to a live RLMResults or a notification token (those are the two things I have tested this with so far).

Interestingly, when I run the project with the default realm configurations (i.e. Realm is stored in the documents directory), this issue disappears.

It might be worth mentioning: It appears that the crash report produced on device for this issue when debugger isn't attached shows: Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc, which would indicate that the application is being terminated by the OS because it held on to a file-lock/database-lock during suspension. (Could be un-related, though there is nothing other than Realm in my project that would hold on to a file-lock/database-lock ever).

Steps for others to Reproduce

Build and run the sample project using Xcode 12 (Beta 3, Beta 4, Beta 5, or Beta 6) on an actual device running iOS 14 (Beta 3, Beta 4, Beta 5, or Beta 6).
(I have tested this on an iPhone & an iPad)

Code Sample

Here is a (quick & dirty) sample project: https://github.com/kunalsood/KSRealmTerminationSample

Most of the relevant code in the project is in: ViewController.m file & one realm model object file KSSampleObject.h.

Version of Realm and Tooling

Realm framework version: 5.3.2 (pre-built dynamic framework)
UPDATE: Sample project has now been updated to include Realm using SPM.

Realm Object Server version: N/A

Xcode version: 12.0 (Beta 3, Beta 4, Beta 5 & Beta 6)

iOS/OSX version: iOS 14 (Beta 3, Beta 4, Beta 5 & Beta 6)

Dependency manager + version: N/A

[meteochu]

+1 on having identical issues. Group Container, Xcode 12b3 + iOS14b3, and crashing when backgrounded.

I'm using the latest RealmSwift (5.3.2.) with Swift PM.

[oncezw]

+1 same issues in + iOS14b3, and crashing when backgrounded.

[bguidolim]

There is a bug report filed for this issue: https://developer.apple.com/forums/thread/655225
I don't think it's a Realm bug.

I'm facing the same problem, it's about locked files in the shared folder.

[SquaredTiki]

+1, author of that thread on the Apple Dev Forums.

Would recommend folks dupe the existing feedback reports (FB8128103 & FB8116961) if they can. May well bring it to Apple's attention more quickly. Especially if you have a sample project repro'ing as my report didn't include one.

Title
iOS 14 Beta 3 triggers consistent 0xdead10cc terminations impacting binary compatibility

Component
UIKit

Type
Application Crash

Description
DESCRIPTION OF PROBLEM
We have observed that as of beta 3 our app now consistently & immediately crashes as soon as it is suspended (i.e. the user 'closes' the app). The crash log has an exception reason of 0xdead10cc indicating that the crash is due to the app holding on to a file lock or sqlite database lock during suspension.

This crash occurs on all recent versions of our app but only on beta 3 and does not occur at all before beta 3. We can consistently reproduce the issue on iOS 14 beta 3 (18A5332f) but are unable to do so on previous betas of iOS 14 or any version of iOS 13.

The console log narrows down the file triggering the crash on suspension:

[application<…>:3879] Terminating with context: <RBSTerminateContext| domain:15 code:0xDEAD10CC explanation:[application<…>:3879] was suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/A66EB78A-2BBC-49D4-BDEA-6A2AF7E8A5A6/default.realm.lock
not in allowed directories:
/var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1
/var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1/tmp reportType:CrashLog maxTerminationResistance:Interactive>

This points to Realm's default.realm.lock being the locked file triggering the crash, suggesting this file is not permitted as it sits within the App Group container as opposed to the app's own container (which I presume is the first 'allowed directory').

Again, whilst this explains the cause of the crash on beta 3 it doesn't explain why the crash has only just begun occurring on this build of iOS 14 (beta 3, 18A5332f). This suggests to me that there may have been a system level change to cause this and potentially break binary compatibility with existing versions of apps (including others than our own that use Realm within an App Group).

STEPS TO REPRODUCE

• Launch an app that uses Realm with App Groups (e.g. <"Your App on the App Store" or "Sample Project Attached">)
• Suspend the application
• Runningboard terminates the app
• Console log indicates a 0xdead10cc termination reason due to default.realm.lock still being locked

PLATFORM AND VERSION
iOS 14.0 Beta 3 (18A5332f)
Devices: All

NOTES
Also raised on the developer forums and discussed with Quinn “The Eskimo!” here: https://developer.apple.com/forums/thread/655225.

[kunalsood]

@SquaredTiki Thanks for including details of for Feedback to Apple. I'm in the process of submitting a dupe with a sample project. Will post the feedback number on your Apple Dev Forums thread when done.

[n1kitus]

+1 identical issues in iOS 14b3. The description of the problem perfectly fits our case.

[zacwest]

This crash continues in 14.0 b4.

[indirect]

I’ve heard this kind of crash can often be fixed with task assertions, are those assertions getting used here?

https://developer.apple.com/documentation/uikit/uiapplication/1623031-beginbackgroundtask

[gongzhang]

I believe this is an iOS bug, since:

1. only reproduce on physical device running iOS 14b3/b4
2. when crash with a debugger attached, Xcode just ends the debug session silently without showing any error messages. (except the termination signal message)

[gongzhang]

I’ve heard this kind of crash can often be fixed with task assertions, are those assertions getting used here?

https://developer.apple.com/documentation/uikit/uiapplication/1623031-beginbackgroundtask

I am also trying to walk around this issue. But it's difficult when you have to save the db in app group container. 😟

[kunalsood]

I’ve heard this kind of crash can often be fixed with task assertions

@indirect That would be the case if we were crashing while in an active a write transaction. We are not. The app crashes even if I'm just holding on to an instance of a live RLMResults, RLMArray, RLMObject, RLMRealm, or RLMNotificationToken. I do not think it makes sense to wrap any of these in task assertions.

What you have heard is probably applicable for versions of iOS that came before iOS 14b3.
Prior to iOS 14b3, this issue used to occur in cases that can broadly be divided in two:-

1. If you were doing a long write transaction when the app entered background, and have either not requested a task assertion, or the additional time allotted by the system to your app ended before you ended your write transaction.
2. You were doing a write transaction from an extension without using performExpiringActivityWithReason:usingBlock: method of NSProcessInfo, and the extension's process was killed by the system, or, performExpiringActivityWithReason:usingBlock: failed to acquire a task assertion (it fails more often than not in my experience), or, the additional time allotted by the system to your extension's process ended before you ended your write transaction.

are those assertions getting used here?

I use them where they makes sense.

This is either a bug introduced in iOS 14b3, or (Apple has decided that) this is the new expected behavior. I'm inclined to believe that it's the former, because as has already been said, this only started happening in iOS14b3, and when this happens with the debugger attached, Xcode just ends the debug session silently without showing any error messages. The latter would make use of Realm unfeasible for apps with extensions that need access to the database.

[meteochu]

Just checking in, has anyone gotten updates from Apple about their submitted feedback and if we know this is a new system behaviour or just a system regression?

[kunalsood]

Just checking in, has anyone gotten updates from Apple about their submitted feedback and if we know this is a new system behaviour or just a system regression?

That's a negative from me on both. Nothing yet for my feedback, still marked as Open.

[gongzhang]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

[owenzhao]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

The question is not why Ream puts a lock file there. The question is why Apple doesn't allow a lock file in App Group folder. Is there any harm?

[meteochu]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

The question is not why Ream puts a lock file there. The question is why Apple doesn't allow a lock file in App Group folder. Is there any harm?

It is possible that this is the new expected behaviour to prevent multiple apps for writing to the same file by checking for file locks (since App group containers can be accessed by more than one process at a time). However, I'd like to believe that it's a regression since it is not being blocked by linked-on checks for iOS 14 and is instead affecting current apps.

[zhangcx627]

+1 same issues in + iOS14b3, and crashing when backgrounded.

[SquaredTiki]

No updates on my bug report to Apple but I can see it says that there are less than 10 'recent' dupes, would recommend folks file a duplicate bug report if they haven't already to help bring increased attention (example/template here).

[gongzhang]

Does anyone know which exact line of code in Realm trigger this bug?

I think it's better to reproduce the bug in a clean and simple demo project without importing the entire Realm framework. I believe it will help Apple to identify the bug.

[zacwest]

This crash continues in 14.0 b5. Probably time to consider migrating off Realm for iOS 14.

[owenzhao]

This crash continues in 14.0 b5. Probably time to consider migrating off Realm for iOS 14.

There are other three ways:

1. Realm adds a new config to put lock file in the app's container, separately from the database file.
2. Realm releases the lock file every time the Realm.write state is finished. I had a test by putting a file named "test.lock" in App Group's container using FileMananger. The app didn't crash as Realm did.
3. Using iCloud syncs the realm database between app and extensions, avoid of using App Group with Realm.

[tgoyne]

The lock file is placed in the app container.

This appears to be Apple's "fix" for the problem where switching between apps sharing a Realm file in a container could deadlock if done during a write transaction because the now-backgrounded app would hold the write transaction forever. Simply killing the app when that happens isn't exactly an ideal solution, but it does ensure things keep working. The problem appears to be that it's being overly aggressive and is also killing the app if a shared lock is held. There's nothing inherently wrong with a shared lock continuing to be held by a suspended process.

[kunalsood]

@tgoyne Say if Apple has decided that this is the new expected behaviour. Is there anything you (or I) can do to work around this?

[tgoyne]

There probably isn't anything easy. I suspect Apple's intended answer is "just close the file when you get the transition to background notification", which is very difficult when multiple threads are involved. The big important thing that the shared lock is doing is letting us clean up after crashes rather than forever holding onto data which was only needed by a process that is no longer running, so we could try to come up with some other approach to doing that.

[owenzhao]

Here is my work around. My app won't crash any longer. If you use this, take your own risk.

func applicationWillResignActive(_ application: UIApplication) {
    // Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
    // Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game.
    let fm = FileManager.default
    let url = lockfile()
    
    if fm.fileExists(atPath: url.path) {
        try? fm.removeItem(at: url)
    }
}

func applicationDidBecomeActive(_ application: UIApplication) {
    // Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
    
    if !FileManager.default.fileExists(atPath: lockfile().path){
        let _ = try! Realm(configuration: Realm.sharedConfiguration)
    }
}

private func lockfile() -> URL {
    let fm = FileManager.default
    let url = fm.containerURL(forSecurityApplicationGroupIdentifier: MyDefaults.sharedSuitName)
    let lockfileName = "default.realm.lock"
    let lockfileURL = URL(fileURLWithPath: lockfileName, isDirectory: false, relativeTo: url)
    
    return lockfileURL
}

Just remove the lock file when app no longer active and recreate the lock file when it become active.

[kunalsood]

@tgoyne Thanks for your response.

@owenzhao The lock file is there for a good reason, simply deleting it will probably cause more problems (probably even data corruption?) than it will solve.

[ianpward]

😂

[SquaredTiki]

Since the changes to resolve this issue we've started seeing our app freeze (sometimes causing the whole of iOS to freeze) when foregrounded on iOS 10, 11, 12. This was briefly mentioned by @thodang888 in #6722 and I've raised a new issue #6749 for this.

[gongzhang]

I'm sad to report that on iOS 15.0 beta 1, I'm once again getting the 0xdead10cc crash when the app opens a new Realm in the background and performs a write operation.

[SquaredTiki]

For what it's worth, I haven't yet been able to reproduce this on iOS 15 beta 1.

[gongzhang]

For what it's worth, I haven't yet been able to reproduce this on iOS 15 beta 1.

It's not 100% reproduced. And I cannot reproduce it on my own devices no matter in dev or release mode.

But I indeed received a lot of crash reports from users who are using iOS 15. ☹️

[tgoyne]

Could it be related to SDK version? I assume any users hitting it would be using versions built against iOS 14 SDKs while the obvious way to test locally would involve building against the iOS 15 SDK. Would be pretty weird if they started enforcing no file locks but only for existing apps, though...

[gongzhang]

The article below discussed a similar issue about sharing a SQLite database in app group. The widget extension will be killed by the system watchdog from time to time, due to 0xdead10cc (hold a file lock after process suspension). Since WidgetKit has no APIs related to process lifecycle management, there is no simple way to work around the problem as long as file locks are used.

https://inessential.com/2020/02/13/how_we_fixed_the_dreaded_0xdead10cc_cras

From my collected analysis data, it is clear that iOS 15 has improved the detection strength for 0xdead10cc. I'm not quite sure how I should address this issue at this point.

[schmidan]

We are seeing this as well (sdk 14 build, iOS 15 beta3).

Does anyone know of a workaround or do we have to move realm out of the shared group container?
Or does the Realm Team work on a fix?

(we are on 10.7.7 atm because of #7344)

[gongzhang]

🙋‍♂️ Update:

I finally fixed the 0xdead10cc crash. And I must admit that the check for file locks in iOS and watchOS are correct, while the detection in the recent 15.0 betas seems to be more comprehensive. It is also possible that the background execution time has been shortened. Anyway, this exposes potential issues in my app.

It's totally ok to store the database in a shared container. The real problem is that the file lock is not released indeed at the moment the process suspends. In other words, the realm.write { ... } transaction does not complete when current process is going to be suspended.

I made the following changes to completely fix the 0xdead10cc crash, which you can refer to.

1. Keep write transactions as short as possible. Avoid putting irrelevant code in a transaction.

// before:
realm.write {
  heavyTaskWithDataModifications()
}

// after:
let diff = heavyTaskThatGenerateDataDiff()
realm.write {
  diff.apply() // much shorter transaction
}

2. Avoid performing irrelevant tasks when the app is wake up in background.

Modern iOS has many reasons to wake up your app in the background and then hang it again quickly, such as handling remote notifications, responding to WatchConnectivity, background HealthKit updates, background refresh tasks, etc. You need to examine each of these situations carefully. In particular, my code is based on the concept of reactive programming, so it's easy to accidentally execute unnecessary code in the background.

3. No secret sauce

I didn't use ProcessInfo.beginActivity(...).

[schmidan]

For what it is worth:
The problem "went away". We are still on 10.7.7 but building with XCode 12.5.1 stops the app from crashing on iOS 15 devices.

[sync-by-unito]

➤ Jason Flax commented:

We're going to close this out because the original user's issue was fixed. However, we are going to open up a new issue to document how Realm works with App Groups and in Background Fetch.

[gongzhang]

Update: My previous fixes stopped working since iOS 15.4.

🙋‍♂️ Update:

I finally fixed the 0xdead10cc crash. And I must admit that the check for file locks in iOS and watchOS are correct, while the detection in the recent 15.0 betas seems to be more comprehensive. It is also possible that the background execution time has been shortened. Anyway, this exposes potential issues in my app.

It's totally ok to store the database in a shared container. The real problem is that the file lock is not released indeed at the moment the process suspends. In other words, the realm.write { ... } transaction does not complete when current process is going to be suspended.

I made the following changes to completely fix the 0xdead10cc crash, which you can refer to.

1. Keep write transactions as short as possible. Avoid putting irrelevant code in a transaction.

// before:
realm.write {
  heavyTaskWithDataModifications()
}

// after:
let diff = heavyTaskThatGenerateDataDiff()
realm.write {
  diff.apply() // much shorter transaction
}

2. Avoid performing irrelevant tasks when the app is wake up in background.

Modern iOS has many reasons to wake up your app in the background and then hang it again quickly, such as handling remote notifications, responding to WatchConnectivity, background HealthKit updates, background refresh tasks, etc. You need to examine each of these situations carefully. In particular, my code is based on the concept of reactive programming, so it's easy to accidentally execute unnecessary code in the background.

3. No secret sauce

I didn't use ProcessInfo.beginActivity(...).

[IAmMichellis]

In case anyone comes across this again: I had a relevant related issue with mysterious SIGKILLS, idle main threads, and suspicious background threads accessing a RLMRealm in a shared group container. Info / solution here #7466 (comment)