Assertion failed in EncryptedFileMapping::get_local_index_of_address

[sync-by-unito]

How frequently does the bug occur?

Sometimes

Description

Crash, that asked to be reported.
We introduced encryption to one of our realm files.
A few users experience this crash.

Stacktrace & log output

OS Version: iOS 16.5.1 (20F75)
Report Version: 104

Exception Type: EXC_CRASH (SIGABRT)
Crashed Thread: 0

Application Specific Information:
(local_ndx, m_page_state.size(), size_t(addr), size_t(m_addr), m_page_shift) > ardServices                  0x00000001c43ff188 11A9F887-776B-31E9-8557-8A24A587DD1B + 70024

Thread 0 Crashed:
0   libsystem_kernel.dylib          0x3b324d558         __pthread_kill
1   libsystem_pthread.dylib         0x3f4edf114         pthread_kill
2   libsystem_c.dylib               0x343e01174         abort
3   app2                          0x204821674         please_report_this_issue_in_github_realm_realm_core_v_13_15_0 (terminate.cpp:65)
4   app2                          0x2048218b4         realm::util::terminate_internal (terminate.cpp:143)
5   app2                          0x204821794         realm::util::terminate_with_info (terminate.cpp:162)
6   app2                          0x2045e8144         realm::util::terminate_with_info<T> (terminate.hpp:52)
7   app2                          0x20480d114         realm::util::EncryptedFileMapping::read_barrier
8   app2                          0x2045944a8         realm::util::do_encryption_read_barrier (file_mapper.hpp:144)
9   app2                          0x204596d40         [inlined] realm::util::encryption_read_barrier<T> (file_mapper.hpp:176)
10  app2                          0x204596d40         realm::SlabAlloc::attach_file (alloc_slab.cpp:825)
11  app2                          0x2045c95d0         realm::DB::open (db.cpp:1159)
12  app2                          0x2045cfebc         realm::DB::create (db.cpp:2766)
13  app2                          0x20465e804         realm::_impl::RealmCoordinator::open_db (realm_coordinator.cpp:454)
14  app2                          0x20465f3e4         realm::_impl::RealmCoordinator::do_get_realm (realm_coordinator.cpp:324)
15  app2                          0x20465f304         realm::_impl::RealmCoordinator::get_realm (realm_coordinator.cpp:274)
16  app2                          0x2046bebbc         realm::Realm::get_shared_realm (shared_realm.cpp:161)
17  app2                          0x2044b4f0c         +[RLMRealm realmWithConfiguration:confinedTo:error:] (RLMRealm.mm:379)
18  app2                          0x2044b4aa8         +[RLMRealm realmWithConfiguration:queue:error:] (RLMRealm.mm:344)
19  app2                          0x204553c3c         RLMRealm.__allocating_init
20  app2                          0x204554810         Realm.init (Realm.swift:95)
21  app2                          0x2042c4f14         AppDelegate.configureAndMigrateFoodBaseRealm (AppDelegate.swift:445)
22  app2                          0x2042c60c4         [inlined] AppDelegate.configureAndMigrateFoodBaseRealm
23  app2                          0x2042c60c4         AppDelegate.application (AppDelegate.swift:54)
24  app2                          0x2042c2560         [inlined] AppDelegate.application
25  app2                          0x2042c2560         AppDelegate.application (<compiler-generated>:40)
26  UIKitCore                       0x3397f17e0         -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
27  UIKitCore                       0x3397f0f18         -[UIApplication _callInitializationDelegatesWithActions:forCanvas:payload:fromOriginatingProcess:]
28  UIKitCore                       0x3397efef4         -[UIApplication _runWithMainScene:transitionContext:completion:]
29  UIKitCore                       0x3397efb40         -[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:]
30  UIKitCore                       0x339537df0         _UIScenePerformActionsWithLifecycleActionMask
31  UIKitCore                       0x339875fdc         __101-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]_block_invoke
32  UIKitCore                       0x3397283e8         -[_UISceneLifecycleMultiplexer _performBlock:withApplicationOfDeactivationReasons:fromReasons:]
33  UIKitCore                       0x339728210         -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]
34  UIKitCore                       0x339727dc0         -[_UISceneLifecycleMultiplexer uiScene:transitionedFromState:withTransitionContext:]
35  UIKitCore                       0x339727c8c         __186-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]_block_invoke
36  UIKitCore                       0x339e14f4c         +[BSAnimationSettings(UIKit) tryAnimatingWithSettings:fromCurrentState:actions:completion:]
37  UIKitCore                       0x339eb3498         _UISceneSettingsDiffActionPerformChangesWithTransitionContextAndCompletion
38  UIKitCore                       0x3395d53fc         -[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]
39  UIKitCore                       0x339a468dc         __64-[UIScene scene:didUpdateWithDiff:transitionContext:completion:]_block_invoke.223
40  UIKitCore                       0x3396a63ec         -[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:]
41  UIKitCore                       0x3396a625c         -[UIScene scene:didUpdateWithDiff:transitionContext:completion:]
42  UIKitCore                       0x3396a57b4         -[UIApplication workspace:didCreateScene:withTransitionContext:completion:]
43  UIKitCore                       0x3396a5540         -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:]
44  FrontBoardServices              0x36145f21c         -[FBSScene _callOutQueue_agent_didCreateWithTransitionContext:completion:]
45  FrontBoardServices              0x36149efdc         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke.80
46  FrontBoardServices              0x361463058         -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:]
47  FrontBoardServices              0x36149ec34         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke
48  libdispatch.dylib               0x343d59ea8         _dispatch_client_callout
49  libdispatch.dylib               0x343d5d918         _dispatch_block_invoke_direct
50  FrontBoardServices              0x36146d184         __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__
51  FrontBoardServices              0x36146cd20         -[FBSSerialQueue _targetQueue_performNextIfPossible]
52  FrontBoardServices              0x36146f5cc         -[FBSSerialQueue _performNextFromRunLoopSource]
53  CoreFoundation                  0x335491124         __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
54  CoreFoundation                  0x33549d7b0         __CFRunLoopDoSource0
55  CoreFoundation                  0x335422644         __CFRunLoopDoSources0
56  CoreFoundation                  0x3354380d0         __CFRunLoopRun
57  CoreFoundation                  0x33543d3e8         CFRunLoopRunSpecific
58  GraphicsServices                0x3abdf5358         GSEventRunModal
59  UIKitCore                       0x3398396e4         -[UIApplication _run]
60  UIKitCore                       0x339839348         UIApplicationMain
61  app2                          0x20410a720         main (AppDelegate.swift:21)
62  <unknown>                       0x1cd8f6dec         <redacted>

Thread 0 Crashed:
0   libsystem_kernel.dylib          0x3b324d558         __pthread_kill
1   libsystem_pthread.dylib         0x3f4edf114         pthread_kill
2   libsystem_c.dylib               0x343e01174         abort
3   app2                          0x204821674         please_report_this_issue_in_github_realm_realm_core_v_13_15_0 (terminate.cpp:65)
4   app2                          0x2048218b4         realm::util::terminate_internal (terminate.cpp:143)
5   app2                          0x204821794         realm::util::terminate_with_info (terminate.cpp:162)
6   app2                          0x2045e8144         realm::util::terminate_with_info<T> (terminate.hpp:52)
7   app2                          0x20480d114         realm::util::EncryptedFileMapping::read_barrier
8   app2                          0x2045944a8         realm::util::do_encryption_read_barrier (file_mapper.hpp:144)
9   app2                          0x204596d40         [inlined] realm::util::encryption_read_barrier<T> (file_mapper.hpp:176)
10  app2                          0x204596d40         realm::SlabAlloc::attach_file (alloc_slab.cpp:825)
11  app2                          0x2045c95d0         realm::DB::open (db.cpp:1159)
12  app2                          0x2045cfebc         realm::DB::create (db.cpp:2766)
13  app2                          0x20465e804         realm::_impl::RealmCoordinator::open_db (realm_coordinator.cpp:454)
14  app2                          0x20465f3e4         realm::_impl::RealmCoordinator::do_get_realm (realm_coordinator.cpp:324)
15  app2                          0x20465f304         realm::_impl::RealmCoordinator::get_realm (realm_coordinator.cpp:274)
16  app2                          0x2046bebbc         realm::Realm::get_shared_realm (shared_realm.cpp:161)
17  app2                          0x2044b4f0c         +[RLMRealm realmWithConfiguration:confinedTo:error:] (RLMRealm.mm:379)
18  app2                          0x2044b4aa8         +[RLMRealm realmWithConfiguration:queue:error:] (RLMRealm.mm:344)
19  app2                          0x204553c3c         RLMRealm.__allocating_init
20  app2                          0x204554810         Realm.init (Realm.swift:95)
21  app2                          0x2042c4f14         AppDelegate.configureAndMigrateFoodBaseRealm (AppDelegate.swift:445)
22  app2                          0x2042c60c4         [inlined] AppDelegate.configureAndMigrateFoodBaseRealm
23  app2                          0x2042c60c4         AppDelegate.application (AppDelegate.swift:54)
24  app2                          0x2042c2560         [inlined] AppDelegate.application
25  app2                          0x2042c2560         AppDelegate.application (<compiler-generated>:40)
26  UIKitCore                       0x3397f17e0         -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
27  UIKitCore                       0x3397f0f18         -[UIApplication _callInitializationDelegatesWithActions:forCanvas:payload:fromOriginatingProcess:]
28  UIKitCore                       0x3397efef4         -[UIApplication _runWithMainScene:transitionContext:completion:]
29  UIKitCore                       0x3397efb40         -[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:]
30  UIKitCore                       0x339537df0         _UIScenePerformActionsWithLifecycleActionMask
31  UIKitCore                       0x339875fdc         __101-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]_block_invoke
32  UIKitCore                       0x3397283e8         -[_UISceneLifecycleMultiplexer _performBlock:withApplicationOfDeactivationReasons:fromReasons:]
33  UIKitCore                       0x339728210         -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]
34  UIKitCore                       0x339727dc0         -[_UISceneLifecycleMultiplexer uiScene:transitionedFromState:withTransitionContext:]
35  UIKitCore                       0x339727c8c         __186-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]_block_invoke
36  UIKitCore                       0x339e14f4c         +[BSAnimationSettings(UIKit) tryAnimatingWithSettings:fromCurrentState:actions:completion:]
37  UIKitCore                       0x339eb3498         _UISceneSettingsDiffActionPerformChangesWithTransitionContextAndCompletion
38  UIKitCore                       0x3395d53fc         -[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]
39  UIKitCore                       0x339a468dc         __64-[UIScene scene:didUpdateWithDiff:transitionContext:completion:]_block_invoke.223
40  UIKitCore                       0x3396a63ec         -[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:]
41  UIKitCore                       0x3396a625c         -[UIScene scene:didUpdateWithDiff:transitionContext:completion:]
42  UIKitCore                       0x3396a57b4         -[UIApplication workspace:didCreateScene:withTransitionContext:completion:]
43  UIKitCore                       0x3396a5540         -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:]
44  FrontBoardServices              0x36145f21c         -[FBSScene _callOutQueue_agent_didCreateWithTransitionContext:completion:]
45  FrontBoardServices              0x36149efdc         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke.80
46  FrontBoardServices              0x361463058         -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:]
47  FrontBoardServices              0x36149ec34         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke
48  libdispatch.dylib               0x343d59ea8         _dispatch_client_callout
49  libdispatch.dylib               0x343d5d918         _dispatch_block_invoke_direct
50  FrontBoardServices              0x36146d184         __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__
51  FrontBoardServices              0x36146cd20         -[FBSSerialQueue _targetQueue_performNextIfPossible]
52  FrontBoardServices              0x36146f5cc         -[FBSSerialQueue _performNextFromRunLoopSource]
53  CoreFoundation                  0x335491124         __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
54  CoreFoundation                  0x33549d7b0         __CFRunLoopDoSource0
55  CoreFoundation                  0x335422644         __CFRunLoopDoSources0
56  CoreFoundation                  0x3354380d0         __CFRunLoopRun
57  CoreFoundation                  0x33543d3e8         CFRunLoopRunSpecific
58  GraphicsServices                0x3abdf5358         GSEventRunModal
59  UIKitCore                       0x3398396e4         -[UIApplication _run]
60  UIKitCore                       0x339839348         UIApplicationMain
61  app2                          0x20410a720         main (AppDelegate.swift:21)
62  <unknown>                       0x1cd8f6dec         <redacted>

Can you reproduce the bug?

No

Reproduction Steps

No response

Version

10.40.2

What Atlas Services are you using?

Local Database only

Are you using encryption?

Yes

Platform OS and version(s)

iOS 16.5.1

Build environment

Xcode version: 14.3.1
Dependency manager and version: SPM

[sync-by-unito]

➤ On 2023-06-27, bb-git commented:

Ok, we went through the same crash from a previous release: please_report_this_issue_in_github_realm_realm_core_v_13_10_1
It seems, that the crash so far happened only for iOS 16.x.x users and only for existing users upgrading to the latest version.

[ironage]

@bb-git thanks for reporting this. Do you know what the previous version was that users were updating from? Do you know if the Realm is being accessed from multiple processes? An assertion failure like this one will be difficult to track unless we can get some more information or a way to reproduce it so anything of relevance you can add will help us!

Note that if the previous version of Realm was an old enough version, this may have been fixed by #5383

[bb-git]

We used version 10.36.0 before. Yes, we access Realm from multiple processes. Looking at where the crash occurs, this is the first time the database is accessed and this happens on the main thread. And it seems, if it's crashing, then it crashes on every launch.

I think I saw the issue mentioned in the release notes, we introduced encryption just a week ago and the the Realm version we used for this release was 10.39.1. For users coming from an older version we create an encrypted copy. If the decryption fails on the first init, we delete the realm file and the data gets downloaded again from the backend.

[ironage]

@bb-git thanks for that info. When you say you "introduced encryption" how did you migrate existing unencrypted data to the encrypted file? Given that you are using multiple processes, you may need to take steps to prevent concurrent access to the file if you are using a file system copy/move (or one of our APIs which use them).

Internal note: Swift version 10.36.0 (2023-02-15) is using core 13.4.1. Swift 10.39.1 (2023-05-05) is core version 13.10.1.

[bb-git]

As mentioned, we make a copy of the database file with encryption. This all happens before accessing the file on multiple threads.

From our investigation: it seems the crash happens in the background. The only explanation we have so far is, that HealthKit launches the app after new data came in. Even more confusing for us, the access to Realm where the crash happens is wrapped in a do catch block.

This crash happens only to a few of our users, so it can't be a general issue with the background calls from HealthKit. Our theory is, that it might be an issue, if the app is updated by the system and then launched by HealthKit, maybe in combination, that the device wasn't unlocked for the first time. This would more or less explain the number and frequency. We see around 10 crashes per affected user.
It must be a quite special constellation.

Do you have any ideas what to look at next, especially what can cause an error like this?
Please correct us, if we're wrong, but (all in background) failing to make the encrypted copy, or not enough disk space or failing to access the directory / file or failing to create the file shouldn't lead to an exception like the one we see here.

[ironage]

@bb-git thanks for that info.

Even more confusing for us, the access to Realm where the crash happens is wrapped in a do catch block.

This is an assertion failure, not an exception that can be caught unfortunately.
This type of assertion could be caused by a process concurrently accessing the Realm while it hasn't finished the filesystem copy operation yet. It could also be caused if the copy operation failed part way through due to the disk being full, leaving the Realm partially written. An incomplete copy would cause an exception that you could catch, but leave the file in a corrupted/incomplete state that may cause this assertion on the next access. If this is actually what is happening, then you could consider catching the original "out of disk space" exception, alerting the user, and deleting the partially written file to leave everything in a consistent state.

[bb-git]

No, that can't be. We already catch and record the error during the copy process and it happened only once and not even for the same device.

Also this is the first access of the realm file it happens at the very early stage in applicationDidFinishLaunching. All runners are initialized at a later stage.
The copy happens on the main thread, before accessing the file for the first time.

[github-actions]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[ironage]

This could be fixed by #7162 assuming multiple encrypted Realms are being opened on an external exFAT drive.

[bb-git]

We are an iOS only app, so I guess not external exFAT drives. Also we use only one encrypted Realm, but this one we access on multiple threads.