File protection issue

[vladasdrejeris]

Hello,

We have encountered a problem with Instabug iOS SDK when using file protection type URLFileProtection.complete.

We are developing a fintech app and our security guys requested us to use URLFileProtection.complete file protection class for all app files, that includes Instabug sqlite database too. And now we are getting quite a lot of crashes from Instabug, because it is unable to access database when app is locked. I am guessing that Instabug SDK is trying to finish some tasks in the background and if the device is locked, it crashes, because SDK cannot access the database file.

Could you help us with this issue?

Here is the stack trace of the crash that happens most offen:

Crashed: SQLQueue 0x11fe03620 for InstabugDataModel
0 libsqlite3.dylib 0x21f3cf6e0 sqlite3_finalize + 24896
1 libsqlite3.dylib 0x21f3cdf94 sqlite3_finalize + 18932
2 libsqlite3.dylib 0x21f3cd864 sqlite3_finalize + 17092
3 libsqlite3.dylib 0x21f3cbcc0 sqlite3_finalize + 10016
4 libsqlite3.dylib 0x21f3cb100 sqlite3_finalize + 7008
5 libsqlite3.dylib 0x21f3ea54c sqlite3_step + 12952
6 libsqlite3.dylib 0x21f3e7464 sqlite3_step + 432
7 CoreData 0x221c6ad50 _execute + 156
8 CoreData 0x221aca4f4 -[NSSQLiteConnection execute] + 2772
9 CoreData 0x221c81818 newFetchedRowsForFetchPlan_MT + 848
10 CoreData 0x221d20e28 _executeFetchRequest + 68
11 CoreData 0x221acd3e4 -[NSSQLFetchRequestContext executeRequestCore:] + 36
12 CoreData 0x221ac3984 -[NSSQLStoreRequestContext executeRequestUsingConnection:] + 252
13 CoreData 0x221c27870 __52-[NSSQLDefaultConnectionManager handleStoreRequest:]_block_invoke + 84
14 libdispatch.dylib 0x21e9e6484 _dispatch_client_callout + 16
15 libdispatch.dylib 0x21e9935d8 _dispatch_lane_barrier_sync_invoke_and_complete + 56
16 CoreData 0x221ac37ec -[NSSQLDefaultConnectionManager handleStoreRequest:] + 256
17 CoreData 0x221ac36a8 -[NSSQLCoreDispatchManager routeStoreRequest:] + 264
18 CoreData 0x221ac3464 -[NSSQLCore dispatchRequest:withRetries:] + 260
19 CoreData 0x221acd2d8 -[NSSQLCore processFetchRequest:inContext:] + 108
20 CoreData 0x221ac4348 -[NSSQLCore executeRequest:withContext:error:] + 496
21 CoreData 0x221c31de4 __65-[NSPersistentStoreCoordinator executeRequest:withContext:error:]_block_invoke + 1712
22 CoreData 0x221ac3254 -[NSPersistentStoreCoordinator _routeHeavyweightBlock:] + 240
23 CoreData 0x221ac3ee4 -[NSPersistentStoreCoordinator executeRequest:withContext:error:] + 860
24 CoreData 0x221acc47c -[NSManagedObjectContext executeFetchRequest:error:] + 900
25 CoreData 0x221ad070c -[NSManagedObjectContext(_NestedContextSupport) _parentObjectsForFetchRequest:inContext:error:] + 464
26 CoreData 0x221c09cec __82-[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:]_block_invoke + 580
27 CoreData 0x221c0bad0 internalBlockToNSManagedObjectContextPerform + 92
28 libdispatch.dylib 0x21e9e6484 _dispatch_client_callout + 16
29 libdispatch.dylib 0x21e9935d8 _dispatch_lane_barrier_sync_invoke_and_complete + 56
30 CoreData 0x221bffd70 _perform + 268
31 CoreData 0x221ad1154 -[NSManagedObjectContext(_NestedContextSupport) executeRequest:withContext:error:] + 172
32 CoreData 0x221acc47c -[NSManagedObjectContext executeFetchRequest:error:] + 900
33 Instabug 0x105b47244 __65-[IBGEventLogger loadSnapshotSynchronouslyWithCompletionHandler:]_block_invoke (IBGEventLogger.m:93)
34 Instabug 0x105c29e3c __26-[IBGDALService readSync:]_block_invoke (IBGDALService.m:108)
35 CoreData 0x221c02e88 developerSubmittedBlockToNSManagedObjectContextPerform + 156
36 libdispatch.dylib 0x21e9e6484 _dispatch_client_callout + 16
37 libdispatch.dylib 0x21e9935d8 _dispatch_lane_barrier_sync_invoke_and_complete + 56
38 CoreData 0x221aca8b8 -[NSManagedObjectContext performBlockAndWait:] + 304
39 Instabug 0x105c29cc8 -[IBGDALService readSync:] (IBGDALService.m:109)
40 Instabug 0x105b47188 -[IBGEventLogger loadSnapshotSynchronouslyWithCompletionHandler:] (IBGEventLogger.m:90)
41 Instabug 0x105ad6b28 -[IBGSnapshotManager surveyConditionsWithCompletionHandler:] (IBGSnapshotManager.m:263)
42 Instabug 0x105b05a1c -[InstabugSurveys pendingSurveyToShowWithCompletionHandler:] (InstabugSurveys.m:238)
43 Instabug 0x105b06348 -[InstabugSurveys surveysRequestDone:isSucess:withSurveys:pausedSurveysIDs:] (InstabugSurveys.m:359)
44 Instabug 0x105b4d474 -[IBGSurveysRequest networkDone:URLSessionDataTask:response:error:responseDictionary:] (IBGSurveysRequest.m:62)
45 Instabug 0x105c23020 __51-[IBGNetwork URLSession:task:didCompleteWithError:]_block_invoke_2 + 128
46 libdispatch.dylib 0x21e9e56c8 _dispatch_call_block_and_release + 24
47 libdispatch.dylib 0x21e9e6484 _dispatch_client_callout + 16
48 libdispatch.dylib 0x21e9929b4 _dispatch_main_queue_callback_4CF$VARIANT$mp + 1068
49 CoreFoundation 0x21ef3bdd0 CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE + 12
50 CoreFoundation 0x21ef36c98 __CFRunLoopRun + 1964
51 CoreFoundation 0x21ef361cc CFRunLoopRunSpecific + 436
52 GraphicsServices 0x2211ad584 GSEventRunModal + 100
53 UIKitCore 0x24c179054 UIApplicationMain + 212
54 Momo-prod 0x104e05174 main (AppDelegate.swift:23)
55 libdyld.dylib 0x21e9f6bb4 start + 4

[Kmohamed]

Yes, we are saving all Data in background tasks.

[vladasdrejeris]

Do you have any ideas how we could fix this?

[Kmohamed]

@vladasdrejeris I will send you a custom build without the background task in order to be sure that it will solve the issue ?

[Kmohamed]

@vladasdrejeris could you give this build a try ?

[vladasdrejeris]

Thanks, I will try it.

[vladasdrejeris]

Hi,

It is very hard to reproduce this crash manually, therefore we decided to release a new app version with your modified SDK build and check if it solves the issue. I will let you know the results of our test later on.

[Kmohamed]

@vladasdrejeris Thanks a lot 🙏

[vladasdrejeris]

Hi,

We are still seeing some crashes even with custom build. We are mostly getting these crashes:

lt.momo.app_issue_75_crash_cdf5fbef10144cd98def164818f7ef93_DNE_0_v2.txt
lt.momo.app_issue_102_crash_c3cae397502840e396c2251ecef47fb2_DNE_0_v2.txt

Maybe you have other ideas how to solve this issue?

[Kmohamed]

Let me check it and get back to you

[vladasdrejeris]

Thanks.

[Kmohamed]

@vladasdrejeris could you share with me how you protect your files? Also please send to me a shareable link of fabric stack trace?

[vladasdrejeris]

Hi @Kmohamed,

We are setting file protection level to all files in app directory:

func setProtectionClass(_ protectionClass: FileProtectionType, toFileAt url: URL) {
    do {
        try (url as NSURL).setResourceValue(URLFileProtection.complete,
                                                forKey: .fileProtectionKey)
     } catch {
        print(error)
    }
}

And a link to stack trace: http://crashes.to/s/89376de10eb

[Kmohamed]

Since you are using URLFileProtection.complete so this custom build should fix this issue.
Could you give it a try and tell me if it works 🙏.

[Vladas-Drejeris]

I will try it.

[Kmohamed]

@Vladas-Drejeris Thanks a lot

[Kmohamed]

@Vladas-Drejeris is everything works fine?

[vladasdrejeris]

Hi @Kmohamed ,

We are still seeing some crashes that happen in background, like these:
http://crashes.to/s/8f2d59cca99
http://crashes.to/s/b7b40e274ad

[vladasdrejeris]

I have looked into symbol map of the last SDK build that you provided and there are methods with associated with background tasks in IBGNetwork and IBGManagedObjectContextController classes, have you disabled all of them? I think the SDK is still using background task somewhere.

[Kmohamed]

Yes, we stop using background task in CoreData but it still exists in network.
We had a bug in the network and it’s no fixed.

[Kmohamed]

Anyway I will check those crashes ana get back to you

[vladasdrejeris]

Ok, thanks.

[Kmohamed]

@Vladas-Drejeris Sorry for being late, After some investigation, we prevented CoreData from writing data to disk in case of ApplicationProtectedDataDidBecomeAvailable but we did not prevent it in reads so the change in this custom build that CoreData will not read from disk in case of ApplicationProtectedDataDidBecomeAvailable also.
pod 'Instabug', :git => 'https://github.com/Instabug/Instabug-iOS.git', :branch => 'fix-coreData-protection'
I think this will work fine.
@vladasdrejeris

[Vladas-Drejeris]

Hi @Kmohamed ,

Good to hear, we will try the new SDK version.

[Kmohamed]

@Vladas-Drejeris is everything works fine with the custom build?

[Vladas-Drejeris]

Hi @Kmohamed ,

There is no way to reproduce these crashes consistently, therefore it takes some time to see the results of a fix. I will inform you when we see the results.

[Kmohamed]

@Vladas-Drejeris Perfect, I will be waiting for your updates.

[Vladas-Drejeris]

HI,

We are still seeing the crashes.

http://crashes.to/s/89376de10eb
http://crashes.to/s/4580c946fc2

[Kmohamed]

@Vladas-Drejeris are you sure that this crash from app with custom build?

[Vladas-Drejeris]

Yes, it is using lib from pod that you provided:
pod 'Instabug', :git => 'https://github.com/Instabug/Instabug-iOS.git', :branch => 'fix-coreData-protection'

[Kmohamed]

let me discuss what I am doing maybe you have some ideas about why this crash still occurred.
I listen on UIApplicationProtectedDataDidBecomeAvailable then I disable reads and writes and whenever UIApplicationProtectedDataWillBecomeUnavailable I allow reads and writes again.

[Vladas-Drejeris]

@Kmohamed , thanks for quick fix. We will try the new build.

[vladasdrejeris]

Hi @Kmohamed ,

Would it be possible to update the custom build to 8.1 version?

[Kmohamed]

@Vladas-Drejeris sure, working on it.

[vladasdrejeris]

Thanks

[Kmohamed]

@vladasdrejeris updated.

[vladasdrejeris]

Hello @Kmohamed ,

We are still seeing some crashes with the latest custom build. There are less crashes than before, but some still occurred.
Most of the crashes happen on [IBGDALService writeASync:completionBlock:].
I have attached couple of logs for the most common crashes.

lt.momo.app_issue_crash_fb34ad25fc3c422aa18c7968fbb7ae2e_DNE_0_v2.txt
lt.momo.app_issue_crash_9e66c56af5224f788bd7360a66beac0c_DNE_1_v2.txt

Also, could you update the custom build to the latest SDK version?

[Kmohamed]

Hello @vladasdrejeris,
The custom build is updated to the latest SDK version. Relate to the crash let me check and get back to you?

[vladasdrejeris]

Great. Thanks.

[vladasdrejeris]

Hi @Kmohamed ,

Are there any news?

[Kmohamed]

@vladasdrejeris I cannot identify exactly the root cause. But in the next release, we added some defensive code to our CoreData layer. Also, can you send me any other crashes you got and the number of occurrences? 🙏
btw the next release will be within three or four days?

[vladasdrejeris]

Some more crashes that happened in the past 30 days:

Crash 1.txt - 167 times
Crash 2.txt - 66 times
Crash 3.txt - 11 times
Crash 4.txt - 9 times
Crash 5.txt - 7 times
Crash 6.txt - 2 times
Crash 7.txt - 1 time

[Kmohamed]

are these crashes from the custom build?

[Kmohamed]

I mean the from the custom build with file protection fix.

[vladasdrejeris]

Yes, they are from the custom build.

[Kmohamed]

@vladasdrejeris I think this may be caused by a race condition between protecting the coreData file when the app is locked and our code preventing access to coreData. I am not sure
In the next release, we removed background tasks with core data and I think this should fix the issue.

[vladasdrejeris]

Ok then, we will check the new release when it is available.
We should try the main SDK not the custom?

[Kmohamed]

We will release it next Sunday morning. Also, we can provide custom build if this will not work with you.

[vladasdrejeris]

Hi @Kmohamed ,

Should we try v8.2.3 SDK?

[Kmohamed]

Yes please, and tell me if everything works fine

[vladasdrejeris]

Thanks, we will try it.

[vladasdrejeris]

Hello,

We are still seeing the crashes while using v.8.3 SDK.
@Kmohamed , could you take a look?

lt.momo.app_issue_crash_f239f9e0c4b34161acd6ec3305811e42_DNE_4_v2.txt
lt.momo.app_issue_crash_3c017a9ce3184eff908ba6e44f750497_DNE_0_v2.txt
lt.momo.app_issue_crash_4821cd80180a48faafc010eab65e7dbf_DNE_0_v2.txt

[yousefhamza]

Hi @vladasdrejeris

Can you tell us whether these crashes have been on foreground or background?

[vladasdrejeris]

Hi @yousefhamza ,

They are happening in background.

[yousefhamza]

Thanks, @vladasdrejeris

I will get back to you on this shortly.

[yousefhamza]

Hi @vladasdrejeris

I added a new custom build for you, this custom build add a data protection flag our coredata's persistent store.

pod 'Instabug', :git => 'https://github.com/Instabug/Instabug-iOS.git', :branch => 'fix-coreData-protection'

[vladasdrejeris]

Thanks, we will try it and let you know if it fixed the problem.

[Kmohamed]

Hello @vladasdrejeris,
I hope you are doing well. I am curious to know if the fix works fine with you.

[Kmohamed]

Hello @vladasdrejeris it's been a long time since your last message. I hope that everything works fine with you now.

[Kmohamed]

@vladasdrejeris please feel free to open this issue again if you face this issue again