iOS 13 Crash - [FIRInstanceIDKeychain itemWithQuery:]

[milo-]

• Xcode version: 10.2.1
• Firebase SDK version: 6.3.0
• Firebase Component: Core (Auth, Core, Database, Firestore, Messaging, Storage, etc)
• Component version: 6.3.0

Describe the problem

We released an update yesterday, going from Firebase 6.0.0 to firebase 6.3.0. We've been getting reports of the app crashing on startup for users with iOS 13 beta (17A5508m).

We've not been able to reproduce this ourselves, but we've had at least 5 crashes in the last 12 hours caused by this. Users have reported that uninstalling and reinstalling the app doesn't fix things.

The thread that crashes:

Thread 10 Crashed:
0   libdispatch.dylib             	0x00000001c3caebe0 __DISPATCH_WAIT_FOR_QUEUE__ + 452 (queue.c:1615)
1   libdispatch.dylib             	0x00000001c3cae758 _dispatch_sync_f_slow + 144 (queue.c:1728)
2   App                     	0x0000000102764448 -[FIRRemoteConfig configValueForKey:namespace:] + 220
3   App                     	0x00000001026dedb8 -[FPRRemoteConfigFlags sessionSamplingRate] + 60
4   App                     	0x00000001026de44c -[FPRConfigurations sessionsSamplingPercentage] + 212
5   App                     	0x00000001026d7ec4 -[FPRSessionManager isGaugeCollectionEnabledForSessionId:] + 60
6   App                     	0x00000001026d7d4c -[FPRSessionManager updateSessionId:] + 292
7   App                     	0x00000001026d7a6c -[FPRSessionManager initWithNotificationCenter:] + 128
8   App                     	0x00000001026d79c8 __35+[FPRSessionManager sharedInstance]_block_invoke + 84
9   libdispatch.dylib             	0x00000001c3ca2114 _dispatch_client_callout + 20 (object.m:495)
10  libdispatch.dylib             	0x00000001c3ca37f0 _dispatch_once_callout + 32 (once.c:52)
11  App                     	0x00000001026d796c +[FPRSessionManager sharedInstance] + 56
12  App                     	0x00000001026e987c -[FPRNetworkTrace start] + 228
13  App                     	0x00000001026f0870 __InstrumentDataTaskWithRequestCompletionHandler_block_invoke + 368
14  App                     	0x00000001026f07ec __InstrumentDataTaskWithRequestCompletionHandler_block_invoke + 236
15  App                     	0x00000001026a7930 -[FIRInstanceIDCheckinService checkinWithExistingCheckin:completion:] + 652 (FIRInstanceIDCheckinService.m:189)
16  App                     	0x00000001026a509c -[FIRInstanceIDAuthService fetchCheckinInfoWithHandler:] + 496 (FIRInstanceIDAuthService.m:188)
17  App                     	0x000000010269e648 -[FIRInstanceID tokenWithAuthorizedEntity:scope:options:handler:] + 1092 (FIRInstanceID.m:324)
18  App                     	0x00000001027729fc -[RCNConfigFetch refreshInstanceIDTokenAndFetchCheckInInfoWithCompletionHandler:] + 392
19  App                     	0x00000001027725c8 __74-[RCNConfigFetch fetchAllConfigsWithExpirationDuration:completionHandler:]_block_invoke + 288
20  libdispatch.dylib             	0x00000001c3ca0cb8 _dispatch_call_block_and_release + 32 (init.c:1408)
21  libdispatch.dylib             	0x00000001c3ca2114 _dispatch_client_callout + 20 (object.m:495)
22  libdispatch.dylib             	0x00000001c3ca8534 _dispatch_lane_serial_drain + 568 (inline_internal.h:2487)
23  libdispatch.dylib             	0x00000001c3ca8f5c _dispatch_lane_invoke + 364 (inline_internal.h:1854)
24  libdispatch.dylib             	0x00000001c3cb1fd0 _dispatch_workloop_worker_thread + 580 (queue.c:6380)
25  libsystem_pthread.dylib       	0x00000001c3cdfa64 _pthread_wqthread + 308 (pthread.c:2329)
26  libsystem_pthread.dylib       	0x00000001c3ce5c44 start_wqthread + 8

I've attached the full crash log, let me know if I can provide any more info

2019-06-27_02-12-29.9158_+0100-8692347274dbd966774cf859c53577c33a82cbb0.txt

[mouness2020]

also firebase remote config crash in iOS13 .
test_issue_crash_0cbd0e86b7fd44d98dac238ea687bb09_DNE_0_v2.txt
test2_issue_crash_1668e1b4c3a24090b2ecf6a4eb157017_DNE_0_v2.txt

[ozzotto]

Same crash for me too.

[morganchen12]

I'm looking into this issue. It seems to be caused by a deadlock in RemoteConfig and InstanceID. Will keep this thread updated with details.

[kross51]

I also have a crash on iOS 13:

libdispatch.dylib`DISPATCH_WAIT_FOR_QUEUE:
0x105a96414 <+0>: pacibsp
...
0x105a965e4 <+464>: ldp x20, x21, [sp], #0x10
-> 0x105a965e8 <+468>: brk #0x1

dump.txt

[morganchen12]

@maksymmalyhin reassigning to you since I'm going on vacation next week. I wasn't able to conclusively prove this crash is caused by a deadlock. Here's what I've figured out thus far:

• The crash is always at launch (+[FIRApp configure] is always in the stack trace).
• The crash always occurs in a background thread on a dispatch_wait after a dispatch_sync, called from RemoteConfig.
• The crash involves fetching data from keychain, and may be a regression of Crash and rejection from the App Store: scene-create watchdog transgression #1399.
• The stack trace usually includes a dispatch_sync on the main thread, called from InstanceID.

Feel free to assign it back to me when I return from vacation on July 8th.

[seanpai96]

I’m also experiencing this problem and I would like to help. What info can I provide?

[paulb777]

@seanpai96 If you see a backtrace different from the one above, please share it. Also, if sharing a reproducible example would be helpful.

[paulb777]

We haven't identified the issue yet, but there's a reasonable chance the bug was introduced in Firebase 6.3.0 in which case the recommended workaround would be adding the following to the Podfile:

pod 'Firebase', '~> 6.2.0'

[seanpai96]

The debug log logged this first then stucked forever:
2019-06-29 22:22:59.267157+0800 TestApp[581:81737] [] tcp_output [C2.1.1:3] flags=[R.] seq=1957931661, ack=149483231, win=1024 state=CLOSED rcv_nxt=149483231, snd_una=1957931661
2019-06-29 22:22:59.345468+0800 TestApp[581:81737] [] tcp_output [C2.2.1:3] flags=[R.] seq=1622070220, ack=3146498977, win=1024 state=CLOSED rcv_nxt=3146498977, snd_una=1622070220
2019-06-29 22:22:59.349965+0800 TestApp[581:81737] Connection 2: received failure notification
2019-06-29 22:22:59.350169+0800 TestApp[581:81737] Connection 2: failed to connect 1:50, reason -1
2019-06-29 22:22:59.350314+0800 TestApp[581:81737] Connection 2: encountered error(1:50)
2019-06-29 22:22:59.371500+0800 TestApp[581:81737] Task .<1> HTTP load failed, 0/0 bytes (error code: -1009 [1:50])
idk if this will help, but it seems like there's something wrong between the client and the server. (my internet connection is well)
when I paused the debug session, the app stayed at line 189 in FIRInstanceIDCheckinService.m, where there is a URLSession task.
Sometimes it stuck at this place, and other times it just crashed and the log is like what you've seen above.
It crashed at launch screen so it's reproducing everytime when opening the app.
btw sorry for the bad English.

[dmandar]

Hi..we are still investigating the root cause. Meanwhile, noticed that you have a remote config fetch call immediately followed by query to get value for a param at startup. Note that Remote Config initializes in an async manner and it is recommended to use ensureInitializedWithCompletionHandler to make sure RC is initialized before fetching and querying for RC values.

[dmandar]

@milo- @seanpai96 can you share the code snippet that causes the crash (did you have Remote Config enabled)? Also, if you are able to repro, would appreciate verbose logs by modifying the loggerLevel https://firebase.google.com/docs/reference/ios/firebasecore/api/reference/Classes/FIRConfiguration

[seanpai96]

The app boots up successfully after two times of rebuild but I'm still putting up the snippet and the log when the crash happened.
The place of crashing is the same as #3283 .
I don't have or initialize RC and I only have push notification and analytics on.

[paulb777]

We suspect the crash may be related to FirebasePerformance. Is anyone seeing the crash who does not have FirebasePerformance installed in their app?

[paulb777]

A workaround is to remove FirebasePerformance from the apps that need to run on iOS 13.

Stay tuned for info about the fix.

[martyfuhry]

I was able to get my app to run after removing FirebasePerformance on iOS 13.

[paulb777]

We now have a fix that will be built into the 6.4.0 release that is planned to publish next week.

[sipersso]

Removing Firebase Performance seems to resolve the issue for iOS 13 users. But I still get crashes on startup for some users on iOS 12.3.1, even after removing firebase performance :/ So I'd strongly recommend to downgrade to 6.2.0

[romin991]

hello @paulb777 , may I know when the 6.4.0 will be released?

[mouness2020]

@romin991 it's already released yesterday
https://firebase.google.com/support/release-notes/ios#version_640_-_july_9_2019

[Valpertui]

I have the same issue (deadlock on startup). Will test 6.4.0 to see if the issues goes away

[sipersso]

@mouness2020 that issue doesn't mention iOS 13. Can anyone confirm that 6.4.0 solves the problem? Is there any more info about what caused this issue? And how it was resolved?

I got a number of reports from users that the app crashed on startup on iOS13 and iOS 12.3.1. Removing Firebase Performance did not fully resolve the issue. After releasing an app update where I downgraded to 6.2.0 it was resolved.

Can we be sure that it won't crash at startup on 6.4.0?

[sipersso]

Downgrading to 6.2.0 doesn't resolve it either. Still having users complaining about crashes.

[Valpertui]

Upgrading to 6.4.0 fixed the deadlock issue.

[ryanwilson]

Removing Firebase Performance seems to resolve the issue for iOS 13 users. But I still get crashes on startup for some users on iOS 12.3.1, even after removing firebase performance :/ So I'd strongly recommend to downgrade to 6.2.0

@sipersso can you please confirm, is the 12.3.1 crash the exact same stack trace you see above? Or is it a separate issue? The crash stems from Performance, so I'm not sure that you're running into the same issue. Any help tracking this down would be greatly appreciated.

@mouness2020 that issue doesn't mention iOS 13. Can anyone confirm that 6.4.0 solves the problem?

The Performance Monitoring changelog does mention the iOS 13 beta crash. Was there another issue you meant?

Is there any more info about what caused this issue? And how it was resolved?

The issue had to do with a timing issue with the UIWindowDidBecomeVisibleNotification notification as well as a Performance dependency on Remote Config (Performance uses Remote Config to dynamically adjust sampling rates as needed). In iOS 12 and below, UIWindowDidBecomeVisibleNotification gets fired before application(_:didFinishLaunching:) but on iOS 13 betas it was being fired afterwards. This caused the Perf specific code to be run during Remote Config's startup sequence and ended up trying to fetch a value from Remote Config on the same serial thread that Remote Config was using for configuration, which ended up in a deadlock. The fix was to wait for Remote Config to successfully a fetch before trying to retrieve values in Remote Config and fall back to a default value if the fetch hasn't occurred yet.

[sipersso]

@ryanwilson I wish I could provide you with a stacktrace, but I can't. I can't reproduce the issue locally. It is only happening in production Firebase Crashlytics lists the app as 100% crash free and no crash reports, which leads me to believe that the error is during initialization. So far the issue has only been on iOS13 and iOS 12.3.1.

Here is what happened.
1: I upgraded to 6.3.0, released it to production:
Result:
Got reports of crashes iOS13 users and a few on iOS12.3.1. Firebase Crashlytics doesn't show any of these errors.

2: I removed Firebase Performance, but still used 6.3.0.
Result:
iOS13 crasches are gone, but there are still reports of the app crashes at startup. Seems as if only iOS 12.3.1 users are affected.

I assume that this is the current issue?

3: I downgraded to 6.2.0:
Result:
This resolved the issue for those that contacted me about the issue after the previous update. But there are still rare reports of the app crashing at startup on iOS 12.3.1. Thankfully the crash seems rare, but I have received, but I have received a couple of reports.

Is this this a separate issue?

I have upgrade to 6.4.0 now and I am submitting it to Apple as we speak. I am thinking 12.3.1 and iOS 13 could possibly share the timing issues?

[ryanwilson]

Thanks for the info. Responses inline...

Here is what happened.
1: I upgraded to 6.3.0, released it to production:
Result:
Got reports of crashes iOS13 users and a few on iOS12.3.1. Firebase Crashlytics doesn't show any of these errors.

I'm not sure why Crashlytics wouldn't show these crashes - @samedson any idea why that could be?

2: I removed Firebase Performance, but still used 6.3.0.
Result:
iOS13 crasches are gone, but there are still reports of the app crashes at startup. Seems as if only iOS 12.3.1 users are affected.

I assume that this is the current issue?

This issue reported above is solely due to Firebase Performance. If you removed FirebasePerformance from your application, it must be a separate issue with the app (either the app itself, a different set of frameworks, for Firebase). For what it's worth, we haven't seen other crashers being reported for 6.3.0 on iOS 12.3.1.

3: I downgraded to 6.2.0:
Result:
This resolved the issue for those that contacted me about the issue after the previous update. But there are still rare reports of the app crashing at startup on iOS 12.3.1. Thankfully the crash seems rare, but I have received, but I have received a couple of reports.

Is this this a separate issue?

Same comment as above, we haven't gotten reports of crashing on iOS 12.3.1 and without a stack trace it's impossible to tell if Firebase is related or not.

I have upgrade to 6.4.0 now and I am submitting it to Apple as we speak. I am thinking 12.3.1 and iOS 13 could possibly share the timing issues?

Based on you removing Performance and the crashes on 12.3.1 still happening, us not receiving other crashing reports for 12.3.1, and Apple's changes to the startup windowing sequence I believe that the crash you're describing is unrelated to the stacktrace and issue reported above.

[Valpertui]

I'm not sure why Crashlytics wouldn't show these crashes - @samedson any idea why that could be?

When you have the issue, you get it at 100% of the launches (at least for me) and it's an instant crash, so I doubt crashlytics was able to send the crash report on the next launch (since it would crash too)

[ryanwilson]

I'm not sure why Crashlytics wouldn't show these crashes - @samedson any idea why that could be?

When you have the issue, you get it at 100% of the launches (at least for me) and it's an instant crash, so I doubt crashlytics was able to send the crash report on the next launch (since it would crash too)

Yep, that's fair 🙂

[sipersso]

@Valpertui Yes, that is correct. It crashes repeatedly for these users. Not really sure how to open a ticket for this since I can't have any stacktraces, but I have upgraded to 6.4.0 and also noticed that my Firebase initialization was done in willFinishLaunching, and not didFinishLaunching, which I believe would increase the likelyhood it being the timing issue described. So I have moved it.

There have been very few reports so far, but so I am leaning towards releasing a phased update containing 6.4.0 and moving the initialization of Firebase to the correct place and see how it goes. A bit scary without the crash reports though.

[sipersso]

Same comment as above, we haven't gotten reports of crashing on iOS 12.3.1 and without a stack trace it's impossible to tell if Firebase is related or not.

I am confident it is related to firebase since the update I started seeing this in only contains a few cosmetic changes and the firebase updates. The other code changes are nowhere near the app startup.

[samedson]

@ryanwilson @Valpertui that's correct about Crashlytics - it needs to start up successfully before it will start collecting crashes. If an SDK starts before Crashlytics it won't collect a crash from it