Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dyld crash at launch #1246

Closed
alloy opened this issue Mar 10, 2016 · 76 comments

Comments

@alloy
Copy link
Member

commented Mar 10, 2016

  • A colleague ran into a situation where the app would crash on launch. Important to note is that she was also seeing it with other apps, such as LinkedIn.app.
  • A restart of her device ‘fixed’ the issue.
  • These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.
  • Extracting the crash reports directly from the device revealed a dyld error.
  • Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.

TL;DR

This appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably this neat trick does so as well.

Info about the errors

Dyld Error Message:
Dyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack
  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy
  Reason: no suitable image found.  Did find:
    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C
  Dyld Version: 370.1
  • The error message and codes are always the same.
  • The error is not specific to a framework, it happens with various frameworks.
  • It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).

Known affected versions

These are the various versions for which we have crash reports.

iOS versions

~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq
OS Version:          iOS 9.1 (13B143)
OS Version:          iOS 9.2 (13C75)
OS Version:          iOS 9.2.1 (13D15)
OS Version:          iOS 9.2.1 (13D20)
OS Version:          iOS 9.3 (13E5214d)
OS Version:          iOS 9.3 (13E5225a)

dyld versions

~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq
  Dyld Version: 370.1
  Dyld Version: 370.6
  Dyld Version: 390.7

Device versions

~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq
Hardware Model:      iPad3,4
Hardware Model:      iPad4,4
Hardware Model:      iPad4,7
Hardware Model:      iPad5,3
Hardware Model:      iPad5,4
Hardware Model:      iPad6,7
Hardware Model:      iPad6,8
Hardware Model:      iPhone6,1
Hardware Model:      iPhone7,1
Hardware Model:      iPhone7,2
Hardware Model:      iPhone8,1
Hardware Model:      iPhone8,2

Retracing

  1. The error is raised here in dyld.
  2. Only place where mremap_encrypted could return -1 is here.
  3. vn_getpath calls build_path, which in turn has a few possible locations where it could be returning -1:
  4. But trying to hunt that further down would take a lot more time, suffice to say that:

Caveats

Alas, the latest dyld sources to be released (at the time of writing) appear to be for version 360.18.

Taking a cursory look at the decompiled code for ImageLoaderMachOCompressed::registerEncryption from a firmware image for iOS 9.2 for iPad3,4, does not appear to have any real changes.

~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw 
Archive:  iPad3,4_9.2_13C75_Restore.ipsw
  inflating: 058-25903-078.dmg       
  inflating: 058-26242-078.dmg       
  inflating: 058-26268-078.dmg
[…]

~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3
Writing out data..
[…]

~/t/ios-9.2-iPad3,4 » open extracted.dmg
~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld
int ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {
    r0 = arg0;
    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;
    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;
    r6 = arg1;
    r8 = r2;
    r4 = r0;
    if (r6 != 0x0) {
            r5 = 0x0;
            do {
                    r2 = *(*r4 + 0xe8);
                    if ((r2)(r4, r5, r2) != 0x0) goto (null);
                    r5 = r5 + 0x1;
            } while (true);
            r1 = r5;
            r2 = *(*r4 + 0x104);
            r0 = (r2)(r4, r1, r2);
            r10 = r6 + 0x8;
            asm{ ldm.w      sl, {r1, r5, sl} };
            r2 = *(r8 + 0xb2);
            r8 = *(r0 + 0x4);
            r6 = r0 + r1;
            r11 = *(r0 + 0x8);
            if (r2 != 0x0) {
                    dyld::log("                      0x%08lX->0x%08lX configured for FairPlay decryption\n");
            }
            r0 = _mremap_encrypted();
            r5 = r0;
            if (r5 != 0x0) {
                    r0 = ___error();
                    r3 = *(r4 + 0x4);
                    r2 = *r0;
                    r1 = r5;
                    r0 = dyld::throwf("mremap_encrypted() => %d, errno=%d for %s\n");
                    //
            }
    }
    return r0;
}

Conclusion

The crash at launch definitely feels like a very bad experience and I feel like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.

I’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.

It would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.

Status

@alloy alloy added the Crash label Mar 10, 2016

@alloy

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2016

Dynamic library sizes:

$ cd DerivedData/Artsy/Build/Products/Debug-iphoneos/Artsy.app/Frameworks
$ find . -perm +u=x -type f ! -iname '*.strings' ! -iname '*.ttf' ! -iname '*.otf' -print0 | xargs -0 ls -lh | awk '{print $5, $9}' | column -t | gsort -h -r -k1
13M   ./libswiftCore.dylib
1.5M  ./libswiftFoundation.dylib
1.4M  ./HockeySDK_Source.framework/HockeySDK_Source
1.0M  ./ReactiveCocoa.framework/ReactiveCocoa
956K  ./FBSDKCoreKit.framework/FBSDKCoreKit
656K  ./AFNetworking.framework/AFNetworking
395K  ./libswiftCoreGraphics.dylib
392K  ./SDWebImage.framework/SDWebImage
368K  ./CocoaLumberjack.framework/CocoaLumberjack
363K  ./Adjust.framework/Adjust
337K  ./libswiftDarwin.dylib
294K  ./libswiftCoreAudio.dylib
294K  ./Analytics.framework/Analytics
288K  ./FBSDKLoginKit.framework/FBSDKLoginKit
273K  ./Bolts.framework/Bolts
255K  ./libswiftObjectiveC.dylib
248K  ./libswiftUIKit.dylib
248K  ./libswiftCoreMedia.dylib
241K  ./MMMarkdown.framework/MMMarkdown
240K  ./libswiftDispatch.dylib
240K  ./libswiftAVFoundation.dylib
224K  ./Mantle.framework/Mantle
210K  ./ARAnalytics.framework/ARAnalytics
208K  ./libswiftWebKit.dylib
208K  ./libswiftCoreLocation.dylib
208K  ./libswiftCoreData.dylib
208K  ./libswiftContacts.dylib
202K  ./libswiftCoreImage.dylib
163K  ./NAMapKit.framework/NAMapKit
151K  ./AFOAuth1Client.framework/AFOAuth1Client
146K  ./iRate.framework/iRate
145K  ./ALPValidator.framework/ALPValidator
138K  ./ARTiledImageView.framework/ARTiledImageView
136K  ./VCRURLConnection.framework/VCRURLConnection
135K  ./Interstellar.framework/Interstellar
135K  ./ARGenericTableViewController.framework/ARGenericTableViewController
131K  ./ARCollectionViewMasonryLayout.framework/ARCollectionViewMasonryLayout
128K  ./FLKAutoLayout.framework/FLKAutoLayout
118K  ./EDColor.framework/EDColor
111K  ./JSDecoupledAppDelegate.framework/JSDecoupledAppDelegate
109K  ./ObjectiveSugar.framework/ObjectiveSugar
109K  ./Keys.framework/Keys
108K  ./FXBlurView.framework/FXBlurView
106K  ./Artsy_UILabels.framework/Artsy_UILabels
106K  ./Artsy_UIButtons.framework/Artsy_UIButtons
105K  ./KSDeferred.framework/KSDeferred
104K  ./ORStackView.framework/ORStackView
104K  ./Aerodramus.framework/Aerodramus
103K  ./PBDCarouselCollectionViewLayout.framework/PBDCarouselCollectionViewLayout
103K  ./JLRoutes.framework/JLRoutes
101K  ./MARKRangeSlider.framework/MARKRangeSlider
101K  ./JSBadgeView.framework/JSBadgeView
100K  ./ISO8601DateFormatter.framework/ISO8601DateFormatter
99K   ./UICKeyChainStore.framework/UICKeyChainStore
96K   ./RSSwizzle.framework/RSSwizzle
81K   ./ORKeyboardReactingApplication.framework/ORKeyboardReactingApplication
81K   ./ARASCIISwizzle.framework/ARASCIISwizzle
80K   ./UIView_BooleanAnimations.framework/UIView_BooleanAnimations
80K   ./DRKonamiCode.framework/DRKonamiCode
80K   ./AFNetworkActivityLogger.framework/AFNetworkActivityLogger
79K   ./UIAlertView_Blocks.framework/UIAlertView_Blocks
79K   ./Artsy_UIFonts.framework/Artsy_UIFonts
78K   ./NPKeyboardLayoutGuide.framework/NPKeyboardLayoutGuide
76K   ./MultiDelegate.framework/MultiDelegate
76K   ./Artsy_UIColors.framework/Artsy_UIColors
75K   ./Then.framework/Then
74K   ./DHCShakeNotifier.framework/DHCShakeNotifier
@alloy

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2016

There is very few mentions about this crash that I could find, maybe because people don’t realise it is happening?


Sadly enough, two of those cases appear to get their app rejected, 1 followed up to say they recompiled and resubmitted and it was accepted. I bet that these were just cases of the reviewer’s device running out of memory and that the second case also got accepted after resubmitting.

@0xced

This comment has been minimized.

Copy link

commented Mar 10, 2016

@JaviSoto had a discussion on twitter last week about the same issue but it looks like no solution surfaced. 🙁

@alloy

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2016

@0xced Aye, he has let me know, it also sparked others chiming in. I have added a ‘Status’ section to the bottom of the OP.

@quellish

This comment has been minimized.

Copy link

commented Mar 15, 2016

The jetsam/memory pressure process is described here for reference: http://newosxbook.com/articles/MemoryPressure.html

If the issue can be reliably reproduced it may be worth testing explicit weak linking of some frameworks as described in SDK based development

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2016

My radar has been closed as a duplicate of 24278648.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2016

@robinsenior Did you get any updates on your DTS?

@robinsenior

This comment has been minimized.

Copy link

commented Apr 1, 2016

They initially told me it was fixed in 9.3, but an Apple dev on Twitter told me otherwise. I see crash logs for it in beta 7 but not the full release (AFAIK, I haven't checked in a few days).

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2016

@robinsenior I have crash reports for 9.3 builds 13E233 and 13E234, so definitely not fixed.

If you don’t have any atm and want to follow-up with DTS I can send you some.

@robinsenior

This comment has been minimized.

Copy link

commented Apr 1, 2016

@alloy thanks; I just checked and we have some with those build numbers so I'll email them to DTS.

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2016

Ace 👍

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

That’s great news! Might update my personal device sooner after all 😉

@lumaxis

This comment has been minimized.

Copy link

commented Apr 1, 2016

Apparently there have been some changes to dyld in 9.3 which resulted in significant performance improvements: https://github.com/stepanhruda/dyld-image-loading-performance
Didn't fix the crash from this issue though 😕

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2016

Yeah, that’s about #586.

@robinsenior

This comment has been minimized.

Copy link

commented Apr 4, 2016

Just got this update from DTS:

We’ve made some progress on this one. It seems to be related to the number of embedded frameworks an app has. I noticed that your app has 40 frameworks (12 of which are the Swift dylibs). You could reduce the chance of your app hitting this bug by combining some of those frameworks. Even cutting the number of frameworks in half would make a difference.

/facepalm

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2016

Oh jeez… So their progress is basically all of what this ticket already describes ¯_(ツ)_/¯

I do get that they want to give you something that’s actionable for you right now, but it would have been nice to know a bit more what their status is on the real issue. (rdar://24278648, which is the one mine got closed in favour of.)

@ctava

This comment has been minimized.

Copy link

commented Apr 7, 2016

Hi Alloy - Have you heard anything more about iOS dynamic library loading crasher? or are you guys making any changes to combine frameworks etc?
Thanks in advance for any insight.
-chris

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 7, 2016

@ctava Nope, if/when we would hear more, or decide a different course of action, it all goes public in this ticket. I don’t think combining frameworks would help, as it would probably still run out of memory and die in the same way. The only possibility to reduce (not fully get rid of) these issues would be to get rid of ‘fat’ (number of larger frameworks), which is not something we will be doing at this time.

To summarise, imo it’s clearly a bug in iOS, and the radar that mine was closed in favour of is still open, so they haven’t ruled it as “works as intended” yet. Until that time, I don’t think the number of crashes vs the number of users warrants a lot of work on our part. YMMV.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2016

@robinsenior Thinking about that answer a bit more, could you ask them if they do in fact mean the number of libs vs the size of those libs? If they are positive that reducing the number of libs, regardless of size, is a way to workaround the issue, then that would definitely be helpful info.

@robinsenior

This comment has been minimized.

Copy link

commented Apr 8, 2016

@alloy done. I'll keep you posted.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2016

Ace, thanks 👍

@maxmeyers

This comment has been minimized.

Copy link

commented Apr 8, 2016

For another data point: we're seeing pretty commonly in our app as well since we switched to using dynamic frameworks with CocoaPods. We have 41 frameworks averaging about 1.5MB each. It's a pretty significant issue for us, so we're going back to a static library in our next release.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2016

@maxmeyers Are you using Swift yet?

@maxmeyers

This comment has been minimized.

Copy link

commented Apr 8, 2016

@alloy Fortunately not in production yet.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2016

@maxmeyers Ok cool. Just wanted to check if the Swift dylibs were not causing this issue for you, but that explains it then. Thanks for the feedback 👍

@robinsenior

This comment has been minimized.

Copy link

commented Apr 13, 2016

@alloy DTS says it's the number of libs, not the size.

@alloy

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2016

@robinsenior Well… that’s unexpected. Are you going to spend time on reducing the number of dynamic libs?

@maxmeyers

This comment has been minimized.

Copy link

commented May 11, 2016

We didn't have any Swift dependencies. One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies (that's assuming you're currently using CocoaPods to install your frameworks).

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 11, 2016

@maxmeyers Do you use Swift in your app? If so, are you still getting this crash report (albeit in low numbers) due to libSwiftCore.dylib etc?

@maxmeyers

This comment has been minimized.

Copy link

commented May 11, 2016

No Swift at all. This crash has completely disappeared.

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 11, 2016

One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies.

Or you could use a second Podfile for your Swift dependencies and use something like Rome, if you prefer CocoaPods ;)

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 11, 2016

@maxmeyers Ah, bugger, had hoped for an extra convincing case to Apple that shows that just using Swift leads to this issue. But great for you of course!

@SquaredTiki

This comment has been minimized.

Copy link

commented May 11, 2016

@alloy: That wouldn't solve the issue though would it? Only reduce the likelihood of the crash because of less frameworks. Either way, unfortunately our app is 50% Swift so most of our pods are used in both 😞

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 11, 2016

@SquaredTiki Correct.

@quellish

This comment has been minimized.

Copy link

commented May 11, 2016

Have any of you attempted to weak link frameworks and load them after launch using the NSBundle API?

@orta

This comment has been minimized.

Copy link
Contributor

commented May 11, 2016

With that technique, it sounds like you're starting to move into some of the territory covered in this issue: #586

@quellish

This comment has been minimized.

Copy link

commented May 11, 2016

There are two, related issues at work here, both centering on the dynamic loader. Memory pressure and CPU. In the crash logs that I have seen memory pressure is the overriding concern. As the loader loads strongly linked frameworks at startup it has to verify their signatures. System processes involved in that increase memory pressure when it may have already been high on the device. The loader can't get enough memory to continue loading frameworks and their dependencies, the application can't be launched or is immediately terminated.

Weak-linking the frameworks and loading after launch using the NSBundle API gives the developer more control over this process.

Monitoring the performance of signature verification may also be helpful.

@patters

This comment has been minimized.

Copy link

commented May 13, 2016

Apple has told us that weak linking does not help the problem.

@tsabend

This comment has been minimized.

Copy link

commented May 13, 2016

We're having a similar problem and just wanted to reiterate the thanks for Artsy's open source ethos. This thread is very helpful.

@freshnitesh

This comment has been minimized.

Copy link

commented May 13, 2016

In order to reproduce this crash consistently, we downloaded a number of apps from the App Store on an iOS 9.3.1 device (iPhone 6+). I opened 14 apps, and when trying to open the 15th app, I experienced a crash on launch. From there, any app I tried to open for the first time would crash on launch, while previously opened apps would continue to launch successfully.

I was able to reproduce this same sequence both after a reboot, and after manually killing all open apps via the app switcher.

After upgrading the phone iOS version to 9.3.2 Beta 4, I was unable to reproduce any crash. At the maximum, I opened 50+ apps successfully. Hopefully this means the issue is fixed or at least ameliorated with iOS 9.3.2 Beta 4.

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 13, 2016

@freshnitesh That’s great to hear, tanks for the update!

@freshnitesh

This comment has been minimized.

Copy link

commented May 13, 2016

@alloy Happy to help :)

@UnsafePointer

This comment has been minimized.

Copy link

commented May 15, 2016

@freshnitesh thank you so much for the detailed instructions on how to reproduce the crash consistently. I'm also able to confirm that iOS 9.3.2 beta 4 fixes this issue. At some point, apps take longer to launch but no crash so far, which is way better.

@mergesort

This comment has been minimized.

Copy link

commented May 16, 2016

9.3.2 is out. So far nothing is crashing, but will keep an eye on this in the coming days/weeks. 🙏 for success.

@gparker42

This comment has been minimized.

Copy link

commented May 19, 2016

Yes, we believe the crashes on launch with mremap_encrypted -1 errno=12 are fixed in iOS 9.3.2 (as of beta 3, I think). Please file new bug reports if you see it again.

@orta

This comment has been minimized.

Copy link
Contributor

commented May 19, 2016

Alright, I'm calling this 👏

👍

@orta orta closed this May 19, 2016

@alloy

This comment has been minimized.

Copy link
Member Author

commented May 19, 2016

Awesome! 👏

@zenangst

This comment has been minimized.

Copy link

commented May 21, 2016

This is great! 🚀

@wolffan

This comment has been minimized.

Copy link

commented Sep 23, 2016

That's an amazing thread.
Despite the fact that is fixed in 9.3.2 (and I hope iOS 10), there's still the issue we're supporting quite a lot of old version devices. What you do on those cases?

@orta

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2016

Aside from the manual work shown in:

A new CocoaPods plugin has been created by @Ruenzuo since then that will automate most of that work:

Other than that, there's no real choice to to write it off. At least 9.3 came with awesome emoji, and that got most of our users to update.

@aaghawaheed

This comment was marked as off-topic.

Copy link

commented Sep 23, 2018

screen shot 2018-09-24 at 3 46 14 am

I am getting this error. App is working fine on simulator, but as I run it on device it crashes .I don't know why it's happening.

@artsy artsy locked as resolved and limited conversation to collaborators Sep 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
You can’t perform that action at this time.