dyld crash at launch #1246

alloy · 2016-03-10T14:39:42Z

A colleague ran into a situation where the app would crash on launch. Important to note is that she was also seeing it with other apps, such as LinkedIn.app.
A restart of her device ‘fixed’ the issue.
These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.
Extracting the crash reports directly from the device revealed a dyld error.
Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.

TL;DR

This appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably this neat trick does so as well.

Info about the errors

Dyld Error Message:
Dyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack
  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy
  Reason: no suitable image found.  Did find:
    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C
  Dyld Version: 370.1

The error message and codes are always the same.
The error is not specific to a framework, it happens with various frameworks.
It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).

Known affected versions

These are the various versions for which we have crash reports.

iOS versions

~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq
OS Version:          iOS 9.1 (13B143)
OS Version:          iOS 9.2 (13C75)
OS Version:          iOS 9.2.1 (13D15)
OS Version:          iOS 9.2.1 (13D20)
OS Version:          iOS 9.3 (13E5214d)
OS Version:          iOS 9.3 (13E5225a)

dyld versions

~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq
  Dyld Version: 370.1
  Dyld Version: 370.6
  Dyld Version: 390.7

Device versions

~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq
Hardware Model:      iPad3,4
Hardware Model:      iPad4,4
Hardware Model:      iPad4,7
Hardware Model:      iPad5,3
Hardware Model:      iPad5,4
Hardware Model:      iPad6,7
Hardware Model:      iPad6,8
Hardware Model:      iPhone6,1
Hardware Model:      iPhone7,1
Hardware Model:      iPhone7,2
Hardware Model:      iPhone8,1
Hardware Model:      iPhone8,2

Retracing

The error is raised here in dyld.
Only place where mremap_encrypted could return -1 is here.
vn_getpath calls build_path, which in turn has a few possible locations where it could be returning -1:
But trying to hunt that further down would take a lot more time, suffice to say that:
- if mremap_encrypted returns -1, it most likely corresponds to ERESTART: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L270
- if errno is 12, it most likely corresponds to ENOMEM: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L102

Caveats

Alas, the latest dyld sources to be released (at the time of writing) appear to be for version 360.18.

Taking a cursory look at the decompiled code for ImageLoaderMachOCompressed::registerEncryption from a firmware image for iOS 9.2 for iPad3,4, does not appear to have any real changes.

~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw 
Archive:  iPad3,4_9.2_13C75_Restore.ipsw
  inflating: 058-25903-078.dmg       
  inflating: 058-26242-078.dmg       
  inflating: 058-26268-078.dmg
[…]

~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3
Writing out data..
[…]

~/t/ios-9.2-iPad3,4 » open extracted.dmg
~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld

int ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {
    r0 = arg0;
    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;
    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;
    r6 = arg1;
    r8 = r2;
    r4 = r0;
    if (r6 != 0x0) {
            r5 = 0x0;
            do {
                    r2 = *(*r4 + 0xe8);
                    if ((r2)(r4, r5, r2) != 0x0) goto (null);
                    r5 = r5 + 0x1;
            } while (true);
            r1 = r5;
            r2 = *(*r4 + 0x104);
            r0 = (r2)(r4, r1, r2);
            r10 = r6 + 0x8;
            asm{ ldm.w      sl, {r1, r5, sl} };
            r2 = *(r8 + 0xb2);
            r8 = *(r0 + 0x4);
            r6 = r0 + r1;
            r11 = *(r0 + 0x8);
            if (r2 != 0x0) {
                    dyld::log("                      0x%08lX->0x%08lX configured for FairPlay decryption\n");
            }
            r0 = _mremap_encrypted();
            r5 = r0;
            if (r5 != 0x0) {
                    r0 = ___error();
                    r3 = *(r4 + 0x4);
                    r2 = *r0;
                    r1 = r5;
                    r0 = dyld::throwf("mremap_encrypted() => %d, errno=%d for %s\n");
                    // …
            }
    }
    return r0;
}

Conclusion

The crash at launch definitely feels like a very bad experience and I feel like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.

I’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.

It would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.

Status

A couple of radar tickets have already been filed:
A DTS ticket has been filed http://twitter.com/senior/status/707996353964728321
A guess has been made as to the issue being a jetsam change https://twitter.com/Catfish_Man/status/707955795493687296 (jetsam is the process in which the kernel basically kills apps to free up memory)

The text was updated successfully, but these errors were encountered:

alloy · 2016-03-10T14:56:57Z

Dynamic library sizes:

$ cd DerivedData/Artsy/Build/Products/Debug-iphoneos/Artsy.app/Frameworks
$ find . -perm +u=x -type f ! -iname '*.strings' ! -iname '*.ttf' ! -iname '*.otf' -print0 | xargs -0 ls -lh | awk '{print $5, $9}' | column -t | gsort -h -r -k1
13M   ./libswiftCore.dylib
1.5M  ./libswiftFoundation.dylib
1.4M  ./HockeySDK_Source.framework/HockeySDK_Source
1.0M  ./ReactiveCocoa.framework/ReactiveCocoa
956K  ./FBSDKCoreKit.framework/FBSDKCoreKit
656K  ./AFNetworking.framework/AFNetworking
395K  ./libswiftCoreGraphics.dylib
392K  ./SDWebImage.framework/SDWebImage
368K  ./CocoaLumberjack.framework/CocoaLumberjack
363K  ./Adjust.framework/Adjust
337K  ./libswiftDarwin.dylib
294K  ./libswiftCoreAudio.dylib
294K  ./Analytics.framework/Analytics
288K  ./FBSDKLoginKit.framework/FBSDKLoginKit
273K  ./Bolts.framework/Bolts
255K  ./libswiftObjectiveC.dylib
248K  ./libswiftUIKit.dylib
248K  ./libswiftCoreMedia.dylib
241K  ./MMMarkdown.framework/MMMarkdown
240K  ./libswiftDispatch.dylib
240K  ./libswiftAVFoundation.dylib
224K  ./Mantle.framework/Mantle
210K  ./ARAnalytics.framework/ARAnalytics
208K  ./libswiftWebKit.dylib
208K  ./libswiftCoreLocation.dylib
208K  ./libswiftCoreData.dylib
208K  ./libswiftContacts.dylib
202K  ./libswiftCoreImage.dylib
163K  ./NAMapKit.framework/NAMapKit
151K  ./AFOAuth1Client.framework/AFOAuth1Client
146K  ./iRate.framework/iRate
145K  ./ALPValidator.framework/ALPValidator
138K  ./ARTiledImageView.framework/ARTiledImageView
136K  ./VCRURLConnection.framework/VCRURLConnection
135K  ./Interstellar.framework/Interstellar
135K  ./ARGenericTableViewController.framework/ARGenericTableViewController
131K  ./ARCollectionViewMasonryLayout.framework/ARCollectionViewMasonryLayout
128K  ./FLKAutoLayout.framework/FLKAutoLayout
118K  ./EDColor.framework/EDColor
111K  ./JSDecoupledAppDelegate.framework/JSDecoupledAppDelegate
109K  ./ObjectiveSugar.framework/ObjectiveSugar
109K  ./Keys.framework/Keys
108K  ./FXBlurView.framework/FXBlurView
106K  ./Artsy_UILabels.framework/Artsy_UILabels
106K  ./Artsy_UIButtons.framework/Artsy_UIButtons
105K  ./KSDeferred.framework/KSDeferred
104K  ./ORStackView.framework/ORStackView
104K  ./Aerodramus.framework/Aerodramus
103K  ./PBDCarouselCollectionViewLayout.framework/PBDCarouselCollectionViewLayout
103K  ./JLRoutes.framework/JLRoutes
101K  ./MARKRangeSlider.framework/MARKRangeSlider
101K  ./JSBadgeView.framework/JSBadgeView
100K  ./ISO8601DateFormatter.framework/ISO8601DateFormatter
99K   ./UICKeyChainStore.framework/UICKeyChainStore
96K   ./RSSwizzle.framework/RSSwizzle
81K   ./ORKeyboardReactingApplication.framework/ORKeyboardReactingApplication
81K   ./ARASCIISwizzle.framework/ARASCIISwizzle
80K   ./UIView_BooleanAnimations.framework/UIView_BooleanAnimations
80K   ./DRKonamiCode.framework/DRKonamiCode
80K   ./AFNetworkActivityLogger.framework/AFNetworkActivityLogger
79K   ./UIAlertView_Blocks.framework/UIAlertView_Blocks
79K   ./Artsy_UIFonts.framework/Artsy_UIFonts
78K   ./NPKeyboardLayoutGuide.framework/NPKeyboardLayoutGuide
76K   ./MultiDelegate.framework/MultiDelegate
76K   ./Artsy_UIColors.framework/Artsy_UIColors
75K   ./Then.framework/Then
74K   ./DHCShakeNotifier.framework/DHCShakeNotifier

alloy · 2016-03-10T15:21:50Z

There is very few mentions about this crash that I could find, maybe because people don’t realise it is happening?

Sadly enough, two of those cases appear to get their app rejected, 1 followed up to say they recompiled and resubmitted and it was accepted. I bet that these were just cases of the reviewer’s device running out of memory and that the second case also got accepted after resubmitting.

0xced · 2016-03-10T21:20:57Z

@JaviSoto had a discussion on twitter last week about the same issue but it looks like no solution surfaced. 🙁

alloy · 2016-03-10T21:38:23Z

@0xced Aye, he has let me know, it also sparked others chiming in. I have added a ‘Status’ section to the bottom of the OP.

quellish · 2016-03-15T07:38:08Z

The jetsam/memory pressure process is described here for reference: http://newosxbook.com/articles/MemoryPressure.html

If the issue can be reliably reproduced it may be worth testing explicit weak linking of some frameworks as described in SDK based development

alloy · 2016-04-01T11:04:10Z

My radar has been closed as a duplicate of 24278648.

alloy · 2016-04-01T11:05:16Z

@robinsenior Did you get any updates on your DTS?

robinsenior · 2016-04-01T12:27:59Z

They initially told me it was fixed in 9.3, but an Apple dev on Twitter told me otherwise. I see crash logs for it in beta 7 but not the full release (AFAIK, I haven't checked in a few days).

alloy · 2016-04-01T14:59:52Z

@robinsenior I have crash reports for 9.3 builds 13E233 and 13E234, so definitely not fixed.

If you don’t have any atm and want to follow-up with DTS I can send you some.

robinsenior · 2016-04-01T15:17:11Z

@alloy thanks; I just checked and we have some with those build numbers so I'll email them to DTS.

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

alloy · 2016-04-01T15:37:20Z

Ace 👍

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

That’s great news! Might update my personal device sooner after all 😉

lumaxis · 2016-04-01T15:41:25Z

Apparently there have been some changes to dyld in 9.3 which resulted in significant performance improvements: https://github.com/stepanhruda/dyld-image-loading-performance
Didn't fix the crash from this issue though 😕

alloy · 2016-04-01T15:43:24Z

Yeah, that’s about #586.

robinsenior · 2016-04-04T19:00:02Z

Just got this update from DTS:

We’ve made some progress on this one. It seems to be related to the number of embedded frameworks an app has. I noticed that your app has 40 frameworks (12 of which are the Swift dylibs). You could reduce the chance of your app hitting this bug by combining some of those frameworks. Even cutting the number of frameworks in half would make a difference.

/facepalm

alloy · 2016-04-05T09:10:17Z

Oh jeez… So their progress is basically all of what this ticket already describes ¯_(ツ)_/¯

I do get that they want to give you something that’s actionable for you right now, but it would have been nice to know a bit more what their status is on the real issue. (rdar://24278648, which is the one mine got closed in favour of.)

ctava · 2016-04-07T16:57:27Z

Hi Alloy - Have you heard anything more about iOS dynamic library loading crasher? or are you guys making any changes to combine frameworks etc?
Thanks in advance for any insight.
-chris

alloy · 2016-04-07T17:07:04Z

@ctava Nope, if/when we would hear more, or decide a different course of action, it all goes public in this ticket. I don’t think combining frameworks would help, as it would probably still run out of memory and die in the same way. The only possibility to reduce (not fully get rid of) these issues would be to get rid of ‘fat’ (number of larger frameworks), which is not something we will be doing at this time.

To summarise, imo it’s clearly a bug in iOS, and the radar that mine was closed in favour of is still open, so they haven’t ruled it as “works as intended” yet. Until that time, I don’t think the number of crashes vs the number of users warrants a lot of work on our part. YMMV.

alloy · 2016-04-08T14:28:42Z

@robinsenior Thinking about that answer a bit more, could you ask them if they do in fact mean the number of libs vs the size of those libs? If they are positive that reducing the number of libs, regardless of size, is a way to workaround the issue, then that would definitely be helpful info.

robinsenior · 2016-04-08T14:32:12Z

@alloy done. I'll keep you posted.

alloy · 2016-04-08T14:37:58Z

Ace, thanks 👍

maxmeyers · 2016-04-08T17:00:46Z

For another data point: we're seeing pretty commonly in our app as well since we switched to using dynamic frameworks with CocoaPods. We have 41 frameworks averaging about 1.5MB each. It's a pretty significant issue for us, so we're going back to a static library in our next release.

alloy · 2016-04-08T19:29:49Z

@maxmeyers Are you using Swift yet?

maxmeyers · 2016-04-08T19:37:09Z

@alloy Fortunately not in production yet.

alloy · 2016-04-08T19:38:11Z

@maxmeyers Ok cool. Just wanted to check if the Swift dylibs were not causing this issue for you, but that explains it then. Thanks for the feedback 👍

robinsenior · 2016-04-13T15:17:00Z

@alloy DTS says it's the number of libs, not the size.

alloy · 2016-04-13T15:30:55Z

@robinsenior Well… that’s unexpected. Are you going to spend time on reducing the number of dynamic libs?

maxmeyers · 2016-05-11T18:13:11Z

We didn't have any Swift dependencies. One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies (that's assuming you're currently using CocoaPods to install your frameworks).

alloy · 2016-05-11T18:15:12Z

@maxmeyers Do you use Swift in your app? If so, are you still getting this crash report (albeit in low numbers) due to libSwiftCore.dylib etc?

maxmeyers · 2016-05-11T18:17:46Z

No Swift at all. This crash has completely disappeared.

alloy · 2016-05-11T18:18:13Z

One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies.

Or you could use a second Podfile for your Swift dependencies and use something like Rome, if you prefer CocoaPods ;)

alloy · 2016-05-11T18:19:08Z

@maxmeyers Ah, bugger, had hoped for an extra convincing case to Apple that shows that just using Swift leads to this issue. But great for you of course!

SquaredTiki · 2016-05-11T18:29:35Z

@alloy: That wouldn't solve the issue though would it? Only reduce the likelihood of the crash because of less frameworks. Either way, unfortunately our app is 50% Swift so most of our pods are used in both 😞

alloy · 2016-05-11T18:35:56Z

@SquaredTiki Correct.

quellish · 2016-05-11T19:18:45Z

Have any of you attempted to weak link frameworks and load them after launch using the NSBundle API?

orta · 2016-05-11T19:20:31Z

With that technique, it sounds like you're starting to move into some of the territory covered in this issue: #586

quellish · 2016-05-11T19:28:53Z

There are two, related issues at work here, both centering on the dynamic loader. Memory pressure and CPU. In the crash logs that I have seen memory pressure is the overriding concern. As the loader loads strongly linked frameworks at startup it has to verify their signatures. System processes involved in that increase memory pressure when it may have already been high on the device. The loader can't get enough memory to continue loading frameworks and their dependencies, the application can't be launched or is immediately terminated.

Weak-linking the frameworks and loading after launch using the NSBundle API gives the developer more control over this process.

Monitoring the performance of signature verification may also be helpful.

patters · 2016-05-13T12:19:54Z

Apple has told us that weak linking does not help the problem.

tsabend · 2016-05-13T18:49:58Z

We're having a similar problem and just wanted to reiterate the thanks for Artsy's open source ethos. This thread is very helpful.

freshnitesh · 2016-05-13T19:39:59Z

In order to reproduce this crash consistently, we downloaded a number of apps from the App Store on an iOS 9.3.1 device (iPhone 6+). I opened 14 apps, and when trying to open the 15th app, I experienced a crash on launch. From there, any app I tried to open for the first time would crash on launch, while previously opened apps would continue to launch successfully.

I was able to reproduce this same sequence both after a reboot, and after manually killing all open apps via the app switcher.

After upgrading the phone iOS version to 9.3.2 Beta 4, I was unable to reproduce any crash. At the maximum, I opened 50+ apps successfully. Hopefully this means the issue is fixed or at least ameliorated with iOS 9.3.2 Beta 4.

alloy · 2016-05-13T20:10:25Z

@freshnitesh That’s great to hear, tanks for the update!

freshnitesh · 2016-05-13T20:26:14Z

@alloy Happy to help :)

UnsafePointer · 2016-05-15T16:42:20Z

@freshnitesh thank you so much for the detailed instructions on how to reproduce the crash consistently. I'm also able to confirm that iOS 9.3.2 beta 4 fixes this issue. At some point, apps take longer to launch but no crash so far, which is way better.

mergesort · 2016-05-16T20:33:40Z

9.3.2 is out. So far nothing is crashing, but will keep an eye on this in the coming days/weeks. 🙏 for success.

gparker42 · 2016-05-19T02:11:42Z

Yes, we believe the crashes on launch with mremap_encrypted -1 errno=12 are fixed in iOS 9.3.2 (as of beta 3, I think). Please file new bug reports if you see it again.

orta · 2016-05-19T03:49:07Z

Alright, I'm calling this 👏

👍

alloy · 2016-05-19T08:53:43Z

Awesome! 👏

zenangst · 2016-05-21T08:36:26Z

This is great! 🚀

wolffan · 2016-09-23T08:49:56Z

That's an amazing thread.
Despite the fact that is fixed in 9.3.2 (and I hope iOS 10), there's still the issue we're supporting quite a lot of old version devices. What you do on those cases?

orta · 2016-09-23T09:33:45Z

Aside from the manual work shown in:

A new CocoaPods plugin has been created by @Ruenzuo since then that will automate most of that work:

https://github.com/Ruenzuo/cocoapods-amimono

Other than that, there's no real choice to to write it off. At least 9.3 came with awesome emoji, and that got most of our users to update.

alloy added the Crash label Mar 10, 2016

openradar-mirror mentioned this issue Mar 10, 2016

25085577: Sporadic Dyld crashes on app launch lionheart/openradar-mirror#7739

Open

openradar-mirror mentioned this issue Mar 12, 2016

25126016: iOS may run out of memory while loading dynamic libraries. lionheart/openradar-mirror#7753

Open

iblacksun mentioned this issue Apr 1, 2016

App rejected with 'Library not loaded:' CocoaPods/CocoaPods#5067

Closed

coreyfloyd mentioned this issue May 11, 2016

Convert CocoaPods to static libs and move frameworks to Carthage wikimedia/wikipedia-ios#695

Merged

orta closed this as completed May 19, 2016

iblacksun mentioned this issue Oct 9, 2016

Dyld Error Message: ccgus/fmdb#482

Closed

wmathurin mentioned this issue Jun 14, 2017

Upgrading to 5.0.1 from 4.3 resulting in crash trying to open smartstore forcedotcom/SalesforceMobileSDK-iOS#1986

Closed

orta mentioned this issue Jan 25, 2018

Better for individuals jamiebuilds/open-source-by-default-manifesto#8

Open

This comment has been minimized.

Sign in to view

artsy locked as resolved and limited conversation to collaborators Sep 24, 2018

dyld crash at launch #1246

dyld crash at launch #1246

Comments

alloy commented Mar 10, 2016

TL;DR

Info about the errors

Known affected versions

iOS versions

dyld versions

Device versions

Retracing

Caveats

Conclusion

Status

alloy commented Mar 10, 2016

alloy commented Mar 10, 2016

0xced commented Mar 10, 2016

alloy commented Mar 10, 2016

quellish commented Mar 15, 2016

alloy commented Apr 1, 2016

alloy commented Apr 1, 2016

robinsenior commented Apr 1, 2016

alloy commented Apr 1, 2016

robinsenior commented Apr 1, 2016

alloy commented Apr 1, 2016

lumaxis commented Apr 1, 2016

alloy commented Apr 1, 2016

robinsenior commented Apr 4, 2016

alloy commented Apr 5, 2016

ctava commented Apr 7, 2016

alloy commented Apr 7, 2016

alloy commented Apr 8, 2016

robinsenior commented Apr 8, 2016

alloy commented Apr 8, 2016

maxmeyers commented Apr 8, 2016

alloy commented Apr 8, 2016

maxmeyers commented Apr 8, 2016

alloy commented Apr 8, 2016

robinsenior commented Apr 13, 2016

alloy commented Apr 13, 2016

maxmeyers commented May 11, 2016 • edited Loading

alloy commented May 11, 2016

maxmeyers commented May 11, 2016

alloy commented May 11, 2016

alloy commented May 11, 2016

SquaredTiki commented May 11, 2016

alloy commented May 11, 2016

quellish commented May 11, 2016

orta commented May 11, 2016

quellish commented May 11, 2016 • edited Loading

patters commented May 13, 2016

tsabend commented May 13, 2016

freshnitesh commented May 13, 2016

alloy commented May 13, 2016

freshnitesh commented May 13, 2016

UnsafePointer commented May 15, 2016

mergesort commented May 16, 2016

gparker42 commented May 19, 2016

orta commented May 19, 2016

alloy commented May 19, 2016

zenangst commented May 21, 2016

wolffan commented Sep 23, 2016

orta commented Sep 23, 2016

This comment has been minimized.

maxmeyers commented May 11, 2016 •

edited

Loading

quellish commented May 11, 2016 •

edited

Loading