Sustained high CPU usage when saving

[sobri909]

My app is being killed by iOS multiple times a day due to sustained 100% CPU usage while Parse is saving objects. This happens with both saveAll and saveEventually.

I haven't hunted through for the exact parts of the Parse SDK code that are causing it yet, partially because the crash reports are a hassle to get to and symbolicate (those resource limit kills are only available via Xcode's device logs thing, and never make it to services like Crashlytics).

Ironically, my app can run for days uninterrupted in the background if the device doesn't have internet. The saveEventually calls will be queued up nicely, using effectively 0% CPU, and then will execute once the device gets connectivity again. But if the device has connectivity, the app is guaranteed to get killed at least once or twice a day while doing saves in the background. Which is unworkable for an app that needs to be running in the background 24/7.

I'll eventually get round to symbolicating the crashes and providing more info. But I first wanted to check to see if this is a known issue, before wasting too much time digging into it myself.

[sobri909]

Okay so I've done a bit of digging with the profiler on a headless build (ie no view controllers loaded, only the background long running tasks active).

The app is spending almost 100% of its CPU time in here:

[PFCommandCache.h _cleanupDiskCacheWithRequiredFreeSize:]

Each new call to that method fires off a burst of 100% CPU usage that lasts for at least 10 seconds (on an iPhone 6s).

I'm still not convinced that this is the cause of the background resource limit kills. But it's at least a high profile (har har) target for optimisation.

Anyway I'll keep digging for a bit longer. It's not matching up yet with the resource limit crash dumps in Xcode. Those always show a last 60 seconds call stack dominated by CFNetworking calls. So those may still be unrelated.

[nlutsenko]

@sobri909, this is awesome! Keep these reports coming!
Am going to look into this either over the holiday or just right after it!

Off-the-top of my head - we are using an enumerator for the disk path, which is somewhat suboptimal and is quite expensive - there is a better way to do that. Also, it looks like there is a bug in how we save new commands and not await for the old ones being cleaned up.

[sobri909]

Thanks @nlutsenko :)

A colleague told me that the way I'm doing it in SGImageCache is also highly inefficient / expensive. I thought he'd gone ahead and fixed it, but there's no relevant commits to that file, so I'm not sure what the story ended up being there. (@staminajim if you remember what that was about, plz let me know).

I haven't had a chance to dig deeper into the background CPU usage yet. I discovered that the reason why my resource limit crash reports aren't symbolicated is because I'd excluded Developer from Spotlight indexing, and Xcode uses Spotlight to find the dSYMs. Oops. So I'm waiting for iOS to kill the app again, so that I can get a properly symbolicated report to look at.

[sobri909]

I've solved the CPU resource limit kills at my end by setting a limit to how many objects I have waiting on saveEventually() completion. The limit I've set is 30, which has allowed the app to run on multiple devices for 5 days without a single resource limit kill.

I don't know how many objects I previously had queued up waiting for saveEventually() to complete, because I simply wasn't tracking it. It's possible I might have had 100 or even multiple hundreds, if the connection was unreliable for an extended period.

My app currently creates a new location event object every 10 seconds, batches them up in the background, and calls saveEventually() on the current batch every 5 minutes. So that's 30 objects in a 5 minute batch, which is why I arbitrarily set the limit at 30.

I was also exacerbating the problem by doing accidental redundant saveEventually() calls on objects that had already been asked to save but hadn't reached their completion block yet. So if for example a batch of 30 objects had all failed to save within the 5 minute window, then the next queue flush call would call saveEventually() on those 30 again, as well as the 30 new objects.

Though fixing that derp didn't stop the resource limit kills. It may have merely made the resource kills slightly less frequent. Though I didn't wait around long enough to find out.

[sobri909]

@nlutsenko Thanks for the quick fix over the holidays! PR looks good :)

Once it goes live I'll remove my 30 object limit and see how it goes.

[nlutsenko]

Was just today, not over the holidays. :P
30 queue limit sounds good. The problem was with directory enumeration being made redundantly, and, as a result, for high number of files - it grows to consuming a lot of CPU.

Taking into account that every single command aka network request is not likely to surpass 128KB and the limit on the cache is 10MB bring us to this happening in worst case, fast path is going to work until you have 80 pending saves. And taking into account that we benefit from system file attributes caching now - it's going to still be faster in the slow path.