runtime: exceeded thread wakeup limitation on iOS

[hajimehoshi]

We found my gomobile iOS application crash with the below message:

Event:           wakeups
Action taken:    none
Wakeups:         45001 wakeups over the last 64 seconds (704 wakeups per second average), exceeding limit of 150 wakeups per second over 300 seconds
Wakeups limit:   45000
Limit duration:  300s
Wakeups caused:  45001
Duration:        17.04s
Steps:           18

Hardware model:  iPhone8,1
Active cpus:     2
Boot args: 

I guess Go runtime switched threads so many times that iOS complained, but I am not sure. It is hard to create a minimal code to reproduce this problem, but I'd like to report this (potential) problem in gomobile.

[eliasnaur]

Please do try to figure out a way to reproduce the error; fixing the problem will be much more likely if you do.

[eliasnaur]

I found a thread with a similar issue: https://forum.unity.com/threads/the-weird-ios8-unity3d-hiccup.273186/. It seems that Unity apps don't actually crash, only freezes while the exception report is generated.

[g3devv]

Hey I want to work on this issue since I "translated" my app to go - and now I face problems here.

I found this thread at https://developer.apple.com/forums/thread/124180 and user eskimo has some insights about what to look for.

In short, it seems to count the number of times that a blocked thread was unblocked because of an ‘interrupt’

At first I tried to reproduce the issue in a minified example by doing thousands of Gosched() calls in multiple threads - unfortunately without success. My next attempt was mixing up some sync.Mutex .Lock() .Unlock() calls in multiple threads. This has lead to a lot of context switches in the Xcode profiler, however it has only produced reports for cpu_resource ("90 seconds cpu time over ...") but not the desired wakeups_resource report.

Now I'm kind of out of ideas what to look out for there, I'm trying to think of other design patterns that I use in my main project that could potentially fall into that "thread unblock or interrupt" territory but the Gosched and the Mutex calls are the only things I can come up with now. Maybe also channels but my understanding of these is quite limited

@hajimehoshi did make any further findings back then?

@eliasnaur has something like that ever been brought up again? Any info that could help me showcase a minimal example of the issue to debug or fix?

[hajimehoshi]

I have not seen this issue recently.

[g3devv]

@hajimehoshi I do 🙈 Is the project where you have experienced the problem a few years ago open source by any chance or would you open-source it (of course only if it has no further value for you that would be destroyed by open sourcing)? Maybe that would make it easier to find the reason for the problem

[hajimehoshi]

I'm afraid it is not an open source...

[g3devv]

Oh too bad but don't worry about it, if at any time you remember something you stumbled across when debugging please let me know. And maybe Elias knows something too haha

[g3devv]

It's the channels. My example project is quite messy, I am going minify my example project and let you know. So happy I found a way to reproduce the error :)

[g3devv]

Please do try to figure out a way to reproduce the error; fixing the problem will be much more likely if you do.

@eliasnaur I have just made a project that reliably produces that error. https://github.com/g3devv/channelwakeups

It would be very exciting to hear from you if you have an idea how we could approach getting this resolved in gomobile

[eliasnaur]

This is unlikely to be a gomobile issue, and more likely to be an iOS limitation that affects Go runtime scheduling. If you have a reliable reproducer, the Go runtime team may be able to help. I can come up with one clarifying question: is this still an issue on the very latest iOS version?

CC @golang/runtime .

[g3devv]

Dear Elias, thank you for your response!

I completely agree that this very well may not be an issue with gomobile intrinsically, however I fear that some "specialties" that Go comes with make this issue more likely to come up in a Go app compared to native apps or apps developed with other frameworks.

What I mean by that (and I'm a complete novice when it comes to golang so maybe it's a bad example) is that for example I once read that Go is super fast at context switching and for that reason does it at a higher frequency than other language runtimes do.

So maybe, if we are able to find a singular "feature" to blame, it would be possible to pass a build flag (or compile flag or what it's called) to optionally disable (or throttle) such a feature; in a way that the generated framework is less likely at hitting these iOS restrictions.

Over the weekend, I was able to come up with a more concise example (5 lines of code) to reproduce the issue, which should make it easier to pin down the underlying cause. The error rate with the new code is still a bit temperamental (either because it's not hitting the nail OR iOS doesn't emit the crash info on each run). Once I was able to confirm my new demo is better, I will ping you again if that's okay with you.

About the iOS version, my production app showcased that issue on 15.2.1 and I currently test my demo project on 14.3 but I will update the phone tomorrow just to make sure.

[eliasnaur]

You're welcome to ping me, but I'm unlikely to be able to do much more for this issue.

So maybe, if we are able to find a singular "feature" to blame, it would be possible to pass a build flag (or compile flag or what it's called) to optionally disable (or throttle) such a feature; in a way that the generated framework is less likely at hitting these iOS restrictions.

It's very unlikely an explicit flag will be accepted by the Go team, especially for a workaround for what seems to be a misfeature of a secondary port. What may be possible is that the Go runtime could somehow automatically limit thread switching when runtime.GOOS == "ios".