Input dispatching timed out ANRs & ANR rate survey #29
Comments
|
App with Air 27, Starling 1.8, Feathers 2.x |
|
Hi, Our App(Starling 1.8-AIR 26/21): ANR Rate : 4% 39 reports last 7 days: Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 (has extras) } "main" prio=5 tid=1 Native 10 reports last 7 days: Input dispatching timed out (Waiting because the touched window has not finished processing the input events that were previously delivered to it.) "main" prio=5 tid=1 NATIVE |
|
Thanks to both of you! Since both apps have such different rates, could you give a little extra info?
|
|
Do you have little or a lot of user inputs ? |
|
Thanks @Seb-AS, will be interesting to compare with @skolesnyk's app regarding the amount of user inputs. |
|
Scrolling lists, tapping on Sound Player play/pause buttons, text entry. |
|
App with Air 25, Starling 2.2 Broadcast of Intent { act=android.intent.action.SCREEN_ON flg=0x50000010 } App with Air 25, Starling 1.8 Broadcast of Intent { act=android.intent.action.SCREEN_ON flg=0x50000010 } Do you have little or a lot of user inputs ? |
|
Thank you @MathiasFabian for sharing! @PrimaryFeather this issue seems to be plaguing a lot if not just about all AIR apps. Could you get someone from the AIR team to look into this? |
|
App with Air 22 and Starling 1.9 Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 5. Wait queue head age: 5502.9ms.) Do you have little or a lot of user inputs ? |
EDIT: The Google Play Console was being extremely flakey – our most common ANR error was in fact
We can replicate the ANRs but it takes a while and they seem to happen randomly. We're completely flummoxed. The ANRs can happen on a completely static screen with only very occasional input. I suspected a memory leak but Scout says no. |
|
Concerned our ANEs are mentioned here, if this is related to us we will chase down the issue immediately, though I'm fairly confident it's not. Is anyone having this issue without distriqt's ANEs? |
|
@marchbold we use Milkman ANEs, I don’t think it’s related to your ANEs in particular, my gut feeling is an issue in the Android/AIR input handling. |
|
Yeah, that's my feeling too, just was a little concerned when saw our extensions mentioned several times. We did actually do a thorough review of our ANEs to identify any potential causes when we encountered this issue in our apps and couldn't find anything. Our largest app:
ANRs are mainly due to two causes:
Similar logs to the ones above. We use Fyber Ads ANE Crashes are mainly related to Fyber:
|
|
Our most common crashes over last 60 days:
|
|
We are also seeing very similar ANR rates and similar causes:
Here are some stats on our app:
We also don't think that these ANRs are extension related. When we have had some other ANRs caused by extensions we were able to see a clear stacktrace to the extension code. Do you have little or a lot of user inputs? Typically 1 or 2 taps every few seconds |
|
We're seeing ANRs and crashes above the bad behaviour thresholds in our app. Do you have little or a lot of user inputs ? Do you have any input intense situations like tapping or dragging on the screen? Do you have ANEs included in the app (notably ads) ? Typical ANR reports: Typical crash traces: |
|
Hi, Thanks, |
|
Hi @Ankit-Adobe, Thanks for joining the conversation. There is already a bug in the tracker here: https://tracker.adobe.com/#/view/AIR-4198519 We have tried to reproduce the bug but without success so far. Could you give us more information on when/how this error is triggered? It always seems to originate from com.adobe.air.customHandler.callTimeoutFunction |
|
Hi guys, we are having the exact same errors on all our apps, we have also updated the forum topic (https://forum.starling-framework.org/topic/anrs-crash-rate-survey?replies=5#post-109884) and voted and commented on the tracker url provided. • ANR rate: 3.60% Thanks, |
|
Just a note to say i made an edit to my original post above (the main ANR error was wrong) |
|
Just a thought (actually an idea of Robby Campano): could it be that the ANRs we are seeing are the result of users hitting the screen while the apps are restoring from a context loss? After all, this takes some time, and the textures are restored synchronously, i.e. all input will be blocked. I could switch the texture restoration to being asynchronous, but it's been reported that this is not entirely reliable on some Android hardware. Could be worth a shot, though — this requires only a small change inside the AssetManager (I think). Then I could make this an "opt-in" flag. Any comments? |
|
As a matter of fact I was just testing that, these are the results: Turns out that, -when the user locks the device- the app has a harder time recovering from a context loss when the app is in foreground rather than in background. At least it took significantly more time on my device (Samsung Galaxy S6). If I send the app to background, then lock the device, unlock it back, and bring it to foreground again the restoration is still visible, but it takes none to very short time. When I try it the other way around (app was in foreground when the device got locked), while the app is restoring, I can still play around and click stuff when, according with what you just said, all the input should be blocked. I have to say though, that I can't really tell an exact restoration time for each example, since sometimes my device takes longer because, I don't know, it doesn't like my face :) I'm not entirely sure if I'm missing something but I can definitely mess around with the app when recovering from the context loss. That said, I still haven't been able to reproduce the ANR itself on my phone while debugging. |
|
Thanks for the additional info, Christian!
You're totally right: it depends on the size of the textures being restored. I imagine it takes a while loading a 4k*4k texture, for example; and during that time, input will be blocked. Loading several smaller ones should not be that much of an issue, because there's a gap for AS3 to process stuff between each upload process. |
|
I see, perhaps what I'm about to suggest isn't doable, but what if the whole Stage was left untouchable as soon as the restoration begins, and it gets back to touchable as soon as the |
|
Hm, no, that wouldn't help. Since it's a synchronous operation, all of AS3 stops accepting input — and that includes the mouse events of the classic Flash (which Starling otherwise queues). The only solution would be to avoid that synchronous (time consuming) operation. Again, I don't know if that's really the issue — it's just a possibility. |
|
I do not think this is related to context loss. ANRs occur during gameplay periods according to our user reports. However it seems to occur after a period without user inputs: Case 1: match 3 game, users report freezing often after waiting to think on a move (5s maybe) |
Aha, that's useful information — thanks! |
|
Cool. Have fun and goodbye :) |
|
By the way for everyone else: Air SDK 33 is available for distribution now directly from Harman. We are making good progress towards a 64 Bits release and did not find any errors so far. As soon as the last remaining ANE has been converted we plan to release 64 bits updates on Google Play. |
|
@rewb0rn that's good news. Did you also see any improvements in performance? |
|
Can't say anything about that honestly, we are in the middle of rewrites for the missing ANE and I haven't paid attention to performance before. I will update once I have some clarity. |
|
Has anyone been able to submit to Google Play since August using Adobe AIR ? I am currently dead in the water with a very large game (3 years in dev) and can't submit to Google Play due to the 64 bit requirements. I have emailed Harman to get on the Beta for Air 33 (Which I am hoping supports 64 bit) and to be able to submit to Google Play. Anybody successfully submit an Adobe Air game lately? Thanks for any updates or help on this subject !! |
|
@SHDgames new games submitted have to support 64 bit since 1st August, so you'll have to use AIR 33. Updates however are allowed with 32bit only until sometime 2020. |
|
@sweetnitro Thanks -- but is AIR 33 available anywhere? I read Harman was supposed to have a beta out in June, and haven't found anything anywhere ... The best I got was to email the adobe support team with Harman for a BETA (emailed 2 days ago) and I haven't heard back.. just wondering if this is the best practise at this point. Thanks guys - I have high hopes still that this can be resolved! |
|
Yes Air SDK 33 is available already, but unfortunately only from the email address you have found. I guess they have problems catching up with all the emails they receive, but you should get access to the SDK after you have agreed to their terms and services (you have to write them an email where you state that you accept the terms of the SDK). Hopefully they will have a website set up soon, so this whole thing becomes a bit more transparent. |
We noticed that using Loader:loadBytes() instead of Loader:load() prevents the ANR caused by click during asset load. |
|
Thanks for those - can I please check the architecture and the exact version of AIR that you've got there? It looks 64-bit but didn't match the addresses in our binary for ARMv8.. |
ghost
mentioned this issue
|
I have also mentioned this issue for some more information |
|
Hi, the following is also our top#1 ANR: Is anybody certain that this is indeed originated by playing sounds? If that's so I'd give @marchbold's Distriqt Multimedia ANE a try. [Edit] Also, in case anybody missed it, here's the bug tracker url |
|
We've been seeing this issue increasingly since in the last 10 days. Our app is using version 33.1.1.476 of the Harmon/AIR SDK sdk. We're getting users reporting that the app freezes whilst showing the Launch screen and other crashes. I'm not sure whether an anonymous function is missing to handle the timeout dispatcher or what is happening. The errors I'm seeing are the same as other peoples. They are more likely to occur on Android 11, then 10 and then 9. I don't know if this is reflective of the percentage of the audience using those devices. |
|
Hi @PhilCase - there's not a lot of changes that have gone into the Android side of things recently, just some simple changes to handle some null references that seem to be happening on some Android devices where the activity lifecycle model isn't working as we would expect... Do you have any call stacks or debug logs from any of these issues and we can look to see if there's an obvious root cause? thanks |
|
Hi @ajwfrost We're still getting ANRs unfortunately, with 33.1.1.575. We didn't try the 33.1.1.633 on Android. Is there any advice for tracking down what is causing them please? Distriqt's Exception ANE looks like it might help if I can set an exception handler up for the entire app. I'm not sure what I can do if the error isn't starting in our code. Their JobScheduler ANE references here saying that it was "developed for usage in resolution of the Adobe AIR Android ANR issue, as discussed" but I don't understand how if I'm not cancelling a scheduled job. |
|
Hi @PhilCase ANR01: processing some ActionScript - within the interpreter, rather than as JITted code. Which may indicate it's a class initialiser, or it may just be the first time the function has run. Lack of "AS stuck" thread makes me think this might be during start-up? So from what we can see from these logs, the application doesn't actually seem to be 'stuck' in a static state i.e. no hint of deadlock/livelock problems caused by multithreading. So it may just be code that's taking too long to run: if you try running this with a connection to Adobe Scout, it would be worth seeing whether you spot any glitches i.e. frames where it suddenly took a lot longer to process. If the problems are related to the start-up processing, then it may help if we can get AOT working for Android... We'd started creating an ANE that could further work out where things may be getting stuck, but it's probably not the best idea to add extra code into the initialisation here - we'll look again at how best we can build in some metrics/diagnostics. thanks |
|
Hi @ajwfrost , regarding the ANRs. Our users have indicated that the app gets slower over the days that they use it for. We've found (experimentally) that if they clear their Application Data (Sandbox) or just the "application storage directory" (which is still in the sandbox but isn't the full sandbox) that the app will return to full responsiveness. Our application storage directory contains files like an sqllite database and files installed by the app to work against. I've done some research and have seen note that a fragmented sqllite database can cause an app to become less responsive and that on a flash based file system the database might not be copied to contiguous blocks of storage when it expands in size. We're wondering could that be a cause of the issue and whether it is something that you can see in the logs? |
|
Also - I've had little success in getting Adobe Scout to run. It mentions a "debug player". The app appears to be out of date and complains about running on an Android 11 device. the support pages on the Adobe web site for it have been archived and removed. I've had no luck with the profiler in IntelliJ IDEA either. Is there any advice on getting either of these running please? |
|
For Scout; you should be able to download the Windows or MacOS client application, from the below link: But the companion app for Android will no longer work because of the folder access changes.. so you would need to create a telemetry config file and include this as part of your APK file. If you open Scout, it will automatically generate a telemetry config file, in your home folder e.g. Not sure about the profiler in IDEA but again it would need the debug builds (apk-debug, or aab-debug) and potentially then it can connect to the debugger. You may be able to provide the IP address of the local/developer machine when the app first opens... I'll try to get details on all this stuff published onto the airsdk.dev website! thanks |
|
Thanks Andrew. That's a massive help. I couldn't find any documentation at all on the Adobe web site for it. |
|
@ajwfrost do you have any thoughts on my other thought about why clearing the ApplicationStorageArea would increase performance in the app? |
|
@PhilCase sorry I'd missed those parts ... that's quite interesting, I can't see why the number of files within the application storage directory would have any effect on performance, it's not like there's a listing of these that happens during start-up.... we can have a look and do a bit of profiling here perhaps. Plus - we will definitely look into this issue of a fragmented sqlite database, I'll see whether we can do something about that! thanks |
|
Thanks Andrew. The "fragmented database" might not be what's causing our slowdown, so I can't say whether it's a dead end or not. Here's the article I found that mentioned it. (Sorry I don't have the hyperlink any more). I know our database is encrypted also (which sounds like overkill if we're running in a secure sandbox folder). I have seen a number of places that recommend clearing the cache or clearing the App's data directory as a step towards problem resolution. Do you profile using Scout also? Cheers. |
and others
Hi guys,
I am mirorring my post from the Starling forum here with Daniel's permission.
We have ANRs on Android in all our apps and would like to know if it's maybe our apps particularly, or if it's AIR related.
Could you please check your Google Dev Console and go to "Android Vitals" > "Overview".
The first section is "Bad Behaviors".
Please share the following values (no need to mention the app, just pick the biggest app you have or the average of all your apps please):
The main ANR we are experiencing is "Input dispatching timed out" triggered by "com.adobe.air.customHandler.callTimeoutFunction". There are heaps of topics on it on the web, and they only proper response by Adobe is that we should avoid doing heavy calculations on user input.
But all our apps use Starling, and Starling defers all user inputs to next frame so this should not be an issue.
The text was updated successfully, but these errors were encountered: