Figure out how to deal with Doze (Android 6+)

[cketti]

Doze reduces battery consumption by deferring background CPU and network activity for apps when the device is unused for long periods of time.

Learn more about it here: Optimizing for Doze and App Standby

As far as I can tell we have these options:

1. Embrace Doze. When the device tells us to go to sleep we do.
2. Fight Doze. Use the new API to schedule checks for new mail (can't be more frequent than every 15 minutes)
3. Evade Doze. Foreground services (think music player) prevent the device from entering Doze. This requires displaying an ongoing notification.

As far as I can tell users can't disable Doze in settings. They have, however, the option to put apps on a whitelist.

An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.

So the interval between mail checks still has to be at least 15 minutes. But there is a chance that Push might work during Doze. Keeping a connection open doesn't require alarms.

My goal is to first collect all options we have and then decide on the best way forward. Please let me know if you can think of any other options and I will add them to this post. Also, if you have a special use-case regarding mail checks that you think should be supported by K-9 Mail let us know.

Update (2015-11-10): It looks like the only way for an app to prevent the device from entering Doze is to keep the screen on. So option 3 is not something that's feasible for us.

Update (2015-11-20): Using a foreground service is possible but tricky due to a bug in Android. See https://commonsware.com/blog/2015/11/18/another-doze-edge-case-foreground.html

[sidamos]

High priority GCM notifications do work normally during Doze (that does not help K9, of course), so Hangout messages do work. For me, mails are as important as Hangout messages. So, doing nothing about Doze (like Gmail does not fetch IMAP) is not on option for me.

I think, push should work if the user whitelists the app (this setting allegedly helps WhatsApp). If the user enables push, K9 could ask the user to put the app on the whitelist (there is an API for that). I have not tested that, because I do not use push.

For periodic mail checking, K9 should use the new AlarmManager methods. K9 should have an option to do that. This still allows the device entering Doze mode and saves battery. I would not call that "fighting Doze".

Having an ongoing notification is annoying IMHO and a mail app should not prevent the device from going into Doze at all.

[ssieb]

If whitelisting the app is required to make push and/or poll to work, then I'm fine with that. As mentioned above, there should be some notification to the user that it's necessary to do so.

[NadirHusain]

Does anyone know for sure how much " inactive " time needs to elapse before doze mode kicks in ?

[cketti]

@NadirHusain: As far as I can tell OEMs are free to tweak the different timeout values. See DeviceIdleController.
From what I've read current Nexus devices enter Doze after 1 hour without significant motion.

[NadirHusain]

I agree with @sidamos . Doing nothing is not an option. Since I am a push user ( and perhaps driven by that) I feel push should be a top priority. Alarm manager should be used to fetch . Sorry I can't code so really can't contribute in any tangible way except just to put forth my opinion. Thanks

[gdt]

Doze mode seems like a reasonable concept, except that it's trying to push people into GCM (which is non-Free and has privacy issues). I would suggest going down two paths simultaneously

• Try to be a good doze client. With polling, just allow the polling interval to get longer as doze requires it. That might be as simple as setting an alarm for now+poll_interval on every wakeup and not worrying that it's longer than configured. This is not really so crazy, as the idea that one needs rapid polling when the device is really idle only seems to fit a use case where one leaves the phone on a table in a room and stays there for hours. I guess it will be interesting to see logs of real people and if their phones doze when they are not asleep or really away from them. Also with this, intercept some done_dozing signal and poll again if it's been longer than the configure interval. Perhaps also set some doze-mode poll interval, so the user can say poll every 10 minutes, but only every hour in doze mode, if that's what they want. Or maybe that's too complicated.
• Make push still work. It seems we really don't understand enough yet, and whitelisting is the only way. Somewhat eparately from this, it would be nice if push could have much longer times in IDLE, to make k-9 use less power in doze mode.

What I don't see being all that useful or important (to me) is enabling polling in doze more often than every 15 minutes. I suppose that could be important to people who can't use push, need to get mail promptly, and leave their phone on a table while being near it. It would be interesting to hear from anybody who sees themselves in that situation.

For me, if my phone goes into doze because I haven't touched it and it hasn't moved in over an hour, having mail arrive up to 15 minutes late is ok. Unless I am mis-guessing how often doze will happen.

[sidamos]

Periodic check:
You don't have to implement, what you are describing in your 1st paragraph, Android does that already.
If you use the old AlarmManager methods, they won't fire during Doze, unless the device enters the maintenance window (every hour or more).
If you use the new AlarmManager methods, alarms are being throttled to fire every 15 minutes at most.
My use case is this: During work, the phone lies on my table for longer periods of time. But I don't want to get mails only every hour or even later.
Also, Doze is supposed to end, when you move the device. But in my test, it only ends if I switch the display on. And I don't want to switch the display on every now and then only to enable K9 to look for new mails.
So, K9 should use the new AlarmManager method, if the user desires to get mails during Doze.

Push:
If we are lucky, push should work as normal if K9 is on the whitelist. So, nothing has to be implemented for this. Optionally, K9 could ask the user to be put on the whitelist. There is an intent for that.

[NadirHusain]

As an aside I would like to mention that I only get real push if my " refresh idle connection" is set to 6 minutes or less. Any higher number and my messages are delayed. I understand my cellular operator is most likely responsible for this but was just wondering if this option would still be available to me in the doze mode ? I am sure there are others like me and k9 needs to be able to " force " this incase doze " kills " this option in default mode . Thanks

[cketti]

My current plan is:

1. By default we embrace Doze. If Push is enabled we disable it while Doze is active. That will prevent the app from establishing a Push connection during the Doze maintenance windows.
2. If K-9 Mail was whitelisted by the user we use the new API for our alarms. That should allow poll frequencies of up to every 15 minutes and Push should work, too. The Push refresh interval can't be smaller than 15 minutes though.

[NadirHusain]

as of now K9 5.107 android 5.1.1 i need a minimum refresh time of 6 minutes to be able to get real time push notifications (tcp/cellular service provider issue). setting the minimum idle refresh time at 15 minutes would mean end of push mail for me (as i know it) unless there is a way to set idle refresh time at 6 minutes or less when doze mode is NOT active and could switch to 15 minutes when doze kicks in. i am sure there are many others like me who need real time push and have idle refresh set at very low intervals . request you to consider this use case issue . thanks

[cketti]

@NadirHusain: Doze is only available on Android 6+. If Doze is not active (or on previous Android versions) K-9 Mail will not impose any new restrictions.

[NadirHusain]

@cketti clear now . thanks .

[sidamos]

Good idea. This should have been done by Google, without the need to call new AlarmManager methods by the app.

[imsaguy]

Any progress on this? If I have to look at K9 periodically just to get my notifications, I might as well set my sync to manual which defeats the whole purpose of push notifications. If you're looking at ways to bypass, do what apps like KeepPass do and set a transparent notification icon which keeps the app in the foreground like a music app and prevents doze.

[daradib]

@imsaguy For now you can manually whitelist K9 under Settings > Battery > Battery Optimization (or similar). Mostly fixes push and #970. Doze still partially applies, since current AlarmManager will only fire during increasingly spaced apart maintenance windows, so I think the IMAP IDLE connection may eventually time out if there is no traffic and the device is stationary long enough. Also, to be specific, the foreground service prevents App Standby but not Doze.

[gadgetfan10]

@cketti So under the "embrace Doze" plan, is the idea to stop push connections during Doze but - and this would seem to be the difference from the user end - re-enabling push connections when the device comes out of Doze?

Sort of separately, and maybe because I don't understand the specifics, why would this approach prevent the establishment of push connections during the Doze maintenance windows?

[cketti]

@gadgetfan10 Poor wording on my part.

Embrace Doze:

• Close existing Push connections when entering Doze; don't attempt to create new ones while in Doze or maintenance window
• During maintenance windows we trigger a simple mail synchronization
• Return to normal operation (e.g. establish Push connections) when exiting Doze (device returns to normal operation; not just Doze maintenance window)

[gadgetfan10]

Sounds like a solid plan. Thanks for the clarification.

Looking forward to seeing it implemented. If I had programming skills, I'd offer to help. Since I don't, I'll just say thanks for your efforts.

[doodhout]

@cketti As I understand how this will work: is there no way to enable doze and keep push connections alive during screen-off? Right now, I'm using a Sony smartphone that features similar functionality called 'Stamina Mode' and by whitelisting K9Mail I am ensured the K9Mail app works as if Stamina Mode hasn't been enabled - other apps that are not whitelisted will still be forced to go to sleep and ceize their activity.

Doze seems to be more refined than Stamina Mode, as it offers an API that allows apps certain activity but still well controlled and more battery-friendly than if the feature was disabled altogether; which isn't the case for Stamina Mode.

[bobpullen]

Just a note to say that this issue is exacerbated on Android N where Doze 2.0 is initiated on screen off rather than a perceived period of inactivity.

[LosD]

@jimimaseye How do you get into the real Android battery settings? On my S8+ (SM-G955F, FXXU2CRF7 software version, NEE region), Samsung seems to have ripped them out, only leaving their own... Which makes it impossible to disable optimization, only leaving it as unmonitored, which apparently doesn't help by itself.

[jimimaseye]

@LosD

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

That's the method I am using.

[LosD]

@jimimaseye

Holy hell, they hid that well. Not even a search found it, so I thought it was in the standard/old Android battery settings. Thanks!

[coogor]

Just for the records - after my S7 Edge was updated to Android 8, everything works as expected again!

[perpetual-hydrofoil]

Under Android 9 ("Pie"), another place to check is: Battery Optimization under Settings (not under Apps), change the dropdown menu from "Optimized Apps" to "All Apps", and change K-9 Mail to "Don't Optimize". Don't forget to restart K-9 (or your entire phone) if it doesn't seem to take effect.

[Draghtnod]

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

This is horrible! After updating to Android 8 my Push-Notifications didn't work anymore and i started searching. The server logs told me that my phone closed the connection after a few minutes of Screen-Off time. Turning off this optimization solved the issue.

You can't expect users to find such a well hidden menu. DAVdroid does turn off this feature for it's users when first starting up. Maybe K-9 should do the same?

[marmistrz]

You can't expect users to find such a well hidden menu. DAVdroid does turn off this feature for it's users when first starting up. Maybe K-9 should do the same?

Conversations does the same.

Btw. one should really create a library to deep link into some vendor-specific app-killer settings (for confusion they're named battery savers). That's also what Conversations does on some phones

[Michele64]

Turning off battery optimization does solve the issue for some hours, after that the settings revert to optimize. which is quite annoying as push doesn't work anymore

[misaligar]

Same as @Michele64. Turning off battery optimization on my stock OnePlus 5T does not solve the problem. K-9 shows "Syncing disabled" message just below where it says Accounts.

Background sync option in K-9 is set to Always.

[Newbie1980]

After how long does this occur? What OS is it. Thanks

[jimimaseye]

As i said above:

.
.
2, 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
3, 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

(Or whatever your phones equivalent is)

[lukefor]

Can I suggest if/when this gets looked at, that the use case of no background syncing is supported, and that no workarounds or other background activity/listeners are used in this case, regardless if the phone is awake or asleep

[beerisgood]

Any news? Problem still exist and nothing helps

[deutrino]

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like *checks* nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?

Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.

Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

[trostc]

Not having issues on Android 9 here (Galaxy S9) please no persistent notifications. HATE those

…

On December 4, 2019 10:33:25 AM CST, Deutrino ***@***.***> wrote: Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like \*checks\* nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems? Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it. Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #857 (comment)

[LosD]

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like checks nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?

Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.

Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

@deutrino The functionality is restored by simply telling Android not to optimise it. Maybe better instructions for how to do that is needed, but putting a persistent notification for a problem that can be solved quite easily by the user seems overkill.

[utack]

davx shows a popup that allows you to easily disable battery optimization for it on first start, maybe you could just copy the code?

[Newbie1980]

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like checks nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?
Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.
Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

@deutrino The functionality is restored by simply telling Android not to optimise it. Maybe better instructions for how to do that is needed, but putting a persistent notification for a problem that can be solved quite easily by the user seems overkill.

Dude have you even read the whole thread this does not fix it for everyone. This thread is years old at this stage and no fix. Its hit or miss from device to device and rom to rom.

[Michele64]

On oneplus devices k9 gets killed and/or removed from battery optimization at random intervals. Eventually I gave up and switched to FairEmail.

[LosD]

@Newbie1980 If you look, it was in it. And it may very well be that some phones (it seems only OnePlus is left) has some weird thing where it disables optimization. For those, it could be made it an option. But there is no need to make a terrible UI for everyone for that.

[tuxayo]

In the hope it helps, here is a collaborative resource made by other app devs to try to list the issues with each manufacturer. And what devs can do to fix, hack, workaround them.

https://dontkillmyapp.com
https://dontkillmyapp.com/problem

[adamf663]

K-9 is very good at disabling itself. How about periodically checking if sync disable is still necessary? If I can restore sync by going into the 'fetching settings', clicking next, and exiting that setup menu, then k-9 should also be able to restore syncing. When k-9 is in the foreground, there is no good reason not to perform this check.

I shouldn't have to let k-9 run loose and chew through my battery to have it function properly when it is in the foreground.

[VA1DER]

I just ran into this problem on my new FP3 with /e/ (Android 10). Interestingly, as this is supposed to be an Android 6+ issue, I never had a whisper of a problem on my Galaxy S5. From the FAQ:

If you are using Android version 6 or later, you are also likely to suffer from problems related to the “Doze” battery saving feature. For more information and potential workarounds see ticket 857 in the K-9 GitHub repository.

This may sound a little bitchy, and for that I apologize, but for the FAQ where people go for answers to simply shrug and ask people to wade through five years of comments in a bug report is ++unhelpful. It can do push, it can't do push. It can for for an hour. Two hours. Three. If you set these arcane settings it might work longer, but probably will still stop. These comments, collectively, are incomprehensible. For people's sanity and to alleviate frustration, please, someone who is an SME needs to update the actual FAQ page with a clear concise answer as to the current state of affairs here and the best remediations.

[Newbie1980]

Unfortunately this issue was never fixed. I gave up and (reluctantly) just use signal now.

[jimimaseye]

I am still issuing k9 on my android 8.0 Samsung having followed the process I outlined here: #857 (comment)

Ultimately, this is android functionality (specific to your device) and you can use the android settings and options it offers to deal with the android fubctionality that's causing you a problem. 'Doze' or 'auto sleep' (or whatever you want to call it) is a battery saving function which affects all apps and its persistent state of 'background' functionality if you let it (leaving each app to suffer or be compromised depending on the individual app). It makes sense to visit the methods offered by the OS and use them accordingly to suit you and your app. I agree that if the flavour of OS doesn't offer you the opportunity to 'exclude' and feature that otherwise compromises an apps functionality the it is a failure and oversight on the design of the OS and will inevitably lead to frustrated users or even loss of custom.

[Newbie1980]

I am still issuing k9 on my android 8.0 Samsung having followed the process I outlined here: #857 (comment)

Ultimately, this is android functionality (specific to your device) and you can use the android settings and options it offers to deal with the android fubctionality that's causing you a problem. 'Doze' or 'auto sleep' (or whatever you want to call it) is a battery saving function which affects all apps and its persistent state of 'background' functionality if you let it (leaving each app to suffer or be compromised depending on the individual app). It makes sense to visit the methods offered by the OS and use them accordingly to suit you and your app. I agree that if the flavour of OS doesn't offer you the opportunity to 'exclude' and feature that otherwise compromises an apps functionality the it is a failure and oversight on the design of the OS and will inevitably lead to frustrated users or even loss of custom.

Dude almost every one tried changing doze and optimise settings etc. It makes no difference

[jimimaseye]

It is true that I don't have the other versions of Android devices (I am using Samsung's implementation) but I do wonder: if (and i mean "if") the option is there to exclude an app from doze or auto sleep then why is it not working for you? I can only assume that the necessary steps have not been taken in the right way. Or if you have, as i say, the failure is with the OS. It doesn't seem correct to then make changes to an individual app to get round a flaw in a particular flavour of operating system because that too might then have adverse consequences on other platforms. For example if you make a change to the app to always keep it alive to get around this problem, then you will affect someone else's choice to put the app to sleep as the operating system would allow if set to do so (for example on my Samsung).

[Entered by mobile. Excuse my spelling.]

[cketti]

The current state (K-9 Mail 5.737 [beta]):

The app uses WorkManager for periodic mail checks. During Doze the check interval might be longer than configured. Putting K-9 Mail on the "Don't Optimize" list should avoid that.

For Push (IMAP IDLE) we use a foreground service to keep the app alive. This will display an ongoing notification in the status bar. To learn how to disable that notification see How to configure Push?