Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Figure out how to deal with Doze (Android 6+) #857

Open
cketti opened this issue Oct 23, 2015 · 173 comments
Open

Figure out how to deal with Doze (Android 6+) #857

cketti opened this issue Oct 23, 2015 · 173 comments

Comments

@cketti
Copy link
Member

@cketti cketti commented Oct 23, 2015

Doze reduces battery consumption by deferring background CPU and network activity for apps when the device is unused for long periods of time.

Learn more about it here: Optimizing for Doze and App Standby

As far as I can tell we have these options:

  1. Embrace Doze. When the device tells us to go to sleep we do.
  2. Fight Doze. Use the new API to schedule checks for new mail (can't be more frequent than every 15 minutes)
  3. Evade Doze. Foreground services (think music player) prevent the device from entering Doze. This requires displaying an ongoing notification.

As far as I can tell users can't disable Doze in settings. They have, however, the option to put apps on a whitelist.

An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.

So the interval between mail checks still has to be at least 15 minutes. But there is a chance that Push might work during Doze. Keeping a connection open doesn't require alarms.

My goal is to first collect all options we have and then decide on the best way forward. Please let me know if you can think of any other options and I will add them to this post. Also, if you have a special use-case regarding mail checks that you think should be supported by K-9 Mail let us know.

Update (2015-11-10): It looks like the only way for an app to prevent the device from entering Doze is to keep the screen on. So option 3 is not something that's feasible for us.

Update (2015-11-20): Using a foreground service is possible but tricky due to a bug in Android. See https://commonsware.com/blog/2015/11/18/another-doze-edge-case-foreground.html

@sidamos
Copy link

@sidamos sidamos commented Oct 23, 2015

High priority GCM notifications do work normally during Doze (that does not help K9, of course), so Hangout messages do work. For me, mails are as important as Hangout messages. So, doing nothing about Doze (like Gmail does not fetch IMAP) is not on option for me.

I think, push should work if the user whitelists the app (this setting allegedly helps WhatsApp). If the user enables push, K9 could ask the user to put the app on the whitelist (there is an API for that). I have not tested that, because I do not use push.

For periodic mail checking, K9 should use the new AlarmManager methods. K9 should have an option to do that. This still allows the device entering Doze mode and saves battery. I would not call that "fighting Doze".

Having an ongoing notification is annoying IMHO and a mail app should not prevent the device from going into Doze at all.

@ssieb
Copy link

@ssieb ssieb commented Oct 24, 2015

If whitelisting the app is required to make push and/or poll to work, then I'm fine with that. As mentioned above, there should be some notification to the user that it's necessary to do so.

@NadirHusain
Copy link

@NadirHusain NadirHusain commented Oct 24, 2015

Does anyone know for sure how much " inactive " time needs to elapse before doze mode kicks in ?

@cketti
Copy link
Member Author

@cketti cketti commented Oct 24, 2015

@NadirHusain: As far as I can tell OEMs are free to tweak the different timeout values. See DeviceIdleController.
From what I've read current Nexus devices enter Doze after 1 hour without significant motion.

@NadirHusain
Copy link

@NadirHusain NadirHusain commented Oct 24, 2015

I agree with @sidamos . Doing nothing is not an option. Since I am a push user ( and perhaps driven by that) I feel push should be a top priority. Alarm manager should be used to fetch . Sorry I can't code so really can't contribute in any tangible way except just to put forth my opinion. Thanks

@gdt
Copy link

@gdt gdt commented Oct 24, 2015

Doze mode seems like a reasonable concept, except that it's trying to push people into GCM (which is non-Free and has privacy issues). I would suggest going down two paths simultaneously

  • Try to be a good doze client. With polling, just allow the polling interval to get longer as doze requires it. That might be as simple as setting an alarm for now+poll_interval on every wakeup and not worrying that it's longer than configured. This is not really so crazy, as the idea that one needs rapid polling when the device is really idle only seems to fit a use case where one leaves the phone on a table in a room and stays there for hours. I guess it will be interesting to see logs of real people and if their phones doze when they are not asleep or really away from them. Also with this, intercept some done_dozing signal and poll again if it's been longer than the configure interval. Perhaps also set some doze-mode poll interval, so the user can say poll every 10 minutes, but only every hour in doze mode, if that's what they want. Or maybe that's too complicated.
  • Make push still work. It seems we really don't understand enough yet, and whitelisting is the only way. Somewhat eparately from this, it would be nice if push could have much longer times in IDLE, to make k-9 use less power in doze mode.

What I don't see being all that useful or important (to me) is enabling polling in doze more often than every 15 minutes. I suppose that could be important to people who can't use push, need to get mail promptly, and leave their phone on a table while being near it. It would be interesting to hear from anybody who sees themselves in that situation.

For me, if my phone goes into doze because I haven't touched it and it hasn't moved in over an hour, having mail arrive up to 15 minutes late is ok. Unless I am mis-guessing how often doze will happen.

@sidamos
Copy link

@sidamos sidamos commented Oct 24, 2015

Periodic check:
You don't have to implement, what you are describing in your 1st paragraph, Android does that already.
If you use the old AlarmManager methods, they won't fire during Doze, unless the device enters the maintenance window (every hour or more).
If you use the new AlarmManager methods, alarms are being throttled to fire every 15 minutes at most.
My use case is this: During work, the phone lies on my table for longer periods of time. But I don't want to get mails only every hour or even later.
Also, Doze is supposed to end, when you move the device. But in my test, it only ends if I switch the display on. And I don't want to switch the display on every now and then only to enable K9 to look for new mails.
So, K9 should use the new AlarmManager method, if the user desires to get mails during Doze.

Push:
If we are lucky, push should work as normal if K9 is on the whitelist. So, nothing has to be implemented for this. Optionally, K9 could ask the user to be put on the whitelist. There is an intent for that.

@NadirHusain
Copy link

@NadirHusain NadirHusain commented Oct 24, 2015

As an aside I would like to mention that I only get real push if my " refresh idle connection" is set to 6 minutes or less. Any higher number and my messages are delayed. I understand my cellular operator is most likely responsible for this but was just wondering if this option would still be available to me in the doze mode ? I am sure there are others like me and k9 needs to be able to " force " this incase doze " kills " this option in default mode . Thanks

@cketti
Copy link
Member Author

@cketti cketti commented Nov 20, 2015

My current plan is:

  1. By default we embrace Doze. If Push is enabled we disable it while Doze is active. That will prevent the app from establishing a Push connection during the Doze maintenance windows.
  2. If K-9 Mail was whitelisted by the user we use the new API for our alarms. That should allow poll frequencies of up to every 15 minutes and Push should work, too. The Push refresh interval can't be smaller than 15 minutes though.
@cketti cketti removed the needs info label Nov 20, 2015
@NadirHusain
Copy link

@NadirHusain NadirHusain commented Nov 20, 2015

as of now K9 5.107 android 5.1.1 i need a minimum refresh time of 6 minutes to be able to get real time push notifications (tcp/cellular service provider issue). setting the minimum idle refresh time at 15 minutes would mean end of push mail for me (as i know it) unless there is a way to set idle refresh time at 6 minutes or less when doze mode is NOT active and could switch to 15 minutes when doze kicks in. i am sure there are many others like me who need real time push and have idle refresh set at very low intervals . request you to consider this use case issue . thanks

@cketti
Copy link
Member Author

@cketti cketti commented Nov 20, 2015

@NadirHusain: Doze is only available on Android 6+. If Doze is not active (or on previous Android versions) K-9 Mail will not impose any new restrictions.

@NadirHusain
Copy link

@NadirHusain NadirHusain commented Nov 20, 2015

@cketti clear now . thanks .

@sidamos
Copy link

@sidamos sidamos commented Nov 20, 2015

Good idea. This should have been done by Google, without the need to call new AlarmManager methods by the app.

@imsaguy
Copy link

@imsaguy imsaguy commented Jan 6, 2016

Any progress on this? If I have to look at K9 periodically just to get my notifications, I might as well set my sync to manual which defeats the whole purpose of push notifications. If you're looking at ways to bypass, do what apps like KeepPass do and set a transparent notification icon which keeps the app in the foreground like a music app and prevents doze.

@daradib
Copy link

@daradib daradib commented Jan 6, 2016

@imsaguy For now you can manually whitelist K9 under Settings > Battery > Battery Optimization (or similar). Mostly fixes push and #970. Doze still partially applies, since current AlarmManager will only fire during increasingly spaced apart maintenance windows, so I think the IMAP IDLE connection may eventually time out if there is no traffic and the device is stationary long enough. Also, to be specific, the foreground service prevents App Standby but not Doze.

@gadgetfan10
Copy link

@gadgetfan10 gadgetfan10 commented Jan 15, 2016

@cketti So under the "embrace Doze" plan, is the idea to stop push connections during Doze but - and this would seem to be the difference from the user end - re-enabling push connections when the device comes out of Doze?

Sort of separately, and maybe because I don't understand the specifics, why would this approach prevent the establishment of push connections during the Doze maintenance windows?

@cketti
Copy link
Member Author

@cketti cketti commented Jan 15, 2016

@gadgetfan10 Poor wording on my part.

Embrace Doze:

  • Close existing Push connections when entering Doze; don't attempt to create new ones while in Doze or maintenance window
  • During maintenance windows we trigger a simple mail synchronization
  • Return to normal operation (e.g. establish Push connections) when exiting Doze (device returns to normal operation; not just Doze maintenance window)
@gadgetfan10
Copy link

@gadgetfan10 gadgetfan10 commented Jan 15, 2016

Sounds like a solid plan. Thanks for the clarification.

Looking forward to seeing it implemented. If I had programming skills, I'd offer to help. Since I don't, I'll just say thanks for your efforts.

@cketti cketti closed this Feb 23, 2016
@cketti cketti reopened this Feb 23, 2016
@doodhout
Copy link

@doodhout doodhout commented Mar 10, 2016

@cketti As I understand how this will work: is there no way to enable doze and keep push connections alive during screen-off? Right now, I'm using a Sony smartphone that features similar functionality called 'Stamina Mode' and by whitelisting K9Mail I am ensured the K9Mail app works as if Stamina Mode hasn't been enabled - other apps that are not whitelisted will still be forced to go to sleep and ceize their activity.

Doze seems to be more refined than Stamina Mode, as it offers an API that allows apps certain activity but still well controlled and more battery-friendly than if the feature was disabled altogether; which isn't the case for Stamina Mode.

@bobpullen
Copy link

@bobpullen bobpullen commented Mar 22, 2016

Just a note to say that this issue is exacerbated on Android N where Doze 2.0 is initiated on screen off rather than a perceived period of inactivity.

@trostc
Copy link

@trostc trostc commented Jun 13, 2018

@ajweber
Copy link

@ajweber ajweber commented Jun 13, 2018

As I said, it doesn't work. In "Optimize Battery Usage", K9 is toggled off. It is also listed in "Unmonitored Apps".

I'm not sure why you don't believe me. I almost didn't mention it, because I didn't want to get into an "argument" about whether the settings truly work.

@trostc
Copy link

@trostc trostc commented Jun 13, 2018

@ajweber
Copy link

@ajweber ajweber commented Jun 13, 2018

Dovecot 2.0.9 IMAPS.

@ajweber
Copy link

@ajweber ajweber commented Jun 13, 2018

So here's where I'm at with this, in case anyone cares...

I have a test app that works "As Expected". I created a user in Firebase and created a tiny DB up there too. Firebase allows an authenticated user to receive GCM and read/write to the DB.

On app startup, I check if my user is correctly authenticated in Firebase (if not, I use an email and password - this can be changed and while it IS my target email, it is NOT my email server password). Once the Firebase user is valid, it pushes its latest device token to the mini DB. DB is KV: email_address and current device token/ID. There is a service running to monitor for token changes and for messages.

I have tested receiving messages with the app in foreground, background and forced the (virtual) device to Doze and it still receives the GCM messages...while that had some tricks to it, it is pretty-much "as advertised".

Currently messages are sent from a test harness: inotifywait script on my linux box that sees a change to the timestamp on the Maildir. For now, I hard-coded my device's token, but the idea is to do a simple lookup of the registered device by email address from Firebase DB. Use that unique ID to send the message via HTTPS. No storing of actual mail account password anywhere. (But that whole thing will get more robust.)

When a message is received, the idea is to launch a previously configured Intent. And the message receiver seems to run reliably, so the next step is to load K9 onto my test device and see if launching an intent on a separate app - that is still "dozing" - will allow the app to come out of doze too. I'm going to try with com.fsck.k9.intent.action.MAIL_SERVICE_RESET and see what happens. I honestly don't know if it's going to allow one app to force another out of doze.

If that doesn't work, I'm afraid the only option is to incorporate the enhancement directly into the app code...this, I'm very confident will work, but IDK if the amount effort is truly worth it. :)

Constructive comments welcome.

@jimimaseye
Copy link

@jimimaseye jimimaseye commented Jun 13, 2018

I dont understand why people are struggling with Samsung phones. I'm not calling them liars or anything negative, Im sure their frustrations are genuine. As stated above I dont have the problem with K9 running on my Samsung Oreo android v8.0 and recently on Nougat 7.0). I have a mailserver that uses push, and I have mail notifications arrive on my home screen as soon as they hit the inbox on the server.

So, to be sure I will repeat the steps of my setup here and give screen shots.

As stated above:

Go to Andriod Settings- Battery.
Click on the "Battery Usage box toward the top center of the screen.
Click the Menu (Three vertical dots in upper right).
Click on Optimize Battery Usage.
Select "All Apps" from the drop down arrow at the top of the screen.
Scroll to K-9 Mail and toggle it Off

Also

  1. K9Mail is entered as an entry in to the battery 'Unmonitored Apps' (thus stopping it from being put to or suggested for putting to sleep)
  2. 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
  3. 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

SCREENSHOTS

K9 allowed to use Background battery usage:

image


K9 set as non-optmised app:

image


K9 as unmonitored app:

image


K9 allowing background data usage:

image

Note: for those that have Data Saver enabled in their phone, it is important to ensure 'Data Saver' is turned off if you are using Data Saver but want the notifications to still work. Otherwise, as the finction says, it will stop K9 from doing background syncing whilst Data Saver is enabled.


and of course...

K9 set to allow Background Sync (within the App Global settings):

image

I hope this helps.

One more thing I would say is to check and ensure the mail server you are receiving emails from actually uses push. Servers such as YAHOO only allow their email client (Yahoo Mail) to receive push notifications even though you are still using IMAP - K9 will not receive them and relies on polling. This is a Yahoo thing and Im sure other providers may do similar. So, double check before concluding it is the fault of K9 (assuming you have all of the above settings).

EDIT:

Over time I have noticed that rare but occasionally (say once a week, if that) it stops bringing in the email and working as expected. But this isn't to do with the settings or setup and turns out to be K9 'getting in a twist'. Simply doing a FORCE STOP of the K9 app and restarting it up again (or alternatively a more long-winded reboot of your phone) sorts it out.

@McNobody
Copy link

@McNobody McNobody commented Jun 14, 2018

I'm really thinking same way as @jimimaseye . I've used K9 at Samsung S8 (now Oreo updated) over a year perfectly fine with IMAP IDLE. The mail server is Dovecot 2.2.22.
I have configured all those settings like @jimimaseye 's great guide. I have Data saver off globally. Extra settings in K9 I have Update IDLE connection = 24 minutes.

One thing come to my mind is the network connection. Mine is working perfectly with Elisa 4G in Finland and with my own wifi. The Elisa 4G gives IP address from the 100.64.0.0/10 private network. Still IMAP IDLE is working perfectly with Elisa's NAT.

I can see the mobile network could possibly broke the TCP connection before the IMAP IDLE timeout. This could be one reason why it's not working for someone. The other reason could be operator specific firmware. And of course if the email service provider really support IMAP IDLE.

@X-Ryl669
Copy link

@X-Ryl669 X-Ryl669 commented Jun 14, 2018

@jimimaseye and @McNobody, not everyone have Samsung phone, and their implementation is different on different vendor's OS. The problem I've on my phone is not reproducible on other's phone because each vendor implement doze the way they want and most of them think they are clever and also add additional "optimization" on it. On my phone, whatever the setting you set for "escaping" battery optimization (I don't have a data usage specific setting either), the OS will kill K9's process after a given time if it's not in focus, or if it needs memory or because the moon is shining.

In all cases, the only fact that each phone is different and that the dialogs to change the right setting is spread in numerous and hard to reach pages make the thing completely unusable for non-tech savvy users.
There is no other way than to bend to google's will here and use FCM/GCM since it's the only way that's guaranteed to work now and in the future.

@ajweber Can you give more details here ?
Once waken up, can you start K9 via an intent, and does it have network connection ?
If it's too hard to have K9 do what you want, maybe you can create another app that's downloading something upon responding to an intent and try with it first ?

@LosD
Copy link

@LosD LosD commented Jul 11, 2018

@jimimaseye How do you get into the real Android battery settings? On my S8+ (SM-G955F, FXXU2CRF7 software version, NEE region), Samsung seems to have ripped them out, only leaving their own... Which makes it impossible to disable optimization, only leaving it as unmonitored, which apparently doesn't help by itself.

@jimimaseye
Copy link

@jimimaseye jimimaseye commented Jul 11, 2018

@LosD

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

That's the method I am using.

@LosD
Copy link

@LosD LosD commented Jul 11, 2018

@jimimaseye

Holy hell, they hid that well. Not even a search found it, so I thought it was in the standard/old Android battery settings. Thanks!

@coogor
Copy link

@coogor coogor commented Aug 23, 2018

Just for the records - after my S7 Edge was updated to Android 8, everything works as expected again!

@jamiesonbecker
Copy link

@jamiesonbecker jamiesonbecker commented Aug 24, 2018

Under Android 9 ("Pie"), another place to check is: Battery Optimization under Settings (not under Apps), change the dropdown menu from "Optimized Apps" to "All Apps", and change K-9 Mail to "Don't Optimize". Don't forget to restart K-9 (or your entire phone) if it doesn't seem to take effect.

@Draghtnod
Copy link

@Draghtnod Draghtnod commented Sep 28, 2018

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

This is horrible! After updating to Android 8 my Push-Notifications didn't work anymore and i started searching. The server logs told me that my phone closed the connection after a few minutes of Screen-Off time. Turning off this optimization solved the issue.

You can't expect users to find such a well hidden menu. DAVdroid does turn off this feature for it's users when first starting up. Maybe K-9 should do the same?

@marmistrz
Copy link

@marmistrz marmistrz commented Sep 30, 2018

You can't expect users to find such a well hidden menu. DAVdroid does turn off this feature for it's users when first starting up. Maybe K-9 should do the same?

Conversations does the same.

Btw. one should really create a library to deep link into some vendor-specific app-killer settings (for confusion they're named battery savers). That's also what Conversations does on some phones

@Michele64
Copy link

@Michele64 Michele64 commented Oct 14, 2018

Turning off battery optimization does solve the issue for some hours, after that the settings revert to optimize. which is quite annoying as push doesn't work anymore

@bisherbas
Copy link

@bisherbas bisherbas commented Nov 20, 2018

Same as @Michele64. Turning off battery optimization on my stock OnePlus 5T does not solve the problem. K-9 shows "Syncing disabled" message just below where it says Accounts.

Background sync option in K-9 is set to Always.

@Newbie1980
Copy link

@Newbie1980 Newbie1980 commented Nov 21, 2018

After how long does this occur? What OS is it. Thanks

@jimimaseye
Copy link

@jimimaseye jimimaseye commented Nov 21, 2018

As i said above:

.
.
2, 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
3, 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

(Or whatever your phones equivalent is)

@lukefor
Copy link

@lukefor lukefor commented Mar 17, 2019

Can I suggest if/when this gets looked at, that the use case of no background syncing is supported, and that no workarounds or other background activity/listeners are used in this case, regardless if the phone is awake or asleep

@beerisgood
Copy link

@beerisgood beerisgood commented Aug 8, 2019

Any news? Problem still exist and nothing helps

@deutrino
Copy link

@deutrino deutrino commented Dec 4, 2019

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like *checks* nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?

Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.

Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

@trostc
Copy link

@trostc trostc commented Dec 4, 2019

@LosD
Copy link

@LosD LosD commented Dec 4, 2019

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like checks nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?

Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.

Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

@deutrino The functionality is restored by simply telling Android not to optimise it. Maybe better instructions for how to do that is needed, but putting a persistent notification for a problem that can be solved quite easily by the user seems overkill.

@utack
Copy link

@utack utack commented Dec 4, 2019

davx shows a popup that allows you to easily disable battery optimization for it on first start, maybe you could just copy the code?

@Newbie1980
Copy link

@Newbie1980 Newbie1980 commented Dec 4, 2019

Did I read correctly in the very first post that most of these sync problems could be solved by simply having an ongoing notification for a persistent process like checks nine other apps already do on my phone, including by somewhat respectable developers like Open Whisper Systems?
Syncing email is mission-critical for many, many people. If the broken functionality can be restored by means of an ongoing notification, please do it.
Context: I'm using Android 9, these ongoing notifications detract approximately 0.0000000001% from my enjoyment of my phone by being ugly, but they are now in 2019 a very standard way of dealing with Android's limitations.

@deutrino The functionality is restored by simply telling Android not to optimise it. Maybe better instructions for how to do that is needed, but putting a persistent notification for a problem that can be solved quite easily by the user seems overkill.

Dude have you even read the whole thread this does not fix it for everyone. This thread is years old at this stage and no fix. Its hit or miss from device to device and rom to rom.

@Michele64
Copy link

@Michele64 Michele64 commented Dec 4, 2019

On oneplus devices k9 gets killed and/or removed from battery optimization at random intervals. Eventually I gave up and switched to FairEmail.

@LosD
Copy link

@LosD LosD commented Dec 5, 2019

@Newbie1980 If you look, it was in it. And it may very well be that some phones (it seems only OnePlus is left) has some weird thing where it disables optimization. For those, it could be made it an option. But there is no need to make a terrible UI for everyone for that.

@tuxayo
Copy link

@tuxayo tuxayo commented Apr 3, 2020

In the hope it helps, here is a collaborative resource made by other app devs to try to list the issues with each manufacturer. And what devs can do to fix, hack, workaround them.

https://dontkillmyapp.com
https://dontkillmyapp.com/problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.