Figure out how to deal with Doze (Android 6+) #857

Open
cketti opened this Issue Oct 23, 2015 · 155 comments

Comments

Projects
None yet
@cketti
Member

cketti commented Oct 23, 2015

Doze reduces battery consumption by deferring background CPU and network activity for apps when the device is unused for long periods of time.

Learn more about it here: Optimizing for Doze and App Standby

As far as I can tell we have these options:

  1. Embrace Doze. When the device tells us to go to sleep we do.
  2. Fight Doze. Use the new API to schedule checks for new mail (can't be more frequent than every 15 minutes)
  3. Evade Doze. Foreground services (think music player) prevent the device from entering Doze. This requires displaying an ongoing notification.

As far as I can tell users can't disable Doze in settings. They have, however, the option to put apps on a whitelist.

An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.

So the interval between mail checks still has to be at least 15 minutes. But there is a chance that Push might work during Doze. Keeping a connection open doesn't require alarms.

My goal is to first collect all options we have and then decide on the best way forward. Please let me know if you can think of any other options and I will add them to this post. Also, if you have a special use-case regarding mail checks that you think should be supported by K-9 Mail let us know.

Update (2015-11-10): It looks like the only way for an app to prevent the device from entering Doze is to keep the screen on. So option 3 is not something that's feasible for us.

Update (2015-11-20): Using a foreground service is possible but tricky due to a bug in Android. See https://commonsware.com/blog/2015/11/18/another-doze-edge-case-foreground.html

@sidamos

This comment has been minimized.

Show comment
Hide comment
@sidamos

sidamos Oct 23, 2015

High priority GCM notifications do work normally during Doze (that does not help K9, of course), so Hangout messages do work. For me, mails are as important as Hangout messages. So, doing nothing about Doze (like Gmail does not fetch IMAP) is not on option for me.

I think, push should work if the user whitelists the app (this setting allegedly helps WhatsApp). If the user enables push, K9 could ask the user to put the app on the whitelist (there is an API for that). I have not tested that, because I do not use push.

For periodic mail checking, K9 should use the new AlarmManager methods. K9 should have an option to do that. This still allows the device entering Doze mode and saves battery. I would not call that "fighting Doze".

Having an ongoing notification is annoying IMHO and a mail app should not prevent the device from going into Doze at all.

sidamos commented Oct 23, 2015

High priority GCM notifications do work normally during Doze (that does not help K9, of course), so Hangout messages do work. For me, mails are as important as Hangout messages. So, doing nothing about Doze (like Gmail does not fetch IMAP) is not on option for me.

I think, push should work if the user whitelists the app (this setting allegedly helps WhatsApp). If the user enables push, K9 could ask the user to put the app on the whitelist (there is an API for that). I have not tested that, because I do not use push.

For periodic mail checking, K9 should use the new AlarmManager methods. K9 should have an option to do that. This still allows the device entering Doze mode and saves battery. I would not call that "fighting Doze".

Having an ongoing notification is annoying IMHO and a mail app should not prevent the device from going into Doze at all.

@ssieb

This comment has been minimized.

Show comment
Hide comment
@ssieb

ssieb Oct 24, 2015

If whitelisting the app is required to make push and/or poll to work, then I'm fine with that. As mentioned above, there should be some notification to the user that it's necessary to do so.

ssieb commented Oct 24, 2015

If whitelisting the app is required to make push and/or poll to work, then I'm fine with that. As mentioned above, there should be some notification to the user that it's necessary to do so.

@NadirHusain

This comment has been minimized.

Show comment
Hide comment
@NadirHusain

NadirHusain Oct 24, 2015

Does anyone know for sure how much " inactive " time needs to elapse before doze mode kicks in ?

Does anyone know for sure how much " inactive " time needs to elapse before doze mode kicks in ?

@cketti

This comment has been minimized.

Show comment
Hide comment
@cketti

cketti Oct 24, 2015

Member

@NadirHusain: As far as I can tell OEMs are free to tweak the different timeout values. See DeviceIdleController.
From what I've read current Nexus devices enter Doze after 1 hour without significant motion.

Member

cketti commented Oct 24, 2015

@NadirHusain: As far as I can tell OEMs are free to tweak the different timeout values. See DeviceIdleController.
From what I've read current Nexus devices enter Doze after 1 hour without significant motion.

@NadirHusain

This comment has been minimized.

Show comment
Hide comment
@NadirHusain

NadirHusain Oct 24, 2015

I agree with @sidamos . Doing nothing is not an option. Since I am a push user ( and perhaps driven by that) I feel push should be a top priority. Alarm manager should be used to fetch . Sorry I can't code so really can't contribute in any tangible way except just to put forth my opinion. Thanks

I agree with @sidamos . Doing nothing is not an option. Since I am a push user ( and perhaps driven by that) I feel push should be a top priority. Alarm manager should be used to fetch . Sorry I can't code so really can't contribute in any tangible way except just to put forth my opinion. Thanks

@gdt

This comment has been minimized.

Show comment
Hide comment
@gdt

gdt Oct 24, 2015

Doze mode seems like a reasonable concept, except that it's trying to push people into GCM (which is non-Free and has privacy issues). I would suggest going down two paths simultaneously

  • Try to be a good doze client. With polling, just allow the polling interval to get longer as doze requires it. That might be as simple as setting an alarm for now+poll_interval on every wakeup and not worrying that it's longer than configured. This is not really so crazy, as the idea that one needs rapid polling when the device is really idle only seems to fit a use case where one leaves the phone on a table in a room and stays there for hours. I guess it will be interesting to see logs of real people and if their phones doze when they are not asleep or really away from them. Also with this, intercept some done_dozing signal and poll again if it's been longer than the configure interval. Perhaps also set some doze-mode poll interval, so the user can say poll every 10 minutes, but only every hour in doze mode, if that's what they want. Or maybe that's too complicated.
  • Make push still work. It seems we really don't understand enough yet, and whitelisting is the only way. Somewhat eparately from this, it would be nice if push could have much longer times in IDLE, to make k-9 use less power in doze mode.

What I don't see being all that useful or important (to me) is enabling polling in doze more often than every 15 minutes. I suppose that could be important to people who can't use push, need to get mail promptly, and leave their phone on a table while being near it. It would be interesting to hear from anybody who sees themselves in that situation.

For me, if my phone goes into doze because I haven't touched it and it hasn't moved in over an hour, having mail arrive up to 15 minutes late is ok. Unless I am mis-guessing how often doze will happen.

gdt commented Oct 24, 2015

Doze mode seems like a reasonable concept, except that it's trying to push people into GCM (which is non-Free and has privacy issues). I would suggest going down two paths simultaneously

  • Try to be a good doze client. With polling, just allow the polling interval to get longer as doze requires it. That might be as simple as setting an alarm for now+poll_interval on every wakeup and not worrying that it's longer than configured. This is not really so crazy, as the idea that one needs rapid polling when the device is really idle only seems to fit a use case where one leaves the phone on a table in a room and stays there for hours. I guess it will be interesting to see logs of real people and if their phones doze when they are not asleep or really away from them. Also with this, intercept some done_dozing signal and poll again if it's been longer than the configure interval. Perhaps also set some doze-mode poll interval, so the user can say poll every 10 minutes, but only every hour in doze mode, if that's what they want. Or maybe that's too complicated.
  • Make push still work. It seems we really don't understand enough yet, and whitelisting is the only way. Somewhat eparately from this, it would be nice if push could have much longer times in IDLE, to make k-9 use less power in doze mode.

What I don't see being all that useful or important (to me) is enabling polling in doze more often than every 15 minutes. I suppose that could be important to people who can't use push, need to get mail promptly, and leave their phone on a table while being near it. It would be interesting to hear from anybody who sees themselves in that situation.

For me, if my phone goes into doze because I haven't touched it and it hasn't moved in over an hour, having mail arrive up to 15 minutes late is ok. Unless I am mis-guessing how often doze will happen.

@sidamos

This comment has been minimized.

Show comment
Hide comment
@sidamos

sidamos Oct 24, 2015

Periodic check:
You don't have to implement, what you are describing in your 1st paragraph, Android does that already.
If you use the old AlarmManager methods, they won't fire during Doze, unless the device enters the maintenance window (every hour or more).
If you use the new AlarmManager methods, alarms are being throttled to fire every 15 minutes at most.
My use case is this: During work, the phone lies on my table for longer periods of time. But I don't want to get mails only every hour or even later.
Also, Doze is supposed to end, when you move the device. But in my test, it only ends if I switch the display on. And I don't want to switch the display on every now and then only to enable K9 to look for new mails.
So, K9 should use the new AlarmManager method, if the user desires to get mails during Doze.

Push:
If we are lucky, push should work as normal if K9 is on the whitelist. So, nothing has to be implemented for this. Optionally, K9 could ask the user to be put on the whitelist. There is an intent for that.

sidamos commented Oct 24, 2015

Periodic check:
You don't have to implement, what you are describing in your 1st paragraph, Android does that already.
If you use the old AlarmManager methods, they won't fire during Doze, unless the device enters the maintenance window (every hour or more).
If you use the new AlarmManager methods, alarms are being throttled to fire every 15 minutes at most.
My use case is this: During work, the phone lies on my table for longer periods of time. But I don't want to get mails only every hour or even later.
Also, Doze is supposed to end, when you move the device. But in my test, it only ends if I switch the display on. And I don't want to switch the display on every now and then only to enable K9 to look for new mails.
So, K9 should use the new AlarmManager method, if the user desires to get mails during Doze.

Push:
If we are lucky, push should work as normal if K9 is on the whitelist. So, nothing has to be implemented for this. Optionally, K9 could ask the user to be put on the whitelist. There is an intent for that.

@NadirHusain

This comment has been minimized.

Show comment
Hide comment
@NadirHusain

NadirHusain Oct 24, 2015

As an aside I would like to mention that I only get real push if my " refresh idle connection" is set to 6 minutes or less. Any higher number and my messages are delayed. I understand my cellular operator is most likely responsible for this but was just wondering if this option would still be available to me in the doze mode ? I am sure there are others like me and k9 needs to be able to " force " this incase doze " kills " this option in default mode . Thanks

As an aside I would like to mention that I only get real push if my " refresh idle connection" is set to 6 minutes or less. Any higher number and my messages are delayed. I understand my cellular operator is most likely responsible for this but was just wondering if this option would still be available to me in the doze mode ? I am sure there are others like me and k9 needs to be able to " force " this incase doze " kills " this option in default mode . Thanks

@cketti

This comment has been minimized.

Show comment
Hide comment
@cketti

cketti Nov 20, 2015

Member

My current plan is:

  1. By default we embrace Doze. If Push is enabled we disable it while Doze is active. That will prevent the app from establishing a Push connection during the Doze maintenance windows.
  2. If K-9 Mail was whitelisted by the user we use the new API for our alarms. That should allow poll frequencies of up to every 15 minutes and Push should work, too. The Push refresh interval can't be smaller than 15 minutes though.
Member

cketti commented Nov 20, 2015

My current plan is:

  1. By default we embrace Doze. If Push is enabled we disable it while Doze is active. That will prevent the app from establishing a Push connection during the Doze maintenance windows.
  2. If K-9 Mail was whitelisted by the user we use the new API for our alarms. That should allow poll frequencies of up to every 15 minutes and Push should work, too. The Push refresh interval can't be smaller than 15 minutes though.

@cketti cketti removed the needs info label Nov 20, 2015

@NadirHusain

This comment has been minimized.

Show comment
Hide comment
@NadirHusain

NadirHusain Nov 20, 2015

as of now K9 5.107 android 5.1.1 i need a minimum refresh time of 6 minutes to be able to get real time push notifications (tcp/cellular service provider issue). setting the minimum idle refresh time at 15 minutes would mean end of push mail for me (as i know it) unless there is a way to set idle refresh time at 6 minutes or less when doze mode is NOT active and could switch to 15 minutes when doze kicks in. i am sure there are many others like me who need real time push and have idle refresh set at very low intervals . request you to consider this use case issue . thanks

as of now K9 5.107 android 5.1.1 i need a minimum refresh time of 6 minutes to be able to get real time push notifications (tcp/cellular service provider issue). setting the minimum idle refresh time at 15 minutes would mean end of push mail for me (as i know it) unless there is a way to set idle refresh time at 6 minutes or less when doze mode is NOT active and could switch to 15 minutes when doze kicks in. i am sure there are many others like me who need real time push and have idle refresh set at very low intervals . request you to consider this use case issue . thanks

@cketti

This comment has been minimized.

Show comment
Hide comment
@cketti

cketti Nov 20, 2015

Member

@NadirHusain: Doze is only available on Android 6+. If Doze is not active (or on previous Android versions) K-9 Mail will not impose any new restrictions.

Member

cketti commented Nov 20, 2015

@NadirHusain: Doze is only available on Android 6+. If Doze is not active (or on previous Android versions) K-9 Mail will not impose any new restrictions.

@NadirHusain

This comment has been minimized.

Show comment
Hide comment
@NadirHusain

NadirHusain Nov 20, 2015

@cketti clear now . thanks .

@cketti clear now . thanks .

@sidamos

This comment has been minimized.

Show comment
Hide comment
@sidamos

sidamos Nov 20, 2015

Good idea. This should have been done by Google, without the need to call new AlarmManager methods by the app.

sidamos commented Nov 20, 2015

Good idea. This should have been done by Google, without the need to call new AlarmManager methods by the app.

@imsaguy

This comment has been minimized.

Show comment
Hide comment
@imsaguy

imsaguy Jan 6, 2016

Any progress on this? If I have to look at K9 periodically just to get my notifications, I might as well set my sync to manual which defeats the whole purpose of push notifications. If you're looking at ways to bypass, do what apps like KeepPass do and set a transparent notification icon which keeps the app in the foreground like a music app and prevents doze.

imsaguy commented Jan 6, 2016

Any progress on this? If I have to look at K9 periodically just to get my notifications, I might as well set my sync to manual which defeats the whole purpose of push notifications. If you're looking at ways to bypass, do what apps like KeepPass do and set a transparent notification icon which keeps the app in the foreground like a music app and prevents doze.

@daradib

This comment has been minimized.

Show comment
Hide comment
@daradib

daradib Jan 6, 2016

@imsaguy For now you can manually whitelist K9 under Settings > Battery > Battery Optimization (or similar). Mostly fixes push and #970. Doze still partially applies, since current AlarmManager will only fire during increasingly spaced apart maintenance windows, so I think the IMAP IDLE connection may eventually time out if there is no traffic and the device is stationary long enough. Also, to be specific, the foreground service prevents App Standby but not Doze.

daradib commented Jan 6, 2016

@imsaguy For now you can manually whitelist K9 under Settings > Battery > Battery Optimization (or similar). Mostly fixes push and #970. Doze still partially applies, since current AlarmManager will only fire during increasingly spaced apart maintenance windows, so I think the IMAP IDLE connection may eventually time out if there is no traffic and the device is stationary long enough. Also, to be specific, the foreground service prevents App Standby but not Doze.

@gadgetfan10

This comment has been minimized.

Show comment
Hide comment
@gadgetfan10

gadgetfan10 Jan 15, 2016

@cketti So under the "embrace Doze" plan, is the idea to stop push connections during Doze but - and this would seem to be the difference from the user end - re-enabling push connections when the device comes out of Doze?

Sort of separately, and maybe because I don't understand the specifics, why would this approach prevent the establishment of push connections during the Doze maintenance windows?

@cketti So under the "embrace Doze" plan, is the idea to stop push connections during Doze but - and this would seem to be the difference from the user end - re-enabling push connections when the device comes out of Doze?

Sort of separately, and maybe because I don't understand the specifics, why would this approach prevent the establishment of push connections during the Doze maintenance windows?

@cketti

This comment has been minimized.

Show comment
Hide comment
@cketti

cketti Jan 15, 2016

Member

@gadgetfan10 Poor wording on my part.

Embrace Doze:

  • Close existing Push connections when entering Doze; don't attempt to create new ones while in Doze or maintenance window
  • During maintenance windows we trigger a simple mail synchronization
  • Return to normal operation (e.g. establish Push connections) when exiting Doze (device returns to normal operation; not just Doze maintenance window)
Member

cketti commented Jan 15, 2016

@gadgetfan10 Poor wording on my part.

Embrace Doze:

  • Close existing Push connections when entering Doze; don't attempt to create new ones while in Doze or maintenance window
  • During maintenance windows we trigger a simple mail synchronization
  • Return to normal operation (e.g. establish Push connections) when exiting Doze (device returns to normal operation; not just Doze maintenance window)
@gadgetfan10

This comment has been minimized.

Show comment
Hide comment
@gadgetfan10

gadgetfan10 Jan 15, 2016

Sounds like a solid plan. Thanks for the clarification.

Looking forward to seeing it implemented. If I had programming skills, I'd offer to help. Since I don't, I'll just say thanks for your efforts.

Sounds like a solid plan. Thanks for the clarification.

Looking forward to seeing it implemented. If I had programming skills, I'd offer to help. Since I don't, I'll just say thanks for your efforts.

@cketti cketti closed this Feb 23, 2016

@cketti cketti reopened this Feb 23, 2016

@doodhout

This comment has been minimized.

Show comment
Hide comment
@doodhout

doodhout Mar 10, 2016

@cketti As I understand how this will work: is there no way to enable doze and keep push connections alive during screen-off? Right now, I'm using a Sony smartphone that features similar functionality called 'Stamina Mode' and by whitelisting K9Mail I am ensured the K9Mail app works as if Stamina Mode hasn't been enabled - other apps that are not whitelisted will still be forced to go to sleep and ceize their activity.

Doze seems to be more refined than Stamina Mode, as it offers an API that allows apps certain activity but still well controlled and more battery-friendly than if the feature was disabled altogether; which isn't the case for Stamina Mode.

@cketti As I understand how this will work: is there no way to enable doze and keep push connections alive during screen-off? Right now, I'm using a Sony smartphone that features similar functionality called 'Stamina Mode' and by whitelisting K9Mail I am ensured the K9Mail app works as if Stamina Mode hasn't been enabled - other apps that are not whitelisted will still be forced to go to sleep and ceize their activity.

Doze seems to be more refined than Stamina Mode, as it offers an API that allows apps certain activity but still well controlled and more battery-friendly than if the feature was disabled altogether; which isn't the case for Stamina Mode.

@bobpullen

This comment has been minimized.

Show comment
Hide comment
@bobpullen

bobpullen Mar 22, 2016

Just a note to say that this issue is exacerbated on Android N where Doze 2.0 is initiated on screen off rather than a perceived period of inactivity.

Just a note to say that this issue is exacerbated on Android N where Doze 2.0 is initiated on screen off rather than a perceived period of inactivity.

@chrcoluk

This comment has been minimized.

Show comment
Hide comment
@chrcoluk

chrcoluk Jun 29, 2016

is there a proper workaround to this?

I can confirm poll checks are not working, I just whitelisted k9 in the power saving options, and am waiting to now see if the poll works.

is there a proper workaround to this?

I can confirm poll checks are not working, I just whitelisted k9 in the power saving options, and am waiting to now see if the poll works.

@doodhout

This comment has been minimized.

Show comment
Hide comment
@doodhout

doodhout Jun 29, 2016

@chrcoluk Even when whitelisted, the poll checks will only occur in (but not outside of) the windows of activity that Doze permits, and the time between those windows will increase continuously thus delaying poll checks by minutes or hours!

@chrcoluk Even when whitelisted, the poll checks will only occur in (but not outside of) the windows of activity that Doze permits, and the time between those windows will increase continuously thus delaying poll checks by minutes or hours!

@chrcoluk

This comment has been minimized.

Show comment
Hide comment
@chrcoluk

chrcoluk Jun 30, 2016

yeah so this is now broken for the forseeable future then?

I thought I managed to disable doze on the entire phone by turning of the battery savings feature, but seems it is still in use. :(

The phone is samsung touchwiz 6.0.1.

yeah so this is now broken for the forseeable future then?

I thought I managed to disable doze on the entire phone by turning of the battery savings feature, but seems it is still in use. :(

The phone is samsung touchwiz 6.0.1.

@chrcoluk

This comment has been minimized.

Show comment
Hide comment
@chrcoluk

chrcoluk Jun 30, 2016

According to greenify dev if you add GCM priority option to the app and the user enables it, then the app can bypass doze mode.

According to greenify dev if you add GCM priority option to the app and the user enables it, then the app can bypass doze mode.

@cdmackay

This comment has been minimized.

Show comment
Hide comment
@cdmackay

cdmackay Sep 30, 2017

@rhari991

This comment has been minimized.

Show comment
Hide comment
@rhari991

rhari991 Sep 30, 2017

Contributor

@segdan It's related to how the push service works internally.

Whenever an error is encountered in the push service (eg trying to connect when no internet is available), a counter is incremented. If this counter goes above 10, then push (ie sync) is automatically disabled in order to preserve battery life.

This is what doze does - it prevents k-9 from being able to access the internet and hence triggers enough errors to disable the push service, so you see "sync disabled" when you unlock your phone after it has spent a long amount of time in Doze.

When a network change is detected (like reconnecting to WiFi), the above mentioned counter is reset and the push service is restarted.

The latest version of k-9 mail solves this issue by resetting the poll service when coming out of doze.

Contributor

rhari991 commented Sep 30, 2017

@segdan It's related to how the push service works internally.

Whenever an error is encountered in the push service (eg trying to connect when no internet is available), a counter is incremented. If this counter goes above 10, then push (ie sync) is automatically disabled in order to preserve battery life.

This is what doze does - it prevents k-9 from being able to access the internet and hence triggers enough errors to disable the push service, so you see "sync disabled" when you unlock your phone after it has spent a long amount of time in Doze.

When a network change is detected (like reconnecting to WiFi), the above mentioned counter is reset and the push service is restarted.

The latest version of k-9 mail solves this issue by resetting the poll service when coming out of doze.

@segdan

This comment has been minimized.

Show comment
Hide comment
@segdan

segdan Sep 30, 2017

@rhari991 I see. Now everything is clear, thank you.

segdan commented Sep 30, 2017

@rhari991 I see. Now everything is clear, thank you.

@2mkb1

This comment has been minimized.

Show comment
Hide comment
@2mkb1

2mkb1 Oct 2, 2017

I had instant success with this fix.
The new update did not help at all. (However there was an update for Accuweather that finally fixed the update issue there, whoopee!)
I went to the k-9 app on google play and there is a recent review there with these instructions and this finally has fixed the issue with K-9 mail not syncing:

  1. Go to Andriod Settings- Battery.
  2. Click on the "Battery Usage box toward the top center of the screen.
  3. Click the Menu (Three vertical dots in upper right).
  4. Click on Optimize Battery Usage.
  5. Select "All Apps" from the drop down arrow at the top of the screen.
  6. Scroll to K-9 Mail and toggle it Off

2mkb1 commented Oct 2, 2017

I had instant success with this fix.
The new update did not help at all. (However there was an update for Accuweather that finally fixed the update issue there, whoopee!)
I went to the k-9 app on google play and there is a recent review there with these instructions and this finally has fixed the issue with K-9 mail not syncing:

  1. Go to Andriod Settings- Battery.
  2. Click on the "Battery Usage box toward the top center of the screen.
  3. Click the Menu (Three vertical dots in upper right).
  4. Click on Optimize Battery Usage.
  5. Select "All Apps" from the drop down arrow at the top of the screen.
  6. Scroll to K-9 Mail and toggle it Off
@jimimaseye

This comment has been minimized.

Show comment
Hide comment
@jimimaseye

jimimaseye Oct 16, 2017

Go to Andriod Settings- Battery.
Click on the "Battery Usage box toward the top center of the screen.
Click the Menu (Three vertical dots in upper right).
Click on Optimize Battery Usage.
Select "All Apps" from the drop down arrow at the top of the screen.
Scroll to K-9 Mail and toggle it Off

This has worked for me. I had the 'syncing disabled' problem (and was just as annoyed and disappointed as everyone else).

I made the change as quoted above whilst also ensuring that K9Mail has the following settings too:

1, K9Mail is entered as an entry in to the battery 'Unmonitored Apps' (thus stopping it from being put to or suggested for putting to sleep)
2, 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
3, 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

With all of the above I think I have effectively allowed K9mail full reign of non-sleeping/unrestricted processor and unhindered background data and battery thus keeping it fully awake and able to operate. and so far (3 days later) it seems to have worked.

Go to Andriod Settings- Battery.
Click on the "Battery Usage box toward the top center of the screen.
Click the Menu (Three vertical dots in upper right).
Click on Optimize Battery Usage.
Select "All Apps" from the drop down arrow at the top of the screen.
Scroll to K-9 Mail and toggle it Off

This has worked for me. I had the 'syncing disabled' problem (and was just as annoyed and disappointed as everyone else).

I made the change as quoted above whilst also ensuring that K9Mail has the following settings too:

1, K9Mail is entered as an entry in to the battery 'Unmonitored Apps' (thus stopping it from being put to or suggested for putting to sleep)
2, 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
3, 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

With all of the above I think I have effectively allowed K9mail full reign of non-sleeping/unrestricted processor and unhindered background data and battery thus keeping it fully awake and able to operate. and so far (3 days later) it seems to have worked.

@Frykky

This comment has been minimized.

Show comment
Hide comment
@Frykky

Frykky Nov 29, 2017

I have a p8 lite 2017 and the problem is still here.

I tried all settings of battery savings but after some time k-9 stops fetching email.
I have several accounts, some gmail, some imap and some pop3.

All these have same problem :(

Frykky commented Nov 29, 2017

I have a p8 lite 2017 and the problem is still here.

I tried all settings of battery savings but after some time k-9 stops fetching email.
I have several accounts, some gmail, some imap and some pop3.

All these have same problem :(

@XCompactUser

This comment has been minimized.

Show comment
Hide comment
@XCompactUser

XCompactUser Dec 1, 2017

I'm using K9 quite a time now and was happy compared to other mail app I tried. So many thanks to all developers!

Unfortunately I'm not able to get push of K9 on my Sony Xperia X compact running (Android 7.1.1)
In total there are 4 devices working w/ K9
Sony Xperia Tipo (Android 4.0.4, K9 5.001)
-> no push issue at all
Samsung Galaxy S4 (Android 6.x, K9 5.001) & Samsung Galaxy Tab A 2016 (Andriod 7.0, K9 5.001)
-> push not working in standby (or doze) after K9 install, but if it was active lately
=> but w/ disabling standby for K9 and allowing background data it finally works as expected :)
Sony Xperia X Compact (Android 7.1.1, K9 5.208, 5.303)
-> push not working at all if phone is in standby (push works when phone was active lately), even with all tips and tricks from this thread (incl. disabling standby, allowing background data, push only / push and timed mail fetch, different push refresh and fetch intervalls ...) nothing worked

Mail accounts are IMAP from web.de & gmx.de.

So even w/ latest version 5.303 problems are still there.

@cketti: Should I give 5.301 also a try (as mentioned in #2841)? Or even 5.001?

Thanks.

XCompactUser commented Dec 1, 2017

I'm using K9 quite a time now and was happy compared to other mail app I tried. So many thanks to all developers!

Unfortunately I'm not able to get push of K9 on my Sony Xperia X compact running (Android 7.1.1)
In total there are 4 devices working w/ K9
Sony Xperia Tipo (Android 4.0.4, K9 5.001)
-> no push issue at all
Samsung Galaxy S4 (Android 6.x, K9 5.001) & Samsung Galaxy Tab A 2016 (Andriod 7.0, K9 5.001)
-> push not working in standby (or doze) after K9 install, but if it was active lately
=> but w/ disabling standby for K9 and allowing background data it finally works as expected :)
Sony Xperia X Compact (Android 7.1.1, K9 5.208, 5.303)
-> push not working at all if phone is in standby (push works when phone was active lately), even with all tips and tricks from this thread (incl. disabling standby, allowing background data, push only / push and timed mail fetch, different push refresh and fetch intervalls ...) nothing worked

Mail accounts are IMAP from web.de & gmx.de.

So even w/ latest version 5.303 problems are still there.

@cketti: Should I give 5.301 also a try (as mentioned in #2841)? Or even 5.001?

Thanks.

@X-Ryl669

This comment has been minimized.

Show comment
Hide comment
@X-Ryl669

X-Ryl669 Dec 22, 2017

Another option that would avoid all the trouble seen above is to do like Conversations is doing that is, embrace GCM when it's available but don't send any data through it.
This has multiple advantages:

  1. No need to make an exception from K9-mail in numerous and obscure system's dialog
  2. Have the best energy saving possible (your device is no more polling or waiting on an idle push connection)
  3. It's future proof

Basically, the idea is to have a server, I'm calling "proxy", that's requesting for changes on your imap accounts and when any change is visible, sends an empty message to google's firebase servers with high priority. This, in turns, wakes up your device from a specific intent where it'd connect to the imap servers to check for changes. Google does not know what changed since it's an empty message (it only knows that something changed).

The "proxy" server could be run by any entity, but it would be best if it was open source and easily installable on your NAS (if you are paranoïd like me). This proxy server does not necessarily need your account passwords (only if PUSH is not supported by the IMAP server) and can work as a TCP proxy. The idea is:

  1. K9 connect to the "proxy" server giving it the destination (IMAP) server's URL on some port.
  2. The "proxy" starts connecting to the IMAP server proxying each packet untouched to K9.
  3. Since most IMAP server use SSL/TLS, it means proxyfing encrypted packets, so it should be safe.
  4. K9 does not use this connection to fetch mail, but only to set up IDLE mode of the server
  5. K9 disconnect to the "proxy" server, yet the "proxy" server's connection is maintained to the IMAP's server
  6. Any event on the IMAP / "proxy" socket is likely due to a change in the IMAP's account state, and the "proxy" server then trigger the GCM/FCM trick (it sends an empty FCM notification message to your device that wakes it up from dozing)
  7. Your device is awaken, it connects to the IMAP server to fetch new mails (polling mode in that case)

This scheme however might have some issues:

  1. If the IMAP server does not support IDLE mode, then the "proxy" must know your credentials to poll it. This is mitigated by either auditing and trusting the source code of the server or running your own server instance.
  2. If the IDLE interval is too short, the connection will be dropped, thus creating an event on the socket. The "proxy" will trigger a notification, then K9 will have to resume the IDLE connection (using the same system). This means that K9 will be running at least with the interval of the IDLE refresh time. It might or might not be an issue. Again, it's possible that it's easier to know your credentials and maintain the IDLE connection by itself, but that's possible anyway.

Honestly, there is a reason why almost all Email app on the AppStore are now using server's side email storage, Google is killing the email application with its new takeover of the Doze feature.

X-Ryl669 commented Dec 22, 2017

Another option that would avoid all the trouble seen above is to do like Conversations is doing that is, embrace GCM when it's available but don't send any data through it.
This has multiple advantages:

  1. No need to make an exception from K9-mail in numerous and obscure system's dialog
  2. Have the best energy saving possible (your device is no more polling or waiting on an idle push connection)
  3. It's future proof

Basically, the idea is to have a server, I'm calling "proxy", that's requesting for changes on your imap accounts and when any change is visible, sends an empty message to google's firebase servers with high priority. This, in turns, wakes up your device from a specific intent where it'd connect to the imap servers to check for changes. Google does not know what changed since it's an empty message (it only knows that something changed).

The "proxy" server could be run by any entity, but it would be best if it was open source and easily installable on your NAS (if you are paranoïd like me). This proxy server does not necessarily need your account passwords (only if PUSH is not supported by the IMAP server) and can work as a TCP proxy. The idea is:

  1. K9 connect to the "proxy" server giving it the destination (IMAP) server's URL on some port.
  2. The "proxy" starts connecting to the IMAP server proxying each packet untouched to K9.
  3. Since most IMAP server use SSL/TLS, it means proxyfing encrypted packets, so it should be safe.
  4. K9 does not use this connection to fetch mail, but only to set up IDLE mode of the server
  5. K9 disconnect to the "proxy" server, yet the "proxy" server's connection is maintained to the IMAP's server
  6. Any event on the IMAP / "proxy" socket is likely due to a change in the IMAP's account state, and the "proxy" server then trigger the GCM/FCM trick (it sends an empty FCM notification message to your device that wakes it up from dozing)
  7. Your device is awaken, it connects to the IMAP server to fetch new mails (polling mode in that case)

This scheme however might have some issues:

  1. If the IMAP server does not support IDLE mode, then the "proxy" must know your credentials to poll it. This is mitigated by either auditing and trusting the source code of the server or running your own server instance.
  2. If the IDLE interval is too short, the connection will be dropped, thus creating an event on the socket. The "proxy" will trigger a notification, then K9 will have to resume the IDLE connection (using the same system). This means that K9 will be running at least with the interval of the IDLE refresh time. It might or might not be an issue. Again, it's possible that it's easier to know your credentials and maintain the IDLE connection by itself, but that's possible anyway.

Honestly, there is a reason why almost all Email app on the AppStore are now using server's side email storage, Google is killing the email application with its new takeover of the Doze feature.

@marmistrz

This comment has been minimized.

Show comment
Hide comment
@marmistrz

marmistrz Jan 6, 2018

It may be wise to suggest the user whitelist the app from doze, just like Conversations does.

It may be wise to suggest the user whitelist the app from doze, just like Conversations does.

@chrcoluk

This comment has been minimized.

Show comment
Hide comment
@chrcoluk

chrcoluk Jan 31, 2018

I been fine for several months having overriden doze settings with "doze settings editor" but I noticed k9 misbehaving the past month or so, and then I discovered android seems to be forcefully resetting doze settings to defaults, so k9 is now broken again on my phone.

I been fine for several months having overriden doze settings with "doze settings editor" but I noticed k9 misbehaving the past month or so, and then I discovered android seems to be forcefully resetting doze settings to defaults, so k9 is now broken again on my phone.

@marmistrz

This comment has been minimized.

Show comment
Hide comment
@marmistrz

marmistrz Feb 4, 2018

With current vendor-specific app-killing systems Doze seems not enough, cf. siacs/Conversations#2775

With current vendor-specific app-killing systems Doze seems not enough, cf. siacs/Conversations#2775

@philipwhiuk

This comment has been minimized.

Show comment
Hide comment
@philipwhiuk

philipwhiuk Feb 4, 2018

Member

@marmistrz

If devices want to randomly kill apps with no documentation on how apps are chosen there's little we can do about it. Doze is not this.

At most we will just be able to warn people when we get reports of it happening.

Member

philipwhiuk commented Feb 4, 2018

@marmistrz

If devices want to randomly kill apps with no documentation on how apps are chosen there's little we can do about it. Doze is not this.

At most we will just be able to warn people when we get reports of it happening.

@maxbraketorque

This comment has been minimized.

Show comment
Hide comment
@maxbraketorque

maxbraketorque Mar 27, 2018

I am using K-9 as an IMAP email client. For some reason, I didn't have this issue with Android 6, but I have it with Android 7 (and its also a new phone). I tried altering various settings in K-9 (Poll when connecting for push, shortening IDLE refresh time, deleting the entire account on K-9 and starting over) and on the phone (moving K-9 to the unmonitored app whitelist) to try to alleviate it, but nothing is helping with obtaining consistent timely notification of new emails.

I am now exploring other email clients starting with BlueMail to see how they fare. Interestingly, BlueMail seems to be finding new email as soon as it hits my email server, so those developers seemed to have figured something out to allow IMAP push to work effectively under Android 7 with their email client. I have not done anything special with their account configuration options, and I have not put BlueMail in the unmonitored app list. I know that doesn't obviously help you folks, but it at least shows that it is possible to have timely discovery of new email with push IMAP under Android 7.

I am using K-9 as an IMAP email client. For some reason, I didn't have this issue with Android 6, but I have it with Android 7 (and its also a new phone). I tried altering various settings in K-9 (Poll when connecting for push, shortening IDLE refresh time, deleting the entire account on K-9 and starting over) and on the phone (moving K-9 to the unmonitored app whitelist) to try to alleviate it, but nothing is helping with obtaining consistent timely notification of new emails.

I am now exploring other email clients starting with BlueMail to see how they fare. Interestingly, BlueMail seems to be finding new email as soon as it hits my email server, so those developers seemed to have figured something out to allow IMAP push to work effectively under Android 7 with their email client. I have not done anything special with their account configuration options, and I have not put BlueMail in the unmonitored app list. I know that doesn't obviously help you folks, but it at least shows that it is possible to have timely discovery of new email with push IMAP under Android 7.

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Mar 27, 2018

trostc commented Mar 27, 2018

@X-Ryl669

This comment has been minimized.

Show comment
Hide comment
@X-Ryl669

X-Ryl669 Mar 28, 2018

Guys, this is quite a long thread already. I'll try to sum this up, to avoid "me too" like answer.
Some ROM have a some offensive method of killing background apps with their own (obscure) reasons and there is nothing K-9 mail can do about that, you have to find how to disable this "feature" in your ROM. Typically, MIUI 7 to 8 use to be in this case, but, yeah, there are plenty of Android derivatives.

Typically, since Doze appeared in Android (version 6, or later for some android "enhanced" rom), it's not possible for a true open-source application to wake up when it wants, unless the application is excluded from Doze optimization (whatever the name the ROM modder called it).

There is only one exception to this rule: it's in Google's application stack, where a socket is allowed to be waken up upon any notification coming from Google. Google built the GCM/FCM system on top of this to (officially) allow other applications to wake your device upon notification, but it's a privacy leak and proprietary (since messages are captured in clear by Google's servers) so K-9 decided not to use it (and also because it requires you to share your email servers' credentials).

Most of other email clients switched to this system so, yes, other email client give the illusion of true "push" email. It's only an illusion, because what really happens under the hood is that your email's credential are shared to a remote email checking server that's doing the IMAP's IDLE/mail check work, and upon new email, this is this server that's calling into Google's GCM/FCM server and then Google wakes your device that's finally waking up the email client.

So, in the end, when using K-9 mail on a recent android phone, you're back in a "hackish" situation because Android's (default) Doze mode does not allow anymore a application to do whatever it wants whenever it wants. You have to disable the optimization for this application and Google is going to make this harder and harder in the future, since they want to spy on every message that goes to your phone (that's their business after all). When you disable optimization for K-9, then K-9 will be able to wake your phone up when it wants, and it'll become the number 1 consumer of your battery (at least, as reported by Android). This is also wrong, since when K-9 is waking up the phone, all other "waiting" application will be allowed to run (those that are usually waiting on Doze), so the phone will actually be running all your pending notification checker stuff. Yet, Android will record K-9 as being the "battery" consumer for the whole time.

For now, (but who knows for how long), the only workaround is to disable the Doze / Battery saver optimization for K-9 and expect an impact on your battery consumption. You might also gain some battery life by disabling pull mode (set it to never) if your IMAP server supports IDLE. Most of the time, it works as well as without pulling, but it's much lighter on the wake up cycle.

X-Ryl669 commented Mar 28, 2018

Guys, this is quite a long thread already. I'll try to sum this up, to avoid "me too" like answer.
Some ROM have a some offensive method of killing background apps with their own (obscure) reasons and there is nothing K-9 mail can do about that, you have to find how to disable this "feature" in your ROM. Typically, MIUI 7 to 8 use to be in this case, but, yeah, there are plenty of Android derivatives.

Typically, since Doze appeared in Android (version 6, or later for some android "enhanced" rom), it's not possible for a true open-source application to wake up when it wants, unless the application is excluded from Doze optimization (whatever the name the ROM modder called it).

There is only one exception to this rule: it's in Google's application stack, where a socket is allowed to be waken up upon any notification coming from Google. Google built the GCM/FCM system on top of this to (officially) allow other applications to wake your device upon notification, but it's a privacy leak and proprietary (since messages are captured in clear by Google's servers) so K-9 decided not to use it (and also because it requires you to share your email servers' credentials).

Most of other email clients switched to this system so, yes, other email client give the illusion of true "push" email. It's only an illusion, because what really happens under the hood is that your email's credential are shared to a remote email checking server that's doing the IMAP's IDLE/mail check work, and upon new email, this is this server that's calling into Google's GCM/FCM server and then Google wakes your device that's finally waking up the email client.

So, in the end, when using K-9 mail on a recent android phone, you're back in a "hackish" situation because Android's (default) Doze mode does not allow anymore a application to do whatever it wants whenever it wants. You have to disable the optimization for this application and Google is going to make this harder and harder in the future, since they want to spy on every message that goes to your phone (that's their business after all). When you disable optimization for K-9, then K-9 will be able to wake your phone up when it wants, and it'll become the number 1 consumer of your battery (at least, as reported by Android). This is also wrong, since when K-9 is waking up the phone, all other "waiting" application will be allowed to run (those that are usually waiting on Doze), so the phone will actually be running all your pending notification checker stuff. Yet, Android will record K-9 as being the "battery" consumer for the whole time.

For now, (but who knows for how long), the only workaround is to disable the Doze / Battery saver optimization for K-9 and expect an impact on your battery consumption. You might also gain some battery life by disabling pull mode (set it to never) if your IMAP server supports IDLE. Most of the time, it works as well as without pulling, but it's much lighter on the wake up cycle.

@maxbraketorque

This comment has been minimized.

Show comment
Hide comment
@maxbraketorque

maxbraketorque Mar 28, 2018

Thanks for providing additional information. I am currently trying PNF by MoboDev to see if that helps. My initial observation is that its not helping, but going to relaunch K-9 to see if that makes a difference.

Thanks for providing additional information. I am currently trying PNF by MoboDev to see if that helps. My initial observation is that its not helping, but going to relaunch K-9 to see if that makes a difference.

@maxbraketorque

This comment has been minimized.

Show comment
Hide comment
@maxbraketorque

maxbraketorque Mar 28, 2018

Well, I can't find any way to make push work with Android 7 on my GN8. PNF doesn't help, dropping IDLE Refresh time doesn't help, putting K-9 as an unmonitored app doesn't help, enabling "poll when connecting for push" doesn't help, and enabling regular polling doesn't keep push alive either. No matter what, after about 15-20 minutes, "Syncing Disabled" appears at the top of the Inbox.

With that said, I can temporarily bring push back to life by switching Background sync in the Global settings back and forth between "Always" and "Auto-sync". Can you guys create a routine that automatically switches back and forth between these settings every 10 minutes? Or perhaps cycles from Always->Never->Always quickly every 10 minutes?

Well, I can't find any way to make push work with Android 7 on my GN8. PNF doesn't help, dropping IDLE Refresh time doesn't help, putting K-9 as an unmonitored app doesn't help, enabling "poll when connecting for push" doesn't help, and enabling regular polling doesn't keep push alive either. No matter what, after about 15-20 minutes, "Syncing Disabled" appears at the top of the Inbox.

With that said, I can temporarily bring push back to life by switching Background sync in the Global settings back and forth between "Always" and "Auto-sync". Can you guys create a routine that automatically switches back and forth between these settings every 10 minutes? Or perhaps cycles from Always->Never->Always quickly every 10 minutes?

@CyberShadow

This comment has been minimized.

Show comment
Hide comment
@CyberShadow

CyberShadow Mar 28, 2018

FWIW, I (and probably many other users) would be fine with using GCM as a "ping-only" channel, i.e. transmit a content-less message that just tells K-9 to check for new mail. Getting an IMAP server to send a message via GCM is a different problem though, so this is probably best left for an add-on.

If it doesn't already, perhaps K-9 could expose an intent that another app can trigger to tell K-9 to check for new mail?

FWIW, I (and probably many other users) would be fine with using GCM as a "ping-only" channel, i.e. transmit a content-less message that just tells K-9 to check for new mail. Getting an IMAP server to send a message via GCM is a different problem though, so this is probably best left for an add-on.

If it doesn't already, perhaps K-9 could expose an intent that another app can trigger to tell K-9 to check for new mail?

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Mar 29, 2018

trostc commented Mar 29, 2018

@maxbraketorque

This comment has been minimized.

Show comment
Hide comment
@maxbraketorque

maxbraketorque Mar 29, 2018

Thanks. I wasn't able to find the option to disable optimization until just now. I have optimization turned off for K-9 now, so I'll report back after I see what that does.

Thanks. I wasn't able to find the option to disable optimization until just now. I have optimization turned off for K-9 now, so I'll report back after I see what that does.

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Mar 29, 2018

trostc commented Mar 29, 2018

@maxbraketorque

This comment has been minimized.

Show comment
Hide comment
@maxbraketorque

maxbraketorque Mar 29, 2018

Well, that appears to have done the trick. Thanks very much. I also have general polling turned off too, however it seems that I need to have "Poll when connecting for Push" to get Push to work. Sounds like its the same situation for you?

On another note, it seems like the devs could create a how-to or a notification within the app to tell users to turn off optimization and monitoring. I know other apps do that for specific Android settings that are needed for an app to function properly.

Well, that appears to have done the trick. Thanks very much. I also have general polling turned off too, however it seems that I need to have "Poll when connecting for Push" to get Push to work. Sounds like its the same situation for you?

On another note, it seems like the devs could create a how-to or a notification within the app to tell users to turn off optimization and monitoring. I know other apps do that for specific Android settings that are needed for an app to function properly.

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Mar 29, 2018

trostc commented Mar 29, 2018

@ikeeki320

This comment has been minimized.

Show comment
Hide comment
@ikeeki320

ikeeki320 May 15, 2018

@trostc What the difference between battery monitoring and optimization? On my S7, when I go into battery settings all I can do is add K9 to the unmonitored apps list. Does that list correlate to monitoring or optimization? Any idea how to do the other?

Did you disable battery monitoring AND optimization for k9? Must do both

@trostc What the difference between battery monitoring and optimization? On my S7, when I go into battery settings all I can do is add K9 to the unmonitored apps list. Does that list correlate to monitoring or optimization? Any idea how to do the other?

Did you disable battery monitoring AND optimization for k9? Must do both

@ajweber

This comment has been minimized.

Show comment
Hide comment
@ajweber

ajweber Jun 13, 2018

@CyberShadow @X-Ryl669
So I started down this road recently. One of the things I think I read (I looked at a lot) is that an app receiving a high priority Firebase Message (GCM) will come out of doze, but only that app.

My prototype idea is to test receiving the message and fire an intent on the mail app with the hope that doze allows that app to wake and use network resources. Again, what I read makes me think that may not work, but I will know soon.

I have tried many settings on my S8 including all those above, but I don't think they really work, and the android docs seem to indicate that they aren't what people think. Probably some luck, probably not actually in doze, etc.

If it doesn't work I may fork K9, only to add the relevant code from my prototype directly to the app. Of course I would give all that back to the K9 maintainers.

BTW, I am not looking at a proxy at all, but rather a different trigger (not sure it will work yet). Proxy is an interesting idea though.

ajweber commented Jun 13, 2018

@CyberShadow @X-Ryl669
So I started down this road recently. One of the things I think I read (I looked at a lot) is that an app receiving a high priority Firebase Message (GCM) will come out of doze, but only that app.

My prototype idea is to test receiving the message and fire an intent on the mail app with the hope that doze allows that app to wake and use network resources. Again, what I read makes me think that may not work, but I will know soon.

I have tried many settings on my S8 including all those above, but I don't think they really work, and the android docs seem to indicate that they aren't what people think. Probably some luck, probably not actually in doze, etc.

If it doesn't work I may fork K9, only to add the relevant code from my prototype directly to the app. Of course I would give all that back to the K9 maintainers.

BTW, I am not looking at a proxy at all, but rather a different trigger (not sure it will work yet). Proxy is an interesting idea though.

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Jun 13, 2018

trostc commented Jun 13, 2018

@ajweber

This comment has been minimized.

Show comment
Hide comment
@ajweber

ajweber Jun 13, 2018

As I said, it doesn't work. In "Optimize Battery Usage", K9 is toggled off. It is also listed in "Unmonitored Apps".

I'm not sure why you don't believe me. I almost didn't mention it, because I didn't want to get into an "argument" about whether the settings truly work.

ajweber commented Jun 13, 2018

As I said, it doesn't work. In "Optimize Battery Usage", K9 is toggled off. It is also listed in "Unmonitored Apps".

I'm not sure why you don't believe me. I almost didn't mention it, because I didn't want to get into an "argument" about whether the settings truly work.

@trostc

This comment has been minimized.

Show comment
Hide comment
@trostc

trostc Jun 13, 2018

trostc commented Jun 13, 2018

@ajweber

This comment has been minimized.

Show comment
Hide comment
@ajweber

ajweber Jun 13, 2018

Dovecot 2.0.9 IMAPS.

ajweber commented Jun 13, 2018

Dovecot 2.0.9 IMAPS.

@ajweber

This comment has been minimized.

Show comment
Hide comment
@ajweber

ajweber Jun 13, 2018

So here's where I'm at with this, in case anyone cares...

I have a test app that works "As Expected". I created a user in Firebase and created a tiny DB up there too. Firebase allows an authenticated user to receive GCM and read/write to the DB.

On app startup, I check if my user is correctly authenticated in Firebase (if not, I use an email and password - this can be changed and while it IS my target email, it is NOT my email server password). Once the Firebase user is valid, it pushes its latest device token to the mini DB. DB is KV: email_address and current device token/ID. There is a service running to monitor for token changes and for messages.

I have tested receiving messages with the app in foreground, background and forced the (virtual) device to Doze and it still receives the GCM messages...while that had some tricks to it, it is pretty-much "as advertised".

Currently messages are sent from a test harness: inotifywait script on my linux box that sees a change to the timestamp on the Maildir. For now, I hard-coded my device's token, but the idea is to do a simple lookup of the registered device by email address from Firebase DB. Use that unique ID to send the message via HTTPS. No storing of actual mail account password anywhere. (But that whole thing will get more robust.)

When a message is received, the idea is to launch a previously configured Intent. And the message receiver seems to run reliably, so the next step is to load K9 onto my test device and see if launching an intent on a separate app - that is still "dozing" - will allow the app to come out of doze too. I'm going to try with com.fsck.k9.intent.action.MAIL_SERVICE_RESET and see what happens. I honestly don't know if it's going to allow one app to force another out of doze.

If that doesn't work, I'm afraid the only option is to incorporate the enhancement directly into the app code...this, I'm very confident will work, but IDK if the amount effort is truly worth it. :)

Constructive comments welcome.

ajweber commented Jun 13, 2018

So here's where I'm at with this, in case anyone cares...

I have a test app that works "As Expected". I created a user in Firebase and created a tiny DB up there too. Firebase allows an authenticated user to receive GCM and read/write to the DB.

On app startup, I check if my user is correctly authenticated in Firebase (if not, I use an email and password - this can be changed and while it IS my target email, it is NOT my email server password). Once the Firebase user is valid, it pushes its latest device token to the mini DB. DB is KV: email_address and current device token/ID. There is a service running to monitor for token changes and for messages.

I have tested receiving messages with the app in foreground, background and forced the (virtual) device to Doze and it still receives the GCM messages...while that had some tricks to it, it is pretty-much "as advertised".

Currently messages are sent from a test harness: inotifywait script on my linux box that sees a change to the timestamp on the Maildir. For now, I hard-coded my device's token, but the idea is to do a simple lookup of the registered device by email address from Firebase DB. Use that unique ID to send the message via HTTPS. No storing of actual mail account password anywhere. (But that whole thing will get more robust.)

When a message is received, the idea is to launch a previously configured Intent. And the message receiver seems to run reliably, so the next step is to load K9 onto my test device and see if launching an intent on a separate app - that is still "dozing" - will allow the app to come out of doze too. I'm going to try with com.fsck.k9.intent.action.MAIL_SERVICE_RESET and see what happens. I honestly don't know if it's going to allow one app to force another out of doze.

If that doesn't work, I'm afraid the only option is to incorporate the enhancement directly into the app code...this, I'm very confident will work, but IDK if the amount effort is truly worth it. :)

Constructive comments welcome.

@jimimaseye

This comment has been minimized.

Show comment
Hide comment
@jimimaseye

jimimaseye Jun 13, 2018

I dont understand why people are struggling with Samsung phones. I'm not calling them liars or anything negative, Im sure their frustrations are genuine. As stated above I dont have the problem with K9 running on my Samsung Oreo android v8.0 and recently on Nougat 7.0). I have a mailserver that uses push, and I have mail notifications arrive on my home screen as soon as they hit the inbox on the server.

So, to be sure I will repeat the steps of my setup here and give screen shots.

As stated above:

Go to Andriod Settings- Battery.
Click on the "Battery Usage box toward the top center of the screen.
Click the Menu (Three vertical dots in upper right).
Click on Optimize Battery Usage.
Select "All Apps" from the drop down arrow at the top of the screen.
Scroll to K-9 Mail and toggle it Off

Also

  1. K9Mail is entered as an entry in to the battery 'Unmonitored Apps' (thus stopping it from being put to or suggested for putting to sleep)
  2. 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
  3. 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

SCREENSHOTS

K9 allowed to use Background battery usage:

image


K9 set as non-optmised app:

image


K9 as unmonitored app:

image


K9 allowing background data usage:

image

Note: for those that have Data Saver enabled in their phone, it is important to ensure 'Data Saver' is turned off if you are using Data Saver but want the notifications to still work. Otherwise, as the finction says, it will stop K9 from doing background syncing whilst Data Saver is enabled.


and of course...

K9 set to allow Background Sync (within the App Global settings):

image

I hope this helps.

One more thing I would say is to check and ensure the mail server you are receiving emails from actually uses push. Servers such as YAHOO only allow their email client (Yahoo Mail) to receive push notifications even though you are still using IMAP - K9 will not receive them and relies on polling. This is a Yahoo thing and Im sure other providers may do similar. So, double check before concluding it is the fault of K9 (assuming you have all of the above settings).

EDIT:

Over time I have noticed that rare but occasionally (say once a week, if that) it stops bringing in the email and working as expected. But this isn't to do with the settings or setup and turns out to be K9 'getting in a twist'. Simply doing a FORCE STOP of the K9 app and restarting it up again (or alternatively a more long-winded reboot of your phone) sorts it out.

jimimaseye commented Jun 13, 2018

I dont understand why people are struggling with Samsung phones. I'm not calling them liars or anything negative, Im sure their frustrations are genuine. As stated above I dont have the problem with K9 running on my Samsung Oreo android v8.0 and recently on Nougat 7.0). I have a mailserver that uses push, and I have mail notifications arrive on my home screen as soon as they hit the inbox on the server.

So, to be sure I will repeat the steps of my setup here and give screen shots.

As stated above:

Go to Andriod Settings- Battery.
Click on the "Battery Usage box toward the top center of the screen.
Click the Menu (Three vertical dots in upper right).
Click on Optimize Battery Usage.
Select "All Apps" from the drop down arrow at the top of the screen.
Scroll to K-9 Mail and toggle it Off

Also

  1. K9Mail is entered as an entry in to the battery 'Unmonitored Apps' (thus stopping it from being put to or suggested for putting to sleep)
  2. 'Allow background data usage' = Enabled (under Data Usage - Mobile Data usage)
  3. 'Allow app while Data saver on' is Enabled (because I also have data saver turned on phone my phone)

SCREENSHOTS

K9 allowed to use Background battery usage:

image


K9 set as non-optmised app:

image


K9 as unmonitored app:

image


K9 allowing background data usage:

image

Note: for those that have Data Saver enabled in their phone, it is important to ensure 'Data Saver' is turned off if you are using Data Saver but want the notifications to still work. Otherwise, as the finction says, it will stop K9 from doing background syncing whilst Data Saver is enabled.


and of course...

K9 set to allow Background Sync (within the App Global settings):

image

I hope this helps.

One more thing I would say is to check and ensure the mail server you are receiving emails from actually uses push. Servers such as YAHOO only allow their email client (Yahoo Mail) to receive push notifications even though you are still using IMAP - K9 will not receive them and relies on polling. This is a Yahoo thing and Im sure other providers may do similar. So, double check before concluding it is the fault of K9 (assuming you have all of the above settings).

EDIT:

Over time I have noticed that rare but occasionally (say once a week, if that) it stops bringing in the email and working as expected. But this isn't to do with the settings or setup and turns out to be K9 'getting in a twist'. Simply doing a FORCE STOP of the K9 app and restarting it up again (or alternatively a more long-winded reboot of your phone) sorts it out.

@McNobody

This comment has been minimized.

Show comment
Hide comment
@McNobody

McNobody Jun 14, 2018

I'm really thinking same way as @jimimaseye . I've used K9 at Samsung S8 (now Oreo updated) over a year perfectly fine with IMAP IDLE. The mail server is Dovecot 2.2.22.
I have configured all those settings like @jimimaseye 's great guide. I have Data saver off globally. Extra settings in K9 I have Update IDLE connection = 24 minutes.

One thing come to my mind is the network connection. Mine is working perfectly with Elisa 4G in Finland and with my own wifi. The Elisa 4G gives IP address from the 100.64.0.0/10 private network. Still IMAP IDLE is working perfectly with Elisa's NAT.

I can see the mobile network could possibly broke the TCP connection before the IMAP IDLE timeout. This could be one reason why it's not working for someone. The other reason could be operator specific firmware. And of course if the email service provider really support IMAP IDLE.

I'm really thinking same way as @jimimaseye . I've used K9 at Samsung S8 (now Oreo updated) over a year perfectly fine with IMAP IDLE. The mail server is Dovecot 2.2.22.
I have configured all those settings like @jimimaseye 's great guide. I have Data saver off globally. Extra settings in K9 I have Update IDLE connection = 24 minutes.

One thing come to my mind is the network connection. Mine is working perfectly with Elisa 4G in Finland and with my own wifi. The Elisa 4G gives IP address from the 100.64.0.0/10 private network. Still IMAP IDLE is working perfectly with Elisa's NAT.

I can see the mobile network could possibly broke the TCP connection before the IMAP IDLE timeout. This could be one reason why it's not working for someone. The other reason could be operator specific firmware. And of course if the email service provider really support IMAP IDLE.

@X-Ryl669

This comment has been minimized.

Show comment
Hide comment
@X-Ryl669

X-Ryl669 Jun 14, 2018

@jimimaseye and @McNobody, not everyone have Samsung phone, and their implementation is different on different vendor's OS. The problem I've on my phone is not reproducible on other's phone because each vendor implement doze the way they want and most of them think they are clever and also add additional "optimization" on it. On my phone, whatever the setting you set for "escaping" battery optimization (I don't have a data usage specific setting either), the OS will kill K9's process after a given time if it's not in focus, or if it needs memory or because the moon is shining.

In all cases, the only fact that each phone is different and that the dialogs to change the right setting is spread in numerous and hard to reach pages make the thing completely unusable for non-tech savvy users.
There is no other way than to bend to google's will here and use FCM/GCM since it's the only way that's guaranteed to work now and in the future.

@ajweber Can you give more details here ?
Once waken up, can you start K9 via an intent, and does it have network connection ?
If it's too hard to have K9 do what you want, maybe you can create another app that's downloading something upon responding to an intent and try with it first ?

X-Ryl669 commented Jun 14, 2018

@jimimaseye and @McNobody, not everyone have Samsung phone, and their implementation is different on different vendor's OS. The problem I've on my phone is not reproducible on other's phone because each vendor implement doze the way they want and most of them think they are clever and also add additional "optimization" on it. On my phone, whatever the setting you set for "escaping" battery optimization (I don't have a data usage specific setting either), the OS will kill K9's process after a given time if it's not in focus, or if it needs memory or because the moon is shining.

In all cases, the only fact that each phone is different and that the dialogs to change the right setting is spread in numerous and hard to reach pages make the thing completely unusable for non-tech savvy users.
There is no other way than to bend to google's will here and use FCM/GCM since it's the only way that's guaranteed to work now and in the future.

@ajweber Can you give more details here ?
Once waken up, can you start K9 via an intent, and does it have network connection ?
If it's too hard to have K9 do what you want, maybe you can create another app that's downloading something upon responding to an intent and try with it first ?

@LosD

This comment has been minimized.

Show comment
Hide comment
@LosD

LosD Jul 11, 2018

@jimimaseye How do you get into the real Android battery settings? On my S8+ (SM-G955F, FXXU2CRF7 software version, NEE region), Samsung seems to have ripped them out, only leaving their own... Which makes it impossible to disable optimization, only leaving it as unmonitored, which apparently doesn't help by itself.

LosD commented Jul 11, 2018

@jimimaseye How do you get into the real Android battery settings? On my S8+ (SM-G955F, FXXU2CRF7 software version, NEE region), Samsung seems to have ripped them out, only leaving their own... Which makes it impossible to disable optimization, only leaving it as unmonitored, which apparently doesn't help by itself.

@jimimaseye

This comment has been minimized.

Show comment
Hide comment
@jimimaseye

jimimaseye Jul 11, 2018

@LosD

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

That's the method I am using.

jimimaseye commented Jul 11, 2018

@LosD

SETTINGS - Device Maintenance - Battery - Battery Usage (under the 'estimated life' display) - three dots (menu) - Optimise Battery Usage.

That's the method I am using.

@LosD

This comment has been minimized.

Show comment
Hide comment
@LosD

LosD Jul 11, 2018

@jimimaseye

Holy hell, they hid that well. Not even a search found it, so I thought it was in the standard/old Android battery settings. Thanks!

LosD commented Jul 11, 2018

@jimimaseye

Holy hell, they hid that well. Not even a search found it, so I thought it was in the standard/old Android battery settings. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment