HTML5 push notifications - not received while phone is unused

[andriej]

Home Assistant release with the issue: 0.90

Last working Home Assistant release (if known): hard to say

Operating environment (Hass.io/Docker/Windows/etc.): Linux / venv / Python 3.7.2

Component/platform: html5 notify

Description of problem:
Since few versions (but it was hard to notice) HTML5 push notifications are not received by android phones while the phones are laying unused.
Push notifications work properly when testing with active screen or few minutes after it. If the screen is off (idle) for longer period, the notification never reaches the phone - not even when the phone is unlocked/active again.

On one of the groups we've managed to confirm problem on different phone brands, all with latest android 9 system. Also all of us run python 3.7.2.

Someone has also noticed that it may be because of some undocumented (in FCM Firebase) priority type, which defined, may change the behaviour and 'force' phone to show notification even when phone is not in use.

There's no issue with other notifications in meanwhile (communicators like whatsapp/messenger etc.)

Could person responsible for the html5 (@robbiet480 ?) could take a closer ook at this issue?

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant): not relevant

Traceback (if applicable): no error messages

Additional information:
What's worth nothing it's not the matter of phone power-save features - we've tried different settings including turning them off.

[bieniu]

Same issue here. OnePlus 6 (Android 9) with turned off all possible battery optimisation services. HA 0.89/0.90, Linux, venv, Python 3.7.2.

[robbiet480]

@andriej I'm not gonna be able to help out with this as I only use Chrome for HTML5 Push Notifications.

[pergolafabio]

yes , happens to me also occasionally, also oneplus 6T user here
when i turn on the phone, i see all notifications coming
i think the "chromebrowser" app is closed after a while , maybe?

[dshokouhi]

I think sometimes it may not be that your phone dozes but that googles servers are delayed. If it does have to do with doze I would be curious to see if the FCM push notification works while HTML5 does not. Personally I have seen myself and others at the same time see the same delays so it leads to me think it was a server issue, I had to stop using HTML5 as a result of that. The notifications would be delayed by like 5 minutes or more and that was not suitable for an alarm.

[asjmcguire]

Doze means the phone will only wake up periodically to download batches of notifications. However if the developers set the urgency of the message then it will normally be delivered immediately to the device causing the device to wake up. As seen here - https://firebase.google.com/docs/cloud-messaging/concept-options#setting-the-priority-of-a-message a webpush would have the priority defined in the header of the push, and I don't see anything setting the priority in the notify/html5.py code.

[andriej]

Seems like lack of priority may be the issue!

[robbiet480]

@andriej Can you submit a PR to update docs to note that?

[andriej]

Isn't it lack of priority declaration in the code that @asjmcguire meant?

[robbiet480]

Misread, sorry @andriej, yep you are correct. Mind submitting a PR to add priority to the code?

[andriej]

That's the moment I can't help - I don't know python. ;-)

[andriej]

@robbiet480 - what one of guys from polish HomeAssistant group described as resolution is working payload.
what he says is differentiation for VAPID and is crucial to get it working is not only 'high' android priority but also "priority" => 10.

Maybe someone (@bieniu ? :-)) could test it to be added too.

[bieniu]

I tested it. I added these lines to the component

payload['android'] = {'priority': 'high'}
payload['priority'] = 10

and unfortunately this didn't help.

[pszafer]

@bieniu so now go to Advanced settings->Battery manager->protected aps or in Sony case Battery -> Excludes from Power Saving and tick Chrome.
For me after 20 minutes, still getting notification with screen off.

[pergolafabio]

That's not possible on a OnePlus, some apps you can indeed exclude from battery saving, but not chrome

[pszafer]

@pergolafabio don't worry. It doesn't help. After 40 minutes phone is not getting notification at all...

Might be related: https://bugs.chromium.org/p/chromium/issues/detail?id=935931

[robbiet480]

If y'all come up with a new fix I can submit a PR with it if no one else is willing to.

[robbiet480]

Related forum thread

[pszafer]

I changed code to something like this:

            jwt_claims = {'exp': jwt_exp, 'nbf': timestamp,
                          'iat': timestamp, ATTR_TARGET: target,
                          ATTR_TAG: payload[ATTR_TAG]}
            #jwt_token = jwt.encode(jwt_claims, jwt_secret).decode('utf-8')
            #payload[ATTR_DATA][ATTR_JWT] = jwt_token
            android_headers = { 'priority': 'high', 'ttl':86400 }
            webpush_headers = {'headers': {'Urgency': 'high'}, 'ttl': 4500}
            info[ATTR_SUBSCRIPTION]['android'] = android_ttl
            info[ATTR_SUBSCRIPTION]['webpush'] = webpush_headers
            info[ATTR_SUBSCRIPTION]['priority'] = 10
            payload['android'] = android_headers
            payload['priority'] = 'high'
            payload['webpush'] = webpush_headers 
            if self._vapid_prv and self._vapid_claims:
                response = webpush(
                    info[ATTR_SUBSCRIPTION],
                    json.dumps(payload),
                    vapid_private_key=self._vapid_prv,
                    vapid_claims=self._vapid_claims,
                    ttl=86400
                )

Now I receive message when I wake up device, but still no wake...
Can somebody check if has similar behaviour with such code?

[bieniu]

@pszafer I tested your fix and the effect is the same. The notification comes after phone wake. This isn't ideal but it's a huge improvement.

[bieniu]

I found this about high priority notifications: "High priority messages generally should result in user interaction with your app or its notifications. If FCM detects a pattern in which they don't, your messages may be de-prioritized. Android P introduced app standby buckets which limit the number of FCM high priority messages you can send to your app that don't result in the user using your app or viewing a notification. If, in response to a high priority message, a notification is displayed in a way that is visible to the user, then your app standby bucket quota will not be consumed by that message."
https://firebase.google.com/docs/cloud-messaging/concept-options
Maybe we have bad habits (dismissing notifications from HA) and that's why high-priority notifications not wake our phones.

[pszafer]

@bieniu let's try to narrow down issue.
https://gist.github.com/pszafer/f8ad3782ef3d48c98471bd6cf1632cf0

I think that we need ttl in webpush method and in webpush headers... It's hard to investigate on one phone.

About second. I always click on notification and get back click event.
Our receiver of messages is Chrome / web browser, so I think html5 push notification to chrome never gonna land in prioritzed bucket.
I think this is our bug: https://bugs.chromium.org/p/chromium/issues/detail?id=777106

[bieniu]

@pszafer On this version also, I receive notification after phone wake up. If I send two notifications when the phone is sleep then I receive only the first ones after wake up.

[Konstigt]

Yikes.. I have both this problem but also that notifications aren't sent at all after "a while" because "UnauthorizedRegistration". That started after switching from GCM to VAPID.

#22314

[asjmcguire]

@bieniu let's try to narrow down issue.
https://gist.github.com/pszafer/f8ad3782ef3d48c98471bd6cf1632cf0

I think that we need ttl in webpush method and in webpush headers... It's hard to investigate on one phone.

About second. I always click on notification and get back click event.
Our receiver of messages is Chrome / web browser, so I think html5 push notification to chrome never gonna land in prioritzed bucket.
I think this is our bug: https://bugs.chromium.org/p/chromium/issues/detail?id=777106

I'd be inclined to agree that is the problem, having read the entire bug post - it's clear that expecting this to be fixed anytime soon is a lost cause. It can't hurt Home Assistant to ensure the priority metadata is set - but it looks like setting Chrome to be excluded from battery optimising techniques is the only solution for the moment.

[bieniu]

@asjmcguire It's not always possible to exclude Chrome from battery optimization. On OnePlus phones Chrome is a system application and You can't exclude system applications. You can exclude Chrome Beta but it doesn't help. I checked this today. With Beta notification when the phone sleeps isn't delivered. The next problem is that if several notification are sent when the phone sleeps, only the first ones will be delivered after phone awake. The rest of the notifications will be lost.

[pszafer]

@bieniu about several notification you have to modify tag to not replace previous notification. Isn't it working for you?

[pszafer]

I commented out jwt_exp in first version because I couldn't find any docs that it is needed in fcm. IMO it's not needed but I'm not sure. In latest version I came back to jwt because it doesn't break things either.

TTL shouldn't be necessary as critical notifications should have TTL 0 (phone call) but for chrome it doesn't always work...

[bieniu]

In my opinion, TTL should be used. For example, the user has switched off the phone and during this time gets critical notification from the Home Assistant (alarm, movement, gas leak, smoke detection, etc.). After turning on the phone, the user should get a notification and not get it with TTL = 0.

[andriej]

In my opinion TTL=0 would work if all notifications would get to the phone and wake it up. Currently because of this parameter those are 'dying' on the way, while phone is in doze.

[robbiet480]

Relevant

[andkit]

In my opinion TTL=0 would work if all notifications would get to the phone and wake it up. Currently because of this parameter those are 'dying' on the way, while phone is in doze.

I rather agree with @bieniu, ttl=0 is not only an issue for Doze. There's also spotty mobile coverage, airplane mode and rumor has it that some people even turn their phone off occasionally, or forget to charge it ;)

Therefore I think making ttl configurable per notification might be the best solution, since not all notifications stay relevant equally long.

[ghost]

To rule out Chrome issues, I have tried testing HTML5 push notifications in Firefox for Android. Without the TTL being set neither push notifications in Chrome, nor in Firefox work when the phone (Moto 5G plus) is in doze. I can confirm that push notifications don't work when the phone is in doze with Brave either (but its chromium based). It doesn't matter if the priority flag is set is to high. With the TTL value set to a different value (I used 36000) Firefox will display the push notification when the phone wakes from doze, but Chrome has still dropped the push notification.

So I think there are two separate bugs causing the problem.... One in Chrome which we can't fix. The other is in home assistant, with the TTL being set to zero, which is expiring the notification when the phone is in doze.

[andkit]

If it helps: I have used patched my instance with a ttl of 3600s and the priorities mentioned above.
On my phone I use home assistant in "webapk" mode or whatever it's called (add to Homescreen)
I generally use Chrome canary (also set as webview implementation), so excluded both assistant and chrome canary from battery optimization.

With these settings my html5 notifications work fine

[pergolafabio]

On OnePlus phones, we can't exclude chrome from battery optimalization

[bieniu]

@pergolafabio You can exclude Chrome Canary from battery optimization but can't stable build.

[pergolafabio]

Aha, didn't know what canary was , thnx for info

[ghost]

Extended testing has revealed that eventually the firefox browser started dropping and completely ignoring push notifications while the phone is in doze. It doesn't matter if battery optomization for firefox is disabled. So I suspect there there is indeed an algorithm at work within android deciding that the behavior of home assistant's push notifications is too spammy, too potentially battery wasting, and not meeting their user interaction metrics and thus the expected behavior is being silently modified in the name of optimizing my android user experience.

[thomasgermain]

Just tested the fix in #22988.

I created an html5 custom component containing changes of the PR
I tested with my oneplus 5 (android 9) with "webapk" (add to homescreen). It works !

I tested like this:

• phone in use -> working
• phone just went to sleep -> working (notification is displayed "instantly" and it's waking up my phone)
• phone in doze (>5 minutes) -> working (notification is displayed with 1~5 seconds delay and it's waking up my phone).

Also tested with my wife's phone (samsung A5 2017, android 8) which was already working, everything works fine too.

Thanks for the fix !

[pszafer]

@thomasgermain so it's far better working for you than for me :)
My phone never wakes, but notification is always received after wake up (Sony XZ1 /Android 9).

[thomasgermain]

@pszafer sorry for being too optimistic :(

I forgot to mention that the webapk is not battery optimized, but I guess is something you already tried.

[ghost]

edit Test environment is a MotoG5 Plus (Android 8.1.0) and Chrome Canary, with home assistant as a webapk with battery optomization disabled. After patching the correct file FCM push notifications now are received reliability upon waking at the very least.

[robbiet480]

Again relevant to this conversation is the previously mentioned website https://dontkillmyapp.com/. You should check the site for details about what extra measures, if any, your manufacturer takes to conserve battery.

[0BuRner]

For those who have time to inspect deeper what's going on, BatteryHistorian is pretty useful : https://developer.android.com/studio/profile/battery-historian

[e2m32]

Just tested the fix in #22988.

I created an html5 custom component containing changes of the PR
I tested with my oneplus 5 (android 9) with "webapk" (add to homescreen). It works !

I tested like this:

• phone in use -> working
• phone just went to sleep -> working (notification is displayed "instantly" and it's waking up my phone)
• phone in doze (>5 minutes) -> working (notification is displayed with 1~5 seconds delay and it's waking up my phone).

Also tested with my wife's phone (samsung A5 2017, android 8) which was already working, everything works fine too.

Thanks for the fix !

#22988 Did the trick! Thanks! I just copied @pszafer' s files from here and put them in <config>/custom_components/html5. After a reboot, the notifications are coming in again 🎉🎉.

I did do a code edit to make all html5 messages 'high priority'. This way there is no delay in the message. Most of my messages are from my alarm, so I need them to come in as soon as possible.

Line 84:
DEFAULT_PRIORITY = 'high'

It looks like this might be able to be configured from config.yaml or even for each automation, but I didn't want to take the time to find out. This way I know they are all set to high priority.

I am on a Pixel 2 XL running Android 9 with battery optimization off on Chrome and the webapk. I tried Chrome Canary at the same time and there was no difference in responsiveness, but I do really like the app-like feel that canary offers with the webapk. I'm going to keep that :-)

[nickm324]

I did the same thing and it prevented HA lovelace from being accessible. I could get to the samba shares so it appeared that HA was loading and running. But until I removed the HTML5 folder from custom_components folder it was not letting the UI come up. Did I miss a step?

[jurgenweber]

Looks like it is all related; #23613

[stale]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[pergolafabio]

hi, whats the actual status on this? is it possible now to have the messages instantly ?

[thomasgermain]

Hi,

you will have to set priority case by case. This is working for me. I have some prior notifications that are waking up my phone and other non-prior notifications I will only see when unlocking my phone.

See https://www.home-assistant.io/integrations/html5/#ttl-and-priority

[pergolafabio]

yeah, i know, setup like this now, but not reliable

  - service: notify.html5
    data_template:
      message: "Er staat iemand aan de deur!"
      title: "Deurbel"
      data:
        tag: alert
        image: !secret snapshot_buiten_deurbel
        vibrate:
          - 300
          - 100
          - 400
        renotify: 1 
        ttl: 86400
        priority: high

[fleshinachair]

I'm hoping this will all be resolved the when the Android app is released?

…

On Tue, 8 Oct 2019, 14:03 pergolafabio, ***@***.***> wrote: yeah, i know, setup like this now, but not reliable - service: notify.html5 data_template: message: "Er staat iemand aan de deur!" title: "Deurbel" data: tag: alert image: !secret snapshot_buiten_deurbel vibrate: - 300 - 100 - 400 renotify: 1 ttl: 86400 priority: high — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#22249>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHRBATYTQ7YPXWX3ZP2RP7LQNSALNANCNFSM4HAFW4KA> .

[andriej]

Hopefully. Right now most of the messages arrive to the phone, except they are not waking it up.

[pergolafabio]

yeah, same behaviour here, when i wake up the phone, i receive the message
so not usefull for a doorbell automation :-)

i started today with the "pushover" integration, seems the work verry good and verry easy
for me an alternative untill html5 is working again