How to enable group chat push notifications?

[gertyq]

I have some questions about Push notifications implemented in Snikket. I have prosody server and before 30 Sep all iOS clients were ChatSecure. All push notifications (1:1 chats and group chats) worked. But I noticed that ChatSecure is not supported now (also because push notifications don't work) and tell all iOS users to delete ChatSecure and install Snikket. Now push notifications work but only in 1:1 chats. What should I change in prosody configuration or tell to change Snikket users in the app for enabling group chat push notifications?

What I understand now:

• When the users that are the members of persistent MUC are offline (not in the room) push notifications should not be sent. This can be fixed by using mod_muc_offline_delivery, but for now we don't need this feature, we just need receiving pushes when the user in the room.
• When the user closes the app by swiping (or how this calls) he goes to offline and push notifications will be deregistered. If the app closed by OS then push notifications will not be deregistered. It is normal behavior and if the user want to receive pushes from the Snikket he shouldn't close the app forcibly.
• So if the user didn't close the app and the new message in persistent MUC where he is a member has been sent by someone in the room then the push registration is enabled and the server should send push to push-ios.snikket.net which should send to iOS device.

But it doesn't. In all cases it is something like:

Oct 26 13:22:41 c2s14c6268	debug	Received[c2s]: <message to='MUC' id='90b2bf1c-4d8c-44a2-85d5-3ee381064789' type='groupchat'>
Oct 26 13:22:41 c2s14c6268	debug	Not archiving stanza: <message to='MUC' id='90b2bf1c-4d8c-44a2-85d5-3ee381064789' from='sender' type='groupchat'> (type)
Oct 26 13:22:41 xmpp_server:mam	debug	Not archiving stanza: <message to='member_who_away' id='90b2bf1c-4d8c-44a2-85d5-3ee381064789' from='MUC/sender_nickname' type='groupchat'> (type)
Oct 26 13:22:41 c2s152e350	debug	Sending[c2s]: <message to='member_who_away' id='90b2bf1c-4d8c-44a2-85d5-3ee381064789' from='MUC/sender_nickname' type='groupchat'>
Oct 26 13:22:41 c2s152e350	debug	hibernating since 2021-10-26T09:12:55Z, stanza queued
Oct 26 13:22:41 c2s152e350	debug	NOT invoking cloud handle_notify_request() for newly smacks queued stanza (session.push_identifier is not set: <nil>)

So as I see there hibernating state doesn't cause sending pushes. In 1:1 chats it cause.
I was trying to load mod_muc_cloud_notify under the global modules_enabled list, under the MUC component list and under both lists but in all these 3 cases after restarting prosody and re-registering pushes on clients nothing changes. Although as I mentioted above ChatSecure app was able to send pushes with this config earlier.

All Snikket clients have inactive toggle Push notifications in group chat settings:

Also in account/server settings all Snikket clients have When in Away/XA/DND state toggle inactive:

Is it normal? And in some cases one of Snikket iOS clients stay Available even when hibernating although another clients stay Away (as far as I understand is expected) - is it wrong app/iOS config or it is normal and it happens sometimes?

prosodyctl about output:

Prosody 0.11.10

# Prosody directories
Data directory:     /var/lib/prosody
Config directory:   /etc/prosody
Source directory:   /usr/lib/prosody
Plugin directories:
  /opt/prosody-modules/ - prosody-modules rev: 71bec9c21dcd
  /usr/lib/prosody/modules/
  

# Lua environment
Lua version:             	Lua 5.2

Lua module search paths:
  /usr/lib/prosody/?.lua
  /usr/share/lua/5.2/?.lua
  /usr/share/lua/5.2/?/init.lua
  /usr/lib/lua/5.2/?.lua
  /usr/lib/lua/5.2/?/init.lua

Lua C module search paths:
  /usr/lib/prosody/?.so
  /usr/lib/lua/5.2/?.so
  /usr/lib/lua/5.2/loadall.so

LuaRocks:        	Not installed

# Network

Backend: select

# Lua module versions
lfs:     	LuaFileSystem 1.8.0
lxp:     	LuaExpat 1.3.0
socket:  	LuaSocket 3.0-rc1
ssl:     	1.0.2

[mwild1]

Hi, thanks for the report :)

As noted in the release announcement:

Group chat notifications are not yet working. This will also be rolled out as a future server update.

We've made good progress on this, but a couple of to-do items remain to be implemented. I'm hoping to finish that this week, and then we'll be launching a new server release. That release will also add the app store link to Snikket invitations for the first time.

[gertyq]

Thanks for answer, but it is confusing me :)
How 3rd party apps like ChatSecure do this for a long time with prosody but your own app and your own server with your own modules can't do it o_0. I'm shocked a little bit... I propose that it is because that iOS app is the fork of Siskin IM that initially was made for Tigase server. But I really hope that you can do what you need as quick as you can and appreciate your efforts on this.
So please describe after that release what should I do on the server side and what should do the users on the client side?

[mwild1]

How 3rd party apps like ChatSecure do this for a long time with prosody but your own app and your own server with your own modules can't do it o_0. I'm shocked a little bit...

Background

In the beginning, the notification mechanism was the XMPP connection. XMPP is a real-time protocol designed around a persistent connection to the XMPP server.

Early mobile XMPP apps just kept a connection open permanently (automatically reconnecting if the connection dropped), and they would notify you if you received a message. However as the smartphone market grew, many kinds of apps were developed. Many (non-XMPP/communication) apps also wanted to notify the user about various things. Many of these apps were not designed efficiently - they would use too much bandwidth and battery power.

Mobile OS vendors (Apple and Google) began gradually restricting what applications could do. At first your app would be terminated by the OS if it used too much CPU or network activity in the background. Apple allowed an exemption for "VoIP apps" (some XMPP clients added VoIP support just to be allowed to continue running in the background). However Apple eventually removed even this exemption. In current releases of iOS and Android, the rules are stricter than they have ever been. People judge phones partly based on battery life, so this is one obvious reason for these restrictions.

Obviously they couldn't just remove the ability for apps to generate notifications, so they developed the concept of "push notifications". The OS itself would maintain a single connection to a notification service (both Apple and Google run such services). If app developers want their app to show a notification while it is not in active use, the app developer can submit the notification to the push service, which would relay it to the device. Ironically, Android and iOS both used XMPP for this push service connection. Android now uses a custom (protobuf?) protocol, iOS I'm not sure (but they still use port 5223 at least - the traditional port for XMPP+TLS).

Now that these push services exist, Apple and Google want all your notifications to go through them. They want apps to never maintain an open network connection, or perform any unnecessary background activity. Instead if your app receives notifications from the network, these have to go through the push service.

Strategy 1

Now, rather than redesign mobile XMPP clients to tunnel everything through these push notification services, a clever hack was introduced. The client connects as normal, and registers its push notification info (app id, etc.). Eventually the OS will terminate it when it is in the background, and the connection will be closed.

Instead of logging out when the OS terminates the client, the clients let the connection drop. On modern XMPP servers using XEP-0198, this would keep the client session appearing online for a period of time, even though the client was not connected. In Prosody we call this a "hibernating session".

When a hibernating session receives some (even relatively unimportant) traffic, this would generate a push notification via Apple/Google. When the OS receives the push notification, it would allow the app on the device to run and handle it. Clients would use this opportunity to reconnect to the XMPP server, resume their hibernating session (a feature provided by XEP-0198), and retrieve any pending stanzas (queued up for them by the server). The notification is not shown to the user (but if any messages are fetched from the XMPP server, notifications are generated).

The app would then continue to run for a while longer, until the OS decides to terminate it again. This leads to a situation where the app is quietly running for a while, terminating, reconnecting, running for while again, terminating, and so on. On Android this constant reconnection has a workaround - the app can register a persistent notification. On many (but not all) Android devices, this will minimize the chances of the app constantly being stopped by the OS. There is no such workaround on iOS that I'm aware of.

Benefits:

• Minimal changes to how XMPP traditionally works
• Sensitive data does not flow through Apple/Google servers (the content of the push notification doesn't matter, because the XMPP client is just going to retrieve messages from the server as normal, when it gets woken up by the notification).

This approach has some downsides:

• There is often a continuous disconnect, reconnect, disconnect cycle going on - particularly on iOS. This can potentially lead to higher network and power consumption than a normal mobile-optimized XMPP connection staying open.
• This "hack" (launching the app for as long as possible when receive a push notification) is not in line with how Apple and Google want apps to behave. Background restrictions have been getting stricter with every OS release, and it's very possible they will attempt to close this loophole in the future.

Clients using this strategy: Conversations (and forks), ChatSecure, Monal (pretty much all)

Strategy 2

The alternative is to play the game the way Apple and Google want you to. Use push notifications only for messages that should be shown to the user. Don't connect to the XMPP server unless there is a reason (i.e. the user opens the app).

Benefits:

• Significantly more battery friendly
• Less likely to be broken by changes to future versions of Android/iOS

Downsides:

• Notification content goes through Apple/Google. This has been solved by adding a layer of encryption around the contents and metadata. This is one of the custom extensions Snikket currently uses.
• The XMPP client is shown as offline to contacts and MUC rooms. This is why group notifications don't just work - a MUC traditionally only sends messages to JIDs that are connected to a room.

Clients using this strategy: Siskin, Snikket iOS

Because this strategy requires changes (such as encryption of push notifications), it currently only works with Tigase (and, just about, Prosody) which have implemented these changes. Eventually these changes (or equivalent ones) will hopefully make it through the XSF standards process, and be more widely adopted by all servers.

Why did we choose strategy 2?

Due to the required changes, it's more work to get it working right in the short-term, however I believe strategy 2 is the "correct" approach due to its increased efficiency and future-proofness. I think that makes this work worth doing.

In an ideal world we would be able to just keep an open connection to the XMPP server and apply optimizations ourselves, but the reality is that today the rules are dictated by Apple, Google and other vendors, and we don't have many choices here.

I know this is very long, but I even missed out some details - such as how Apple/iOS has different types of push notifications with different characteristics. It's a minefield, and we're doing the best we can :)

[gertyq]

Very thanks for this great explanation. Now I've seen everything :)
I really hope that this longread was written not only for me and you didn't waste your time because of only this question.
I think you choose right strategy and short-time difficulties with long-term benefits is much better than easy way.
I've heard that Apple has different type of pushes (and so called VoIP pushes) and even that they provide some undocumented API for a little group of partners (https://habr.com/en/post/580272/) that I think is very sneaky. So now I know who can be blamed in this situation and I wish you to win in this dirty unfair game for the good of all mankind :)

[tpikonen]

Thanks a lot for this thorough explanation. Apparently the group chat notifications still do not work in snikket-ios but this issue is closed. Is there an open bug somewhere, so that I can test this feature when it's closed?

[mwild1]

We're working on a new server release, it should be published this week. It will be announced on the Snikket blog, Mastodon and Twitter.

[gertyq]

Thanks for your efforts. What should be done to make pure prosody-stable compatible with this feature if it is possible?

[mwild1]

@tpikonen In case you missed it, the release is now available: https://snikket.org/blog/nov-2021-server-release/

@gertyq Prosody 0.11.x cannot work with this because a core change was required. This change will be in Prosody 0.12 which we hope to release "soon" (coming weeks/months, as time allows). Apart from that change, the MUC service needs mod_muc_offline_delivery and the user's host needs mod_cloud_notify_extensions. There are also two Snikket-specific modules, mod_snikket_client_id and mod_snikket_ios_preserve_push - these work around some issues I discovered recently that affect all Siskin and Snikket pushes from Prosody in certain edge cases. Unfortunately they are a bit hacky, and specific to the Snikket iOS client, which is why they are not in prosody-modules like the others. I want to develop a standard cross-client solution in Prosody and the app, hopefully by early next year (but releasing Prosody 0.12 takes priority for now). Hope this helps!

[tpikonen]

Thanks again for your work. I'm also interested in this feature in pure Prosody. Did I understand correctly that Snikket-IOS group chat push notifications will not work on Prosody version 0.12, but maybe on 0.13?

[mwild1]

@tpikonen they will work in 0.12 with the necessary supporting modules (which will not be included by default, but will be easily installable). You can try this today with Prosody trunk nightly builds for example.

[gertyq]

Thanks, is it possible to build trunk version for arm?

[mwild1]

Yes, Prosody trunk packages are built for ARM on Debian/Ubuntu. See https://packages.prosody.im/debian/

[gertyq]

I have converted deb package to alarm and after solving several dependency problems, I was finally able to run it with all modules mentioned above.
It works! Now group chat notifications via muc_offline_delivery successfully invoke for every member who hasn't Available status. But there are two problems: Snikket UI has strange behaviour: after enabling "Group chat notifications" toggle it becomes inactive. The second problem: after enabling mod_cloud_notify_encrypted every push notification fails with the error:

Traceback[c2s]: ...od_cloud_notify_encrypted/mod_cloud_notify_encrypted.lua:127: bad argument #2 to 'final' (string expected, got light userdata)
	stack traceback:
	[C]: in method 'final'
	...od_cloud_notify_encrypted/mod_cloud_notify_encrypted.lua:127: in field '?'
	/usr/share/lua/5.4/prosody/util/events.lua:81: in function </usr/share/lua/5.4/prosody/util/events.lua:77>
	(...tail calls...)
	/opt/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:385: in upvalue 'handle_notify_request'
	/opt/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:569: in field '?'
	/usr/share/lua/5.4/prosody/util/events.lua:81: in function </usr/share/lua/5.4/prosody/util/events.lua:77>
	(...tail calls...)
	/usr/lib/prosody/modules/mod_mam/mod_mam.lua:447: in field '?'
	/usr/share/lua/5.4/prosody/util/events.lua:81: in function </usr/share/lua/5.4/prosody/util/events.lua:77>
	(...tail calls...)
	/usr/share/lua/5.4/prosody/core/stanza_router.lua:188: in upvalue 'core_post_stanza'
	/usr/share/lua/5.4/prosody/core/stanza_router.lua:128: in upvalue 'core_process_stanza'
	/usr/lib/prosody/modules/mod_c2s.lua:322: in upvalue 'func'
	/usr/share/lua/5.4/prosody/util/async.lua:127: in function </usr/share/lua/5.4/prosody/util/async.lua:125>

At this time I have disabled this module and all we need works now. Thank you for your efforts and good luck with the next stage of your work.

[mwild1]

Snikket UI has strange behaviour: after enabling "Group chat notifications" toggle it becomes inactive

Ensure you have a recent (within the past ~2 weeks) trunk nightly build of Prosody, and add this option under the Component line for your MUC service:

muc_registration_include_form = true

after enabling mod_cloud_notify_encrypted every push notification fails with the error:

I really can't explain this one. It says a light userdata (a pointer-like object that Lua modules written in C can create) is being passed instead of a string. However the value it is complaining about (push_json) comes from Prosody's util.json library, which definitely returns a string.

[gertyq]

muc_registration_include_form = true

After enabling this option the toggle doesn't work as expected but we have faced strange behavior on Snikket-iOS devices:
Tap on the toggle causes leaving the room and after closing/opening the app this room disappears from the conversations list

And one more strange thing has been going on for the last week in group chats:

• Sending from Snikket-iOS for another Snikket-Android viewing is always good
• Sending from Snikket-Android for another Snikket-Android viewing is always good
• Sending from Snikket-Android for another Snikket-iOS viewing is always good
• Sending from Snikket-iOS for another Snikket-iOS viewing is good only if both accounts/devices are online, but if viewing device is offline then push notification is successfully sent but when the app is opened again this offline message looks like:

After that if both devices are online the messages between them are seen good again. Unloading muc_offline_delivery turns off group chat pushes but this behavior remains. Loading/unloading cloud_notify_* modules doesn't change anything. Personal communication is good in any cases so I think it somehow related to archiving offline messages in group chats for or from iOS devices.