Push message when mentioned, followed, etc

[wirehack7]

App should send push message to the user when in Mastodon a notification occurs. Should be optional.

[kindlyfire]

Push notifications exist.

It's just a pain in the ass to setup as you have to recompile the app to change the tusky-api server.

[wirehack7]

I have to recompile it? I'm just using the version from Google Play.

[kindlyfire]

Yeah, you have to actually download the source, edit tusky_web_url in the file app/src/main/res/values/donottranslate.xml and then compile it.

[Vavassor]

@wirehack7 You're supposed to be receiving push notifications and there's a choice in preferences to disable it. The server for it is just broken. Or at least, it's delivering push notifications very unreliably and some users haven't received any.

[wirehack7]

@Vavassor will this get fixed? Or do we all have to compile the app ourself with own servers?

[Vavassor]

@wirehack7 It's going to be fixed. Originally Eugen (Gargron) wrote the server and I'm going to have a hard time figuring out how to fix it, myself. I've mostly been trying to fix crashes in the app and a major login bug for a release. So it won't be until after that.

I don't expect anyone to have to compile the app themselves unless they plan on working on the project or they want to fork it.

[kindlyfire]

If you need help to rewrite the server, I can offer some help.

[SkyzohKey]

@Vavassor Why not to use OwnPush instead ?

[ghost]

@SkyzohKey very interesting, bookmarked.

[pierreduchemin]

I've already had to push data on device without using firebase before. I have solved it with a service and a http server running on the device. This option totally solved the problem and was really good for my use case but it may be not very battery effective for Tusky.
Another option could be to rely on long polling, but it has some serious drawbacks especially with sleep mode.
Http/2 also have a push mode, but it doesn't seems to be made to do that.

I didn't know OwnPush before, it seems to be really interesting, but it is not usable yet : https://github.com/ownpush/docs/blob/master/FAQ.md
Anyway, I guess it also run a http server. Once again, not very battery effective, but it is smarter to have only one server running on device.

Last idea for now : https://f-droid.org/repository/browse/?fdfilter=push&fdid=org.androidpn.client
Do you know any app using it ? It seems to mimic firebase push mechanism.

First of all, I will try to implement a http server and to make it work with existing server. Then I will try to optimize it. Finally, when OwnPush became usable, it might be a really good option to migrate.

[SkyzohKey]

Up, just found some alternatives

• http://mqtt.org/
• https://pushy.me/ (uses MQTT under the hood)

Ok just tested Pushy and it looks really keen and polished, looks a nice alternative 😎

[pierreduchemin]

@SkyzohKey Pushy may be quite good, but it is not free. It's not better than Firebase. However, MQTT seems promising !
While searching for how to implement MQTT on Android, I found this project http://www.eclipse.org/paho/clients/android/. It could be a really good alternative to a whole http server in a service !

[SkyzohKey]

Yep forgot to link to Paho, sorry xD. Definitely something that could be used and works nice :)

[pierreduchemin]

I just tried MQTT push notifications : it works well !
At the moment, it is just a quick and dirty POC, so don't expect much : https://github.com/pierreduchemin/Tusky/blob/mqtt_notifications/app/src/main/java/com/keylesspalace/tusky/notifications/MQTTClient.java
This protocol seems to be what we need.

I used https://github.com/eclipse/paho.mqtt-spy/wiki to push notifications for now.

[pierreduchemin]

I've been working on MQTT last few days. Here are some conclusions :

• this protocol have been designed for iot : low traffic, low memory impact
• it's secured enough (ACL + TLS handled by broker)
• messages length limit is high enough (worked with a whole lorem ipsum)
• there are many implementations : https://github.com/mqtt/mqtt.github.io/wiki/software?id=software
• there are even public broker instances (useful for quick tests) such as iot.eclipse.org:1883

Since https://github.com/Gargron/tusky-api is in NodeJs, I think https://github.com/mcollina/mosca/wiki/Mosca-basic-usage might be a good option for us. If you know NodeJs and want to help, you can have a look on how to implement Mosca in tusky-api 😉

In my tests, I've been using http://emqtt.io/ and then Mosca (standalone) as broker, Paho as a client and MQTT-spy as a debug client.

I'm now trying to implement secure communications with Paho.

[Vavassor]

To my understanding, the current server just sends status ID's for the notifications through firebase? So the message length is very short and there's no visible notification contents even if the connection security was compromised.

@pierreduchemin Thanks for putting so much effort looking into this, by the way!

[Vavassor]

Also, an important thing to note with this system is there are currently roughly 35k active installs of Tusky. So volume is a real concern.

[pierreduchemin]

@Vavassor

• That's it, right now, server just send push an id, and we execute a new request to get notification content.
  Even this information could be sensitive and I really want to focus on user's privacy.
  It also allows us to avoid the 2nd useless request by sending notification content in push request. That would be a nice evolution 😉

• Since MQTT is designed for iot, it is totally able to handle these requests ! 😄
  There are many enterprise grade servers out there :

  • http://www.hivemq.com/
  • http://emqtt.io/
    Here is a complete tutorial on how to install such a server with Mosca :
    https://medium.com/@lelylan/how-to-build-an-high-availability-mqtt-cluster-for-the-internet-of-things-8011a06bd000

[jurassic-c]

@Vavassor @pierreduchemin If either of you are familiar with RabbitMQ, it's designed to handle a lot of traffic and comes with an adapter for MQTT. https://www.rabbitmq.com/mqtt.html

[wryk]

There is an ongoing implementation for the server ?

[Vavassor]

@wryk The currently-running server is this one https://github.com/gargron/tusky-api. Unless you're talking about the prototype being discussed here, in which case no not yet. But I'll post it here when I start on it (possibly tomorrow? or a couple days at most).

[SkyzohKey]

The version of Tusky from f-droid seems to have notifications o.O

[wryk]

@SkyzohKey It's only pull notifications. Tusky app regularly request server and show notifications. It's not instant, like push notifications.

@Vavassor Thanks, I'm talking about the mqtt server prototype. I'll have some free time and I wanted to contribute to the push notification server. I thought to use aedes (https://github.com/mcollina/aedes) from the mosca author and the redis backend (https://github.com/mcollina/mqemitter-redis).

[wryk]

Started a prototype implementation at https://github.com/wryk/tusky-api

[connyduck]

Yes we decided to switch to pull notifications for now. Of course push would be preferabe, but it would require a) a reliable open source push system and b) someone to maintain the server.

[RX14]

This should be done using the plain mastodon streaming API, no fancy separate servers. We just need an option in the streaming API to make it only send websocket messages for notifications, not other events (assuming the streaming API already sends enough information to render a notification, which might not be true).

Websockets are fine for push notifications in android, as long as the websocket only sends messages very infrequently. Websockets don't require pings, and neither does TCP, so an already-open websocket connection will not be sending any battery-wasting packets unless there's data, which is perfect for push. Sure, you'll probably need coordination with @Gargron to upgrade the streaming API (for which I can't find any docs to even tell if it does need to be changed) but this is vastly preferable to using a closed-source or tusky-specific service. Here's a quite interesting article describing the same problem in XMPP and how they fixed it.

[charlag]

@RX14

1. Maintaining an open connection and keeping device awake is not cheap no matter how frequent messages are
2. We cannot do it in the recent Android versions anyway because of the background execution limits.
   We probably could use streaming API while the app is open but we could as well just check for notifications more frequently while it's open.

We're not the messaging app. It is not part of our core functionality. In 2017 it's either FCM or pull notifications, I am sorry.

[RX14]

@charlag maintaining an entirely unused connection absolutely does not keep the device awake - it's an almost free thing. If android doesn't allow that, then obviously devs have abused that in the past to keep an active TCP connection open in the background which would drain the battery. That's a huge shame.

What causes battery drain is that recieving packets means the radios waking up the host CPU. If the TCP connection recieves no packets (which an idle websocket - with tcp and websocket pings turned off, but nobody uses either of those - will not do) then the CPU never has to be woken up to do anything with those packets. Sitting blocked on read(3) is free. It should be using much less battery than pull notifications.

I almost don't believe it's impossible though, considering that blog post clearly references keeping TCP connections open for notifications in android, and was only written in 2016.

EDIT: I'm not an android developer in any way, but these recent stackoverflow posts suggest it's possible: https://stackoverflow.com/a/43682770/2035962

[Vavassor]

There are limitations specifically on background services added in Android 8.0 that basically only allow a few minutes maximum after the screen is off before they're force-shutdown. Firebase seems to have some special privilege that allows it to get woken up by network events.

That said, I know pushy.me has accomplished this without Firebase and using MQTT somehow. But, they're closed source so I have no way of finding out how. From their documentation it seems like they have a custom intent and some way of broadcasting that to their service when a packet is recieved.

I know it's possible to open TCP connections in native code. So, it might just be that they're doing keeping them open there since it might not have the same restrictions applied to it? That's just a wild guess, though.

[RX14]

@Vavassor There are XMPP android apps which hand-roll XMPP push over a normal network socket. I'm not sure how they do it, perhaps a look at the source code would help: https://github.com/siacs/Conversations. Again, this blog post is relevant and worth a read.

Is this class relevant? XmppConnectionService

Again, I just know how TCP and networking work on typical OSes, i'm not sure what special considerations android has other than power usage. Firebase is just another android app, they probably use TCP too.

[Vavassor]

Also, for this thread's sake, I didn't follow up much about the MQTT prototype I did and removed. Basically it had a background service set up through Paho MQTT. The main issue was its service would get killed as soon as the device went into standby and it would also lose connection sometimes even in the foreground.

The server-side I was able to run reasonably stable. I only made a couple of hack changes to @wryk's code, so I didn't put them up anywhere. But the client-side I just could not get it to receive notifications as reliably as pull notifications.

[Vavassor]

@RX14 From the sound of it, here https://github.com/siacs/Conversations#how-do-xep-0357-push-notifications-work they use Google Cloud Messaging, which is the predecessor of Firebase Cloud Messaging. But all they use it for is to send an empty message to wake up the device, and then after it's awake, the background service then is able to talk to the server. And it says the non-play store version doesn't have push notifications.

[RX14]

@Vavassor yeah I saw that. There seems to be conflicting information of exactly how that app does push. The blog post talks about protocol improvements to make XMPP a true push protocol, implying heavily that they could do push direct to android. The readme however states they use GCM. Strange. Perhaps @iNPUTmice could bring some clarity.

I know it's possible to evade android doze with a foreground activity - the IRC app I use does it (and yeah it drains battery but IRC isn't optimized for push, it even has protocol pings...). The notification for doing this is hidden 99% of the time, until you open the notifications drawer - it doesn't appear in the activity bar. I'm not sure if doze is per-app though. Perhaps this act of opening a foreground activity disables doze globally, meaning other apps will increase their power draw.

At the least, I haven't noticed the (actually surprisingly small) battery drain affecting my day at all, even with IRC connected 24/7! It should at least be an option to do push notifications, even if it's not the default and comes with a battery warning.

[iNPUTmice]

The blog post talks about protocol improvements to make XMPP a true push protocol, implying heavily that they could do push direct to android.

XMPP has always been a push protocol. The blog post talks about making it more battery friendly.

The readme however states they use GCM.

GCM is entirely optional. If you make a doze exception for Conversations it will just keep a TCP connection to the server and GCM is not being used. If you decide to keep doze enabled it will use GCM to wake up the background service.
F-Droid builds won't use GCM and thus you always have to disable doze.

since it was mentioned elsewhere. A foreground service doesn't disable doze. All a foreground service does is make it less likely the service is being killed. Doze is orthogonal to that. Meaning having only a foreground service wont save you. And meaning you don't necessarily need a foreground service. Conversations doesn't have one.

That's for app targeting Android < 8.

On Android 8 things become a little more complicated. That's why Conversations won't target Android 8 for the foreseeable future. (It does run fine on Android 8 though)

[Gargron]

Just as a note, Mastodon supports Web Push Notifications natively. The protocol is open and some server implementations are open source (Mozilla's, for example). You can try to use those.

• https://developer.mozilla.org/en-US/docs/Web/API/Push_API
• https://github.com/mozilla-services/autopush

[RX14]

I run an IRC client on my phone that connects using native IRC to 6 IRC servers, i.e. it gets an IRC message every 10 seconds at least (irc has PING messages). Still doesn't drain my battery very much and never gets killed. I'd really enjoy the same mode (just give me push notifications, I don't care about battery life) in tusky, so I can finally migrate from tootdon to something open source. Really sad this still isn't fixed. There should at least be the option to "kill my battery life" (my personal evidence says this is a lie) if I want.

[charlag]

Okay
It's not us
It's Android
You cannot have unlimited background if you want to target at least Android O (and we want to).
We would not even care that it drains your battery
We just cannot do it.
We can run 4 times and hour, that's it.
I'm sorry but I'm afraid it's not going to be fixed because we cannot do anything about it. We can try dirty tricks like Riot does but it is tons of complexity and it still doesn't work reliably.
Google's monopoly sucks, I personally hate it.

[RX14]

@charlag you're saying on O+ doze exceptions no longer exist? If so, wtf?

[charlag]

Maybe I'm missing something, could you point me to it?

[RX14]

@charlag Last I heard (and i'm not an android dev) foreground services can still do whatever they want in the background, and the user can be prompted to do manual doze exceptions in the settings even if it's still a problem. I use android 7 still though, so I can't test that. It's just what I'm reading from the docs

[charlag]

From docs the only possibility I see is starting a foreground service.
I am not sure that I like it but it may be a kind of solution. It adds a lot of complexity, though. I think we have a lot more things to implement before considering this. Thanka anyway.

I didn't find any info about settings and doze exceptions, though, only opposite - applying doze to apps which don't target 26 - through settings.

[iNPUTmice]

From docs the only possibility I see is starting a foreground service.

FWIW coming November (and Google enforcing API 26 in the PlayStore) all IM apps will be using a foreground service. I guess that's something the user will have to get used to. Even if we don’t like it.

[RX14]

@charlag settings -> battery -> tripple-dot menu in top right -> battery optimization -> all apps -> find tusky -> don't optimize.

[RX14]

Foreground services aren't a problem to me. They don't show in your notification bar until you expand it anyway, even then they're pushed right to the bottom. They're effectively hidden.

[charlag]

Okay, thanks, I cannot confirm it works because I'm on 7.1 for now.

[RX14]

@charlag I tested this on 7.0 lol... Doze is still a thing on 7 it just doesn't bite as hard.

[charlag]

This is great but background limits are in effect only since O (8), aren't them?

[pierreduchemin]

As far as I can see, there are 2 different subjects here :

• how to transmit push notifications to device
  MQTT is a good solution, but @Gargron's solution is the Mastodon way to do that.
  It would however require an Android implementation of the JS Push API.

• how to stay listening to network on Android
  It now seems obvious that ForegroundService is the solution since it's not affected by doze : https://developer.android.com/training/monitoring-device-state/doze-standby.html#understand_app_standby
  Google seems to be willing to give control to users over what is running on their phones. Great! But it comes with some more complexity.

[charlag]

If we are to stay awake, we could use streaming but it requires a connection per client.
I have no idea how webpush works but maybe we could use it instead?

[RX14]

Web push seems to be implemented using http2, and could be quite complicated to implement. My first inclination would be to implement a new websocket endpoint for mastodon that sends notifications only, not timeline and notifications. Websocket clients are easy and numerous, and it should be easy to implement on mastadon's side, but the downside is that it requires people to update mastadon.

[RX14]

Also, perhaps this issue should be reopened?

[SkyzohKey]

It looks like Facebook Messenger does use MQTT instead of GCM:

• https://www.ibm.com/developerworks/community/blogs/mobileblog/entry/why_facebook_is_using_mqtt_on_mobile
• https://www.facebook.com/notes/facebook-engineering/building-facebook-messenger/10150259350998920