Websocket support #1000

Closed
JavaJens opened this Issue Mar 4, 2014 · 232 comments

Projects

None yet
@JavaJens
JavaJens commented Mar 4, 2014

As #127 is a little overloaded and goes beyond the technical, I thought to open up this issue to gather implementation details and coordinate efforts for this.

Support for WebSockets is already present on the server, so the idea is to use WebSockets on Android as well (for now instead of a possible more suited protocol like MQTT)

@geileszeuch mentioned a guide on how to use websockets in native apps: http://www.elabs.se/blog/66-using-websockets-in-native-ios-and-android-apps

However I don't think this would be a good guide as it doesn't talk about Wake/Sleep and connection loss.
This article list several issues one has to deal with in a mobile enviroment: http://dalelane.co.uk/blog/?p=1599
I believe the approach mentioned here http://stackoverflow.com/a/16602970 to be reasonable, if I find the time I will give it a try.

I guess for developing this one has to run an own server, as WS is disabled per default.

So please, if you have a question, concern or are already working on this: comment!

@moxie0 moxie0 self-assigned this Mar 4, 2014
@nameless-

Is there any technical reason to use web technology ? why not any XMPP like ?

@JavaJens
JavaJens commented Mar 4, 2014

There is already support in the Server :)

@nameless-

As you said, a mobil environment is likely to break connections and thus we will be force to reimplement what already exists in session protocol like XMPP or SIP that is registration, session management, etc.

I think it would be easier to develop something on top of a session protocol even though it will require some slight modification on the server side.

@JavaJens
JavaJens commented Mar 4, 2014

Do you have anything particular in mind? I haven't checked the code, but
maybe the GCM server part is using XMPP already which could be reused?
I know @v-0-d mentioned MQTT in the other thread, which sounds pretty good
as well.

Nevertheless, I think the current server can handle session mobility to
some extend, but I haven't looked very thoroughly.
From what I see the client simply has to connect (and reconnect if needed)
to the server and listen to incoming messages which are queued until
delivered. And as a client as a clear identification number the mapping to
a WebSocket connection shouldn't be such a big deal.

@JavaJens
JavaJens commented Mar 5, 2014

I've created a local branch with very rudimentary support, more a PoC: https://github.com/JavaJens/TextSecure/tree/feature-1000
Maybe @thors you can take this as a starting point?
@TheBlueMatt is working on the desktop client afaik. Maybe he can provide some insight to the registration for WebSocket devices?

@geileszeuch

He recently open sourced bis early version of the textsecure browser extension. Hopefully this will provide some insight: https://github.com/TheBlueMatt/textsecure-chrome

@JavaJens

Yes, I looked into that. right now multiple devices can be registered on one number.

For the Android version i've got to make little changes therefore.

Right now the device connects and all seems well, I only have to test sending/receiving messages which doesn't work right now.

Probably because I only have one device and try to work with the emulator and a modified server version.

If you want you can checkout my feature branch and test yourself :)

cheers

@mank319
mank319 commented Mar 18, 2014

Thank you for your efforts! I will checkout your branch as soon as my current exams are over and maybe even contribute something.

I hope that we will soon be able to (optionally) use TextSecure without GCM (and PlayStore in general!).

@SecUpwN
SecUpwN commented Apr 1, 2014

@JavaJens, thank you for opening up this issue, I can't wait for this to be implemented! ;-)

@JavaJens
JavaJens commented Apr 1, 2014

Feel free to test by branch and contribute. Sadly I'm pretty busy right
now, but rudimentary support exists :)

@SecUpwN
SecUpwN commented Apr 1, 2014

@JavaJens, is there a compiled WIP-version somewhere or do I have to compile it by myself?

@JavaJens
JavaJens commented Apr 1, 2014

Sadly, no!

@mcginty mcginty added the feature label Apr 24, 2014
@gordon-morehouse

Just wanted to comment after reading https://missingm.co/2014/02/fighting-dishfire-the-state-of-mobile-cross-platform-encrypted-messaging/ that XMPP and WebSockets will probably work, sort of, but will be painful on mobile devices due to frequent IP switching, sleep/wake, etc.

Unfortunately this is not (yet) a library that anyone can use, or tunnel over like ssh, but have a look at http://mosh.mit.edu/ - it's a partial ssh replacement (terminal emulator only, no tunneling or forwarding yet) which has solved ssh's problems with IP switching and sleep/wake, while managing to get around many many firewalls and NATs. Their code itself probably can't be used, but the ideas may be worth reading over.

@seroma
seroma commented May 13, 2014

@JavaJens Great Work you are doing!!! Me and quiet a couple of friends are longing desperately to get their hands on a GCM free TextSecure version. I am so glad to see that you are making it happen. This means really the world to us, since installing Google Play is absolutely no option for us. We are all really excited and following your efforts day to day. Is there anything we can do to help and speed up development other than coding (since we don't have any coding experience)? Since the day TextSecure with data came out (WhisperPush for CM) we all are dreaming of the day we can enjoy this wonderful app with all our friends. Thank you already for all you have done. You are a hero :).

👍 for Websocket support for TextSecure.

@SecUpwN
SecUpwN commented May 13, 2014

@seroma, I second that. Awesome work, @JavaJens! Keep rockin'! 😸

@Wikinaut
Contributor

Just in addition https://devcenter.heroku.com/articles/websocket-security

You should strongly prefer the secure wss:// protocol over the insecure ws:// transport. Like HTTPS, WSS (WebSockets over SSL/TLS) is encrypted, thus protecting against Man-in-the-Middle attacks. A variety of attacks against WebSockets become impossible if the transport is secured.

@JavaJens

I've updated my branch. The most recent updates feature a different package name for parallel installation, increased debug output for the Wakelocking and fixed an error for a minor issue with a Wakelock not being released.

However one of the main issues, severe battery drain, I could not find a reason for; my phone does not go into deep-sleep mode when WiFi is enabled. This has nothing to do with TextSecure though, Wakelocks are held for about 10s in 2h.
As I don't have 3G I can't test the battery consumption without Wifi.
I would suspect this being an issue with my CM11.

The last issue for the high battery drain is the low timeout frequency of @TheBlueMatt's server of about 50s; that is we have to send a ping every ~50s, which is far to often. In real life, this could be decreased to 30min or so.

One last thing I want to address, @Wikinaut: this build is purely for testing and uses the non secure WS for developing. I've updated the Readme to point this out more clearly. In the end certificate pinning will of course be used!

Please report any issues back here or as an issue in my fork.
I don't know if you can use the "Report a Bug" feature or have to use ADB....

@moxie0
Member
moxie0 commented Jun 12, 2014

The last issue for the high battery drain is the low timeout frequency of @TheBlueMatt's server of about 50s; that is we have to send a ping every ~50s, which is far to often. In real life, this could be decreased to 30min or so.

With RedPhone, we've found that mobile data networks often require you to send keepalives as little as every 15 seconds. So 50s is potentially too long.

@JavaJens

Ok...that comes as a surprise....
Current GCM timeout is apparently ~15min on Wifi, which is stated here as being to high: https://productforums.google.com/forum/#!msg/nexus/fslYqYrULto/lU2D3Qe1mugJ

That would support your findings!
How about the Battery life with RedPhone? As I said my phone doesn't deep sleep with Wifi on...

@moxie0
Member
moxie0 commented Jun 12, 2014

WiFi is unpredictable, since it depends on the NAT. So you'll have to deal with that the same way GCM does. The advantage GCM has is that Google has made agreements with mobile carriers not to timeout connections on those networks. Without those, websocket connection idle timeouts on mobile data connections will be unpredictable, and they tend to be more trigger happy (and also often just silently close without sending an RST). I think some kind of adaptive solution is probably going to be necessary.

@geileszeuch

As I don't have 3G I can't test the battery consumption without Wifi.
I would suspect this being an issue with my CM11.

I am building your branch right now. I will test on WiFi and 3G and report as soon as possible (probably in around 10 hours). I am on CM10.2 so let's hope the battery drain really is just a problem of CM11.

@JavaJens

@geileszeuch
Maybe you can check the wakelocks for TS and the deep sleep with/without TS on WiFi/3G.

And report any other issues u have :)

@geileszeuch

After 10 hours of use Android's build in battery stats show that TextSecure only had 2min keep awake and did use only 2% battery. The thing is that Android OS has a keep awake time of 9 hours and 30min, which used almost the entire battery (from 100% to 20% in 10 hours without any use of the phone). This was on 3G. Further tests with WiFi will follow.

UPDATE: Those results are not very helping, because I noticed another app was running in the background, which might have influenced this drain drastically. More to come soon.

@JavaJens

@geileszeuch
This behaviour of the OS is the same for me on WiFi...
Although keeping WiFi enabled is disabled.

@agrajaghh
Contributor

Perhaps you should have a look at K-9 Mail. I'm using it with three different IMAP accounts with push enabled. And push is working there with a imap idle connection to the server. I don't have significant battery use, so it should be possible for Websocket aswell. no?

@h-2
h-2 commented Jun 15, 2014

Just hanging out at the FSFE Germany meeting, and we really think this bug report is important. Thanks for your work in trying to make TextSecure work without GooglePlay, it is really appreciated!

@JavaJens

I will look into k9. Especially regarding the timeout and heartbeat interval.
The aosp stock client does the same, but I think without wake locks.

Maybe some of you could test the battery usage, as the wakelocks being held aren't held for long...
Furthermore having a group of friends sitting around is also great as no one else is on the server....

You can register phony numbers as well, they're confirmed automatically. I'm there as
+49123456666

@1200200200

I am registered as +1200200200 on the server.

@JavaJens Is your number +49123456666 correct? I can't send anything to you. Maybe you also just unchecked Push Messages and only check it when doing some tests/developing? I'd like to exchange messages for tests and maybe create a group for everyone willing to contribute to battery/bug testing of websockets branch.

@SecUpwN
SecUpwN commented Jun 15, 2014

@JavaJens and @1200200200, why are you posting your private phone number publicly?

@agrajaghh
Contributor

I don't think these numbers are real ;) They just registered with this fake number...

@JavaJens

Yes, they are fake. As I said the server verifies every number :)
@1200200200 How does sending of the message fail for you?
I've sent you a message which appears to be sent by the server....

@1200200200

Saying Error sending message under the message in a green box with a red triangle on the left.
Try again please. I reregistered. last time I did uncheck Push Messages.

@TheBlueMatt

Note that the server has gotten reset a few days ago since the database was reaching the free size limit.

On June 15, 2014 3:28:22 PM EDT, 1200200200 notifications@github.com wrote:

Saying Error sending message under the message in a green box with a
red triangle on the left.


Reply to this email directly or view it on GitHub:
#1000 (comment)

@JavaJens

I also reregistered...
Can send messages to you, but don't know if they're delivered...
If you still have issues, try getting an adb log or so....

@1200200200

Got your messages and sent you two messages 25 minutes ago. Did you get anything?

@1200200200

I am now 6 hours on battery and went from 100 to 90 percent while doing nothing and only having one other app running which I assume used maybe 5 out of these 10 percent. This is on 3g. Pretty decent I would say.

@geileszeuch

I am now 6 hours on battery and went from 100 to 90 percent while doing nothing and only having one other app running which I assume used maybe 5 out of these 10 percent. This is on 3g. Pretty decent I would say.

I got the same experience. 6 hours on 3G with running apps mail, chatsecure and Textsecure-Websockets-branch, and my battery went from 100 to 93 percent. This is awesome and probably even better than with GAPPS installed and TextSecure over GCM.
Doing WiFi test at the moment. Am done in about 4 hours.

@McLoo
Contributor
McLoo commented Jun 16, 2014

during your tests, do you have TS just idle or do you send/receive messages to?

@geileszeuch

For now I am just testing it running in the background without doing anything else, so no message exchange yet.

@geileszeuch

Results of my WiFi test are:
6 hours on WiFi doing nothing and the only apps running in the background being Mail, Chatsecure and TextSecure let my battery go from 100 to 95 percent. These are very good results in my eyes.

@1200200200

Results of my WiFi test are:
6 hours on WiFi doing nothing and the only apps running in the background being Mail, Chatsecure and TextSecure let my battery go from 100 to 95 percent. These are very good results in my eyes.

I got the exact same result.

@1200200200

during your tests, do you have TS just idle or do you send/receive messages to?

@McLoo The WebSockets branch only adds websockets for receiving messages so I guess the battery consumption for sending messages should be just the same as for the official TS app.

@grote
grote commented Jun 16, 2014

@JavaJens Thanks a lot for working on WebSockets support! This is really appreciated!

At the FSFE German Team meeting, we tested your app branch with various people, but messages only arrived with a long delay and some people never received messages while other people's messages never arrived. It was hard to see any pattern.

Now at home, I was testing more and noticed the following message. Maybe it is related to the problems we were seeing?

W/SystemStateListener(16014): java.lang.IllegalArgumentException: Receiver not registered: org.thoughtcrime.securesms.service.SystemStateListener$ConnectivityListener@4301d518
W/SystemStateListener(16014):   at android.app.LoadedApk.forgetReceiverDispatcher(LoadedApk.java:674)
W/SystemStateListener(16014):   at android.app.ContextImpl.unregisterReceiver(ContextImpl.java:1511)
W/SystemStateListener(16014):   at android.content.ContextWrapper.unregisterReceiver(ContextWrapper.java:489)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SystemStateListener.unregisterForConnectivityChange(SystemStateListener.java:46)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.unregisterForRadioChanges(SmsSender.java:180)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.handleSentMessage(SmsSender.java:145)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.process(SmsSender.java:64)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SendReceiveService$SendReceiveWorkItem.run(SendReceiveService.java:273)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.util.WorkerThread.run(WorkerThread.java:46)
@JavaJens

@grote

I noticed the missing/delayed messages as well. For me I didn't get any errors when sending and was always connected when other parties send delayed messages.

I'll check the error message you send.
Maybe @TheBlueMatt can check the error messages on the server?

@grote
grote commented Jun 16, 2014

@JavaJens Thanks for the quick reaction! :)

It is a good idea to ask @TheBlueMatt about the server error logs to get to the bottom of these problems. We also did not get any errors, but still the messages did not arrive or arrive only delayed.

Since, there might be many people who want to help with testing without app building skills, I created an F-Droid repository for the GCM-free branch. Just get F-Droid and add a new repository with the following URL:

http://grobox.de/fdroid/repo

The only downside is that this requires version code increases to get new versions to people.

This picture was made at the FSFE Germany meeting last weekend:
Thanks JavaJens!

@JavaJens

I just updated to the most recent upstream changes. However it seems as if the server is down/damaged, one can't connect to the WebSocket URL.
Maybe @TheBlueMatt can give some information on this?

@grote
grote commented Jun 17, 2014

Maybe the client should throw an error when the WebSocket URL can't be reached instead of silently failing?

Would you mind bumping the version code so users of the F-Droid repo can receive a new version?

@JavaJens

I don't know if a message is needed, but the least that is needed is some
kind of exponential backoff so that we don't loop and continue to fail.

I'll try to make that later today.

@1200200200

However it seems as if the server is down/damaged, one can't connect to the WebSocket URL.

This would explain the higher battery usage I had in the last 10 hours.

I don't know if a message is needed, but the least that is needed is some kind of exponential backoff so that we don't loop and continue to fail.

That is a very good idea. It should prevent battery drain in such a case.

@SecUpwN
SecUpwN commented Jun 17, 2014

@JavaJens, you ROCK! 😸 Quick question: Will these changes be incorporated onto the GooglePlay version, too? Or do I have to re-install TextSecure through F-Droid?

@JavaJens

@SecUpwN
I don't know if and most importantly when these changes might be integrated
in the main version.
Right now the branch is way to buggy and the main server doesn't support
WebSockets yet.
Also the my fork is highly insecure! It might have log messages that leak
private data and doesn't use secure Websockets.

Therefore DON'T use this branch as a replacement!
This is only for developing until the branch is more stable and the
official server supports Websockets.

@grote
grote commented Jun 17, 2014

To clarify what the F-Droid version is: It uses a separate repository from the official F-Droid one that can be installed as an add-on and it just contains the apk build from the feature branch source. So it is only the test version and can therefore also be installed alongside the official TextSecure and is not meant as a replacement.

This is purely to distribute the test version to a larger group of people for testing purposes. In order to get the new version show up in F-Droid, the version code would need to be increased by one though.

@kirschner

Thanks @JavaJens for your work!

I am using the build by @grote . I can receive messages from @grote and one other user but it seems they do not receive my messages. Furhtermore there is a longer delay in messages.

Is there anything specific I can test to help you solving the problems?

@h-2
h-2 commented Jun 17, 2014

I also exerience a delay with textsecure messages, to me it looks like the clients pulls from the srver every x minutes. Is this the case? If yes, could the behaviour be changed to where it holds a continuous connection for as along as the screen is turned on (i.e. the device is being used) and switch to polling every x minutes once the phone is in sleep?
Ideally this could be configured in the options!

I can also confirm that automatic key exchange and encrypted SMS work with regular TextSecure users (don't now how exactly, but it does).

@JavaJens

@h-2 The way the websocket connection works is the following: When TextSecure is opened or Internet connection is established, a websocket connection to the server hosted by @TheBlueMatt is opened. This connection is always kept open!
Every ~50s a heartbeat is sent to the server and replied by a Pong.
If a message arrives it should therefore be immediately delivered.

There are three scenarios in which a delivery could be delayed:

  1. No connection to the Internet->Delivery on connection
  2. Dropped connection -> Connection should be re-established in at most 50s
  3. Error on the server side that denies connections -> The phone tries to reconnect every 50s, this is bad and will be replaced by an exponential back-off soon.

Using this method should be effective as the phone can sleep during the Ping interval and will be woken on incoming messages or by the timer.
However the issues we are currently experiencing are different, at least from my point of view, as I seem to have a working( i.e. ping/pong messages arrive) connection, but the messages are delayed anyway. As the same issues are experiences in the Chrome extension, I could imagine the server being at fault here, but this is just a guess.

As far as debugging goes: The best setup would be if you either use several devices or have a second communication channel to the other party so you know when messages are sent.
When sending or waiting for a message always grab a debug log using logcat. This way any errors on the phones can be seen.

I have seen several errors of MAC mismatches, maybe you experience the same?
Also are PONG messages received in a time where other messages should also be received?
Do the PING messages work for you?

@grote The error you described seems to me to be of a different nature. Without further research I think the "receiver" in this case is more of an android Intent/Service receiver. This could be due to the different package name I used. I will look further into this if I find the time.

Lastly, @TheBlueMatt do you experience similar issues while developing the Chrome extension? Maybe you can chip in with your knowledge.

@grote
grote commented Jun 17, 2014

I now updated the version in the test repository for F-Droid to the latest commit:

http://grobox.de/fdroid/repo

@JavaJens I managed to fake a higher version code, so you don't need to bother with it.

Thanks for all your work! I tried sending you and other people messages, but they seem to never arrive at the moment. Thanks also for looking into this "Receiver not registered" issue. Could this maybe be related to the fact that I did not make TextSecure my default SMS app? Otherwise, the changed package id might indeed be a clue, but then you should see this as well.

@TheBlueMatt

re: getting logs, I have no problem providing any logs you want, but it may be best to just coordinate a time where I can provide logs immediately when someone is testing...ping BlueMatt on freenode or email.

@eighthave
Contributor

Nice to see progress on this, and to see an FDroid developer repo in action. For a little extra security, it is possible to include the signing key fingerprint in the URL:
http://grobox.de/fdroid/repo?fingerprint=28e14fb3b280bce8ff1e0f8e82726ff46923662cecff2a0689108ce19e8b347c

And of course QR Codes are sometimes handy:
grobox de-fdroid-repo

@moxie0
Member
moxie0 commented Jun 21, 2014

@JavaJens @grote I would prefer it if you'd stop passing around APKs via F-Droid. I think the only people who should be using this build are people who are capable of building themselves.

Another thing to consider is how you plan on eventually integrating this work. A PR with an enormous diff will probably take a very long time to get merged, so if possible I would recommend a more incremental approach.

@JavaJens

@moxie0 I know how you feel about that, thus I haven't promoted that. I totally agree with you especially regarding the many security issues with the current status.

About integrating. I think it will not be a big diff, as I keep integrating the upstream changes. Plus there aren't too many changes anyway.

However there are still several open issues.

Maybe you can give me a hint on how you'd like to continue ?

@grote
grote commented Jun 21, 2014

@moxie0 Please note that the apk is not distributed to all F-Droid users, but only to the people that find this discussion and add the test repo manually. Then, there's a big fat warning about the test nature and insecurity of this build in the app description.

I personally think that this is good enough and widens the group of people I can test with. Considering the open issues we face, this is appreciated.

There's an open question pending on the mailing list which you might know an answer to.

@moxie0
Member
moxie0 commented Jun 21, 2014

@JavaJens If the PR is going to be small don't sweat it. I just wanted to give you a heads up about doing some incremental changes if you expect it to be large.

@grote Unfortunately people have started promoting it elsewhere, which I think will cause problems.

I think that if you want to do testing, it might be more effective to either run 1% tests across the installed TextSecure base, or to distribute APKs which test specific functionality but aren't full TextSecure builds. For instance, if you want to test the effect of websockets on battery life, I think it would be more effective to put together a small APK which only encapsulates websocket functionality, measures connectivity, measures the effect on battery, and packages up all the device info, network type, etc, into results which are reported back to you. I think having people manually try to do the same with a full TextSecure build will result in more unknowns and more anecdotal results that are more difficult to quantify.

@tinloaf tinloaf referenced this issue in glowing-bear/glowing-bear Jun 30, 2014
Closed

[future] GCM transport for Android #340

@xylo
xylo commented Jul 2, 2014

@moxie0
Providing a test repository with pre-compiled binaries is a very common solution to find testers (see for instance debian, synology, mozilla, ...). Moreover it is the best solution you can do if you really want to find testers for this feature, since it addresses exactly the people that do not have the Google Market installed and would be therefor unable to install TextSecure at all. If you force all testers to compile the app themselves you'll get a lot of testers with outdated versions since no tester is going to rebuild the app every day.

Furthermore, the people that use the F-Droid repository posted by eighthave know exactly what they do and can definitely distinguish between official versions of TextSecure and unofficial testing versions hidden behind a personal F-Droid repository that one needs to manually add before being able to install the test version.

Regarding your suggestion of running 1% tests across the installed TextSecure base I'm very sceptical. First, you need to find enough volunteers for testing unless you plan to secretly choose some of your stable users as guinea pigs (I hope you are not doing this). Second, those users are more likely not to be able to distinguish between a testing and a stable version than those that came to this thread, read about the problem and installed the app from a special test repository.

@georgkrause

Well, i want to test this app, but i dont know how to do it. I installed the app, but there is the warning i shouldn't enter private data, and i agree with that. But now i dont have any contact for testing, which is the recommended way?

@JavaJens
JavaJens commented Jul 9, 2014

Right now testing the app has pretty much halted. The websocket
implementation seems to work quite well as indicated by other tests, but
there are issues with properly connecting to the server that need to be
solved before picking user tests up again.

I am working with @grote on debugging some of these issues, but my time is
very limited right now.

Nevertheless the way you could test is the following:
Install TS-Websocket on two devices and add each other. Then you could
message between these devices.
The Server is a different one than the regular TS server therefore none of
your friends are there.

You also should be able to communicate with the chrome extension, but I
also had severe issues with that.

Good luck :)

2014-07-09 18:35 GMT+02:00 Georg Krause notifications@github.com:

Well, i want to test this app, but i dont know how to do it. I installed
the app, but there is the warning i shouldn't enter private data, and i
agree with that. But now i dont have any contact for testing, which is the
recommended way?


Reply to this email directly or view it on GitHub
#1000 (comment)
.

@geileszeuch

The Telegram app for Android also introduced a GCM free Notification Service. I didn't check whether it's based on websockets, but I can tell that it works perfectly and uses almost no battery. Maybe this can help somehow. Here is the source code: https://github.com/DrKLO/Telegram

@JavaJens
JavaJens commented Aug 3, 2014

We now have support for the Staging Server! There you have to use your real phone number and get a verification code.

There are still some things left to do, but it seems usable right now.

@moxie0 I made the following change to the PushServiceSocket https://github.com/JavaJens/TextSecure/blob/0ae8a8c0e8b0a65733eda7619fb34999bd8d0989/library/src/org/whispersystems/textsecure/push/PushServiceSocket.java#L429

Could you think of any side-effects from changing that from private to public static?
Also how does one delete/unregister a non GCM client?

@moxie0
Member
moxie0 commented Aug 3, 2014

Could you think of any side-effects from changing that from private to public static?

That wouldn't make it pass code review. We're sticklers for correctness.

@JavaJens
JavaJens commented Aug 3, 2014

So what do you suggest? Create an interface that handles the certificate pinning on a system level?

Or should we move that to one of the utility classes?

@ddorian1
ddorian1 commented Aug 7, 2014

Thanks for your work!
Are the google services still needed for registration? I have build your branch and installed it on my phone without google apps. When I tried to registry for push, I received the verification code and the phone generates the keys, but afterwards it complains to be not able to connect to the server.

@JavaJens
JavaJens commented Aug 7, 2014

Mhh, thats strange. @grote is using the branch also without Gapps. There
might still be some issues, but it should work in principle.
Maybe you could grab a debug log of the exact error message?

2014-08-07 20:27 GMT+02:00 Johannes Schwab notifications@github.com:

Thanks for your work!
Are the google services still needed for registration? I have build your
branch and installed it on my phone without google apps. When I tried to
registry for push, I received the verification code and the phone generates
the keys, but afterwards it complains to be not able to connect to the
server.


Reply to this email directly or view it on GitHub
#1000 (comment)
.

@georgkrause

i have the same problem and logged it. I don't have a PC at the Moment, but i can send the log file per mail or post it at the weekend.

@JavaJens
JavaJens commented Aug 8, 2014

Thanks for the Log.
Looks like the same issue that @grote had.
Could you try to delete all your app data (via Settings->App) and re-register?
Just if you don't mind deleting the data of course ;)

@ddorian1
ddorian1 commented Aug 8, 2014

Deleting the app data didn't help, but this fix works for me: ddorian1/TextSecure@17d0280

@ddorian1
ddorian1 commented Aug 8, 2014

It seems that isUserRecoverableError returns true if it is possible to install gcm on your device (e.g. on cyanogenmod). So I would suggest to change the code to https://github.com/ddorian1/TextSecure/blob/d1147fce336e6310980bdb4277a61d78a9200f61/src/org/thoughtcrime/securesms/RegistrationActivity.java#L182

@JavaJens
JavaJens commented Aug 8, 2014

@ddorian1 I thought the same about the user recoverable deal.

I'll look into your changes, but on my mobile they look like they are not needed, as usually the error should not occur if we dont force to disable gcm.

I can't reproduce the error after deleting my prefs. Maybe you could check the error log again after deleting the preferences?

I know that voice registration doesn't work right now, but the rest should.

Thanks for your help!

@haffenloher
Contributor

When I tried to registry for push, I received the verification code and the phone generates the keys, but afterwards it complains to be not able to connect to the server.

FYI, I can reproduce this problem using a fresh build of @JavaJens' feature-1000 branch. As I did a factory reset of my device some days ago, this was the first ever TextSecure install on this Android system, so there shouldn't be any data from previous installations.

@JavaJens
JavaJens commented Aug 9, 2014

That's too bad. I will do some debugging next week to see if I can find the root cause here.

I think that disabling the check altogether is bad, as there might be occasions where this would be a genuine error.

@ddorian1 @brumsoel Do you have Gapps installed? Maybe the error occurs because we force it to not use GCM although it could?

Sorry, I am a little tied down right now, but will look into it asap.

@ddorian1
ddorian1 commented Aug 9, 2014

No, I don't have gapps, but I use cyanogenmod, so gapps can be installed. As I see it, that makes the error 'user recoverable' and I will not be able to use sockets with the check.
Another solution may be to ask the user wether to use gcm or sockets when its user recoverable, but I think that's not realy user friendly.

@ddorian1
ddorian1 commented Aug 9, 2014

One last thought:
From the documentation it seems to me, that isUserRecoverableError only returns true, if either gcm is not installed or disabled in the settings.
If it's disabled, the user has decided to do so. And if it's not installed, then it would be more convenient to silently fall back to sockets then to prompt the user to install additional software. So in my oppinion the check is not necessary.

@haffenloher
Contributor

Do you have Gapps installed?

Nope, no Gapps installed here either.

@haffenloher
Contributor

Here are some results of my tests (using the registration fix by @ddorian1):

  • Communicating with someone who has the official play store client installed only works through (encrypted or plain text) SMS, push messages are not offered as a choice on both devices
  • Communication between two websocket users over the data channel works, but with a strange side effect: The sender of a message immediately receives an empty, unencrypted reply. The other client does not display the empty message it (apparently) just sent. See this screenshot.

Later on, my friend reinstalled his phone, now with play services. I kept using the websocket client. Now, my client's push messages aren't received on my friend's client, while he's only offered to send me (encrypted or plain text) SMS.

Are these issues already known?

@JavaJens

Hey, thanks for testing.
To address your problems:

  • The official play store client uses a different server (unless you
    changed the server in your built). That is why you don't "see" each other.
  • These empty messages are, I believe, the receipt messages. I figured that
    they are shown due to an outdated client? I will try to merge the master in
    soon and they should disappear. Otherwise that is an issue.

So when testing, make sure that you both are using the same server, please
report back then what issues you still have.
Thanks a lot!

2014-08-13 17:34 GMT+02:00 brumsoel notifications@github.com:

Here are some results of my tests (using the registration fix by @ddorian1
https://github.com/ddorian1):

  • Communicating with someone who has the official play store client
    installed only works through (encrypted or plain text) SMS, push messages
    are not offered as a choice on both devices
  • Communication between two websocket users over the data channel
    works, but with a strange side effect: The sender of a message immediately
    receives an empty, unencrypted reply. The other client does not display the
    empty message it (apparently) just sent. See this screenshot
    https://cloud.githubusercontent.com/assets/264472/3899484/55da6482-227c-11e4-9134-b4e16f1a7603.JPG
    .

Later on, my friend reinstalled his phone, now with play services. I kept
using the websocket client. Now, my client's push messages aren't received
on my friend's client, while he's only offered to send me (encrypted or
plain text) SMS.

Are these issues already known?


Reply to this email directly or view it on GitHub
#1000 (comment)
.

@haffenloher
Contributor

The official play store client uses a different server (unless you changed the server in your built). That is why you don't "see" each other.

Woops, I wasn't aware of that, sorry. Thanks for pointing it out. I will try it using the official server then.

@JavaJens

I just merged the latest upstream master.
However the empty messages are still there and a problem of the WebSocket message handling.
Maybe @moxie0 can shed some light on why encrypted "empty" messages are sent?

@haffenloher
Contributor

Great, thank you! I just updated my build and will report back if I encounter any issues.

... shed some light on why encrypted "empty" messages are sent?

Just FYI, the empty auto-responses don't seem to be encrypted (no lock icon is shown for them while every regular message has one).

@geileszeuch

@JavaJens The app currently has 5 services running on my phone with your branch.
SendReceive-, KeyCaching-, Push-, DirectoryRefresh- and GcmRegistrationService. Are all of those especially GcmRegistrationService necessary for a Google free phone? It would be great if the GCM library was not used at all on a phone without Google stuff and flavours to even build the app without that library would be terrific, but let's see how moxie feels about this.

@geileszeuch

I just remembered that @moxie worked on an alternative distribution platform for the apk to make it available to people without Play Store and mentioned it is going to be released as soon as the app becomes independent from GCM. With this in mind he will hopefully accept a GCM-free flavor.

@JavaJens

@geileszeuch Do you know how many are running on your device with the master branch? I could imagine that the GcmRegistrationService is not running there, as GCM was successfully registered, but all others should not be influenced by my branch (except the PushService of course...)

The problem here is that one cannot distinguish between a user that has no Gapps and a user that has an outdated version of Gapps and wants to use GCM as soon as he updates, therefore the checks of the GcmRegistrationService are used.

This would of course be solved when using a different build flavor, but I think that requires some more refactoring as the Play services are used at several places I believe. Also I don't know how far the different flavors would diverge, regarding maintainability etc.

@geileszeuch

@JavaJens I don't have Google on my phone so I can't check how many services are running on the master branch. Sorry.

@JavaJens

@geileszeuch was just guesstimating, so NP.

The question is: does it work and can we create acceptable build flavours....
I guess we have to create two build variants right?

@JavaJens

I've created a PR #1960
Hopefully the devs will take a look into this and provide feedback, as I am sure it misses some style and coding guidelines, but it should be a good starting point.

@moxie0 I wasn't sure as to what I should set the URL for the service in the Release.java; if they should point against the regular server, just let me know.

@openpaul

@JavaJens Is your apk in your F-Droid respository up to date?
I just registered myself with +49123456789 maybe someone will add me, to test the websockets.
I have no playstore or google apps installed.

Sadly I am not able to send to: +1200200200 or +49123456666

@JavaJens

@openpaul I don't maintain the F-Droid repository nor have I ever used it, so I don't know.
Regarding the numbers you used: my current branch is using the TextSecure staging server, which requires real numbers and SMS verification.

@TheLastProject

@openpaul The F-Droid repo, maintained by @grote, hasn't been updated in a while.

@pejakm
Contributor
pejakm commented Oct 17, 2014
@openpaul

Thanks for the fast reply
@pejakm I tested the most recent version and it seems to work very well.

@grote
grote commented Oct 19, 2014

I updated the repo and am now using exactly what is in the @JavaJens repo. So that means that it uses the staging server and needs reregistering. Also, the app name changed to the official one. So official and testing version can not be installed side by side anymore and real phone numbers are required.

@schiessle

@grote does it mean that I can now use your build to communicate with contacts using the google play version of TextSecure? Of course only as a experimental/testing version, but this should work now with the latest changes, right?

@grote
grote commented Oct 24, 2014

@schiesbn That depends on whether the staging server federates with the production server. Since I do not run these servers, I can not say for sure whether that is and will be the case.

@georgkrause

@grote Which staging server is used in the fdroid-version in your repo?

@grote
grote commented Oct 27, 2014

@georgkrause This is textsecure-service-staging.whispersystems.org

@rehans
rehans commented Oct 27, 2014

Since sunday afternoon (26.10.2014) the "emtpy messages" are back. For each message I send to a contact I get an empty message back. Anyone else having this problem? Did sth change on the server?

@JavaJens

@rehans Did you update your TS installation? I didn't commit any changes since 23.10.2014.
However the master branch was updated with the delivery receipts recently, maybe the server was updated as well.

Nevertheless have I merged the latest upstream changes in my master branch, please consider that it was a forced push due to the ongoing pull request.
If you could report back, that would be great.

@moxie0
Member
moxie0 commented Oct 27, 2014

The websocket protocol is going to change radically in the near future, before this is merged. So this is going to continue to break on a regular basis.

Also this PR needs to be more concise, and shouldn't include code from other packages in this repository.

@h-2
h-2 commented Oct 27, 2014

Has someone been monitoring the influence on battery-usage in a controlled environment? I am seeing a reduction in uncharged uptime of more than 50%. The effect is a lot stronger than that of Xabber, which also works without Google Push...

@pejakm
Contributor
pejakm commented Oct 27, 2014

@h-2 I didn't notice such thing. Is there any way we can measure this more precisely?

@h-2
h-2 commented Oct 27, 2014

Well, I don't know, my phone stays well above 50% charge during 24hours (even with Xabber on permanently), but as soon as I activate push in the TS settings I barely get through the day.
Maybe you can just deactivate push in the options and have the phone idle around somewhere for a day and then do the same again with push activated. that is not really a controlled environment, I know, but if you don't see a significant differences than it's just me who is experiencing a bug.

@SecUpwN SecUpwN referenced this issue in tinfoilhat/tinfoil-sms Oct 28, 2014
Closed

Switching from TextSecure to Tinfoil-SMS #70

@freerider123

@grote
you said, you updated the repo. for f-droid? for me the latest accessible version is 2.1.5b1 from 28.8. am i wrong? or just is it just a missunderstanding?

@pejakm
Contributor
pejakm commented Oct 30, 2014

@freerider123 You can try apk I compiled: link removed (sorry folks).

@freerider123

@pejakm thanks for the brandnew compile.

@moxie0
Member
moxie0 commented Oct 30, 2014

@pejakm Please don't distribute APKs.

@pejakm
Contributor
pejakm commented Oct 30, 2014

@moxie0 I understand your concern, and rest assured that my intention is not to distribute anything, I am simply sharing apk with websockets support with my friend who can't compile it herself (so I posted the link for people who want to test it). I'll remove the link if you want.

@agrajaghh
Contributor

@pejakm look at #127

@JavaJens

@rehans Do you still see the problem with the empty messages? I haven't found the time yet to check it myself and honestly lost some interest, if everything is going to break in the near future anyway.

@moxie0 If you can communicate a clear roadmap on what and how the websocket protocol is going to change and comment on the conflicting problems in the PR in a detailed manner, I'd be happy to further contribute.

@moxie0
Member
moxie0 commented Oct 30, 2014

@pejakm If someone can't compile this themselves, they shouldn't be running it at this point. It's not production ready.

@rehans
rehans commented Oct 30, 2014

@JavaJens Just did an update and it seems to be ok. I was wondering because the empty messages appeared again without me changing anything. I keep on testing but until now it is very reliable. It is a bit sad to hear that It will break again in the future.

@ganomi
ganomi commented Nov 3, 2014

Ignore!! #1000 (comment)

Hi, just want to report my observations:
Running the websocket version in a stable 3g environment. Functionality is all good, but over night the app will cause 50% battery drainage in about 7 hours. Usually this would only cost ~10-15%.

BetterBatteryStats will show me a huge increase in bam_dmux_wakelocks, which are network related. This claims for about 50 minutes cpu time if i do not interpret the numbers wrong.

WakeLockDetector also shows me about 2300 wake ups caused by TextSecure. In comparison WhatsApp causes only about 1000 wake ups during that time. But with only WhatsApp and k9mail-push running i do not see such battery drainage.

Thanks for your efforts!

@JavaJens
JavaJens commented Nov 3, 2014

Right now we are using a very short fixed ping interval, I guess dynamically adjusting the ping timeout to a more reasonable time would decrease the amount of wakelocks and hence increase battery time, nevertheless can't I wrap my head around the number of wakeups as there should be a ping every 30s, that is the device should wake up twice a minute (assuming the ping takes no time) resulting in a total of 840 wakeups for sending pings.
Now on each Pong message, the device should/could wake up again, resulting in twice the amount of wakeups but still less than what you have reported.

The question is why are there so many more wakeups and if the ping interval can be reduced.
Google Play Services uses something like 15 minutes, but that causes some anger with delayed messages (see e.g #970), but I think a couple of minutes of delay are tolerable.

@pejakm
Contributor
pejakm commented Nov 3, 2014

I don't have issues with battery drain, although BetterBatteryStats did report large amount of wakeups.

@ganomi
ganomi commented Nov 3, 2014

Ignore!! #1000 (comment)

i checked the numbers again for a period of 4.5 hours this morning and it gives about 540 wakelocks, which is 2 wakelocks per minute :) this correlates with your numbers.
Still there is 30 minutes bam_demux_wakelock and 30 minutes PowerManagerService.WakeLocks. Each cause 10 % battery loss of a total of 30% in the last 4.5 hours.

Maybe its a problem with my device. Im Running a CM 11 nightly. No stable available for my Xperia M.

@ganomi
ganomi commented Nov 3, 2014

Ok, this is embarrasing. On further investigation this might all be caused by the Spotify app or CyanogenMod (http://community.spotify.com/t5/Help-Android/Spotify-for-Android-causing-massive-battery-drain-and-heating-of/td-p/476714/page/9) which has problems with sd storage. i installed the sd card also a week ago and would not have thought this would have any impact. Spotify was running without battery drain long time before i installed textsecure. but there might be an sdcard bug related to spotify with cyanogenmod.

@h-2
h-2 commented Nov 3, 2014

@ganomi Like I said previously, I also experience significant battery drainage and I don't use spotify, so I don't think its related... although I do use CM.

@freerider123

my experience while testing are terrible: most messages didn't reach, some do hours later. all without error messages. my device is on cm7 without gapps. the other one is a standard android installation with google & co.

@JavaJens
JavaJens commented Nov 4, 2014

@freerider123 Would be good if you could provide a debug log. Maybe you could also add additional debug lines in the PushService, as I removed them for the PR.
Helpful would be e.g: error log, successful connects, unsuccessful connection attempts, ping+pong and the like

Also make sure that both devices run on the same server, i.e. the staging server.

@freerider123

debug log: http://hastebin.com/raw/rokepoqefe
error log: http://hastebin.com/aturehixiv.avrasm

the app version on the devices are exactly the same, so i think the server is too.

more information i can't provide at the moment, cause i don't know how. its my first log report, sorry. but if more is needed, i'll dig further.

@ganomi
ganomi commented Nov 9, 2014

@h-2 after some more investigation textsecure really was the reason for battery usage.

@JavaJens i compiled your master branch and observed its behavior on my CM11 device. every minute i received the following:

11-09 17:12:19.264    7014-7014/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Doing a ping
11-09 17:12:19.905   7014-10101/org.thoughtcrime.securesms W/WebSocketClient﹕ WebSocket EOF!
    java.io.EOFException
            at java.io.DataInputStream.readByte(DataInputStream.java:77)
            at com.codebutler.android_websockets.HybiParser.start(HybiParser.java:123)
            at com.codebutler.android_websockets.WebSocketClient$1.run(WebSocketClient.java:156)
            at java.lang.Thread.run(Thread.java:811)

after that it would do a full reconnect every minute. i guess this may be the battery draining i have observerd. This surely costs a lot of wake up time..

i recognized that the code would never ever go into

am.setInexactRepeating(AlarmManager.RTC_WAKEUP, System.currentTimeMillis(), seconds * 1000, operation);
PendingIntent operation = PendingIntent.getService(this, 0, PushService.pingIntent(this), PendingIntent.FLAG_UPDATE_CURRENT);

inside the PushService. The 30 second alarm for the ping had no effect.

I commented out

PendingIntent operation = PendingIntent.getService(this, 0, PushService.pingIntent(this), PendingIntent.FLAG_NO_CREATE);
if (operation == null) {
}

and would let it run into FLAG_UPDATE_CURRENT all the time.

In that case i get the effect of keep alives which on average keeps the connection alive for about 5 minutes up to even 25 minutes without a full reconnect. This might not be a proper solution but it may give a direction for a fix.
i still have not checked yet wether this improves battery, but i hope it will.

@ganomi
ganomi commented Nov 10, 2014

Unfortunately this did not have as much effect on the battery as i had wished. might still have saved a little cpu time.
what is still bugging me now are the wakeups of "WebSocket.PushService.Client" with 4 wakeups per minute. Although this does not account for much cpu time. Is WebSocketClient running in an endless loop? where is the 15 seconds waiting time coming from?

@ganomi
ganomi commented Nov 10, 2014

@agrajaghh that is the code i know and that i have edited during my debugging. now i am talking about a 15 second wakelock caused by "WebSocket.PushService.Client", whatever that is. that name is from the Wakelock Detector App. But it is not the class "WebSocket.PushService". That one has its own identifier.

@JavaJens

@ganomi Thanks for your investigation. The ".Client" Wakelock is the following: https://github.com/WhisperSystems/TextSecure/pull/1960/files#diff-80a20bd0af0a0e0df9465062f1a74c1fR137
That is the wakelock the actual websocket connection holds to make sure that all data is send etc.
The 15s probably come from it trying to connect or similar.

I've implemented a back-off feature that limits the number of retries/minute to not cause a battery drain like you experience, this works by keeping a count of failed attempts and calculate a delay time from that, however in your situation the connection is successfully established and then it tries to ping the server, sadly this forces a disconnect (EOF of the stream). This successful connection establishment then resets the counter and the retry is done immediately, hence the frequent retries.

The question is why do you receive so many EOF errors. I had a few during testing, but couldn't determine their cause or debug it further, could you maybe look if you get any errors from the websocket client itself?

@ganomi
ganomi commented Nov 10, 2014

@JavaJens I do not think that EOF comes from pinging. The EOF comes after every 60 seconds because there was no Keep Alive ping. The implementation would never send a keep alive because it would never run into the initialization of the 30 * 1000 alarm. after i disabled the if clause and let it run into https://github.com/WhisperSystems/TextSecure/pull/1960/files#diff-80a20bd0af0a0e0df9465062f1a74c1fR132 every time i do not get EOF anymore because there would be a ping every 30 seconds.
'''
Ah, now i see how you got to your conclusion. The log i posted might have implied your interpretation. I guess the exception follows directy to the ping because when the ping is tried, there is no connection anymore and it will recognize this and throw the exception and then reconnect. i think i will post a more complete log tonight which does not need as much guessing.
'''

@ganomi
ganomi commented Nov 11, 2014

@JavaJens Ok, great, i feel like a retard right now. 😀 i updated my cyanogenmod yesterday afternoon and then i could not reconstruct the behavior afterwards. so it seems all is good. maybe the restard did something or there was a problem with cyanogenmod. But the battery drainage still remains!
I was not able to find a reason for that yet.

I have collected battery statistics and put them into a pdf for comparison (There are also some comments by me):
https://github.com/ganomi/TextSecure/blob/master/BatteryTextSecureWebSocket.pdf?raw=true

In general TextSecure Websocket doubles the battery usage from 2,5 to 5 % per hour. It seems like the websocket library needs a huge amount of wakeup time for network communication without a reason. Maybe someone else can interpret these numbers better.

The events marked in the pdf seem to come from the process "system_server".

@JavaJens

@ganomi Thanks for your input! I will try to take a deeper look at your PDF the next days.

As for the numbers: The 800x for the PushService are spot on for a 6:40h runtime, but you are right that the amount of wakelocks for the client seem high.
I think the reason is that the wakelock is used in both the WebsocketClient and HybiParser and is acquired each time on send and receive.
However as you also noted the actual time spend on the wakelocks is quite low and if I understand the data correct, most time is spend in the bam_dmux_wakelock which is the one for the radio.

This would lead to the assumption that most time is spent in actually sending data, if you could check how long it takes on your network to send/receive on ping/pong maybe we can extrapolate the data?

The problem right now is, I cannot really say if the numbers are crazy/slow/whatever, what I do can say though is that I believe the ping timeout to be way too low. @moxie said somewhere that they experienced severe issues when developing RedPhone and had to use a timeout as low as 30s which is the reason I chose that here as well, but I think TextSecure allows for larger delays than RedPhone (also I don't recall the correct context maybe I have a horrible mixup, I was under the impression RedPhone also uses Google Play?).

To sum up I think we should decrease the ping timeout (Google Play has something like 15min) and preferably adjust it automatically (see e.g. http://technet.microsoft.com/en-us/library/ff459598.aspx) and get more data and afterwards evaluate if another WebSocket library should be used or how it can be improved.

@ganomi
ganomi commented Nov 11, 2014

@JavaJens
I have measured the ping pong times.
A ping/pong either takes 200-300 ms or ~1200 ms.

In 12 minutes i counted 12 short pings and 12 long pings.

Here is a full log: https://gist.github.com/ganomi/2352d9f4d6f34dd429a5

The case if long or short ping seems random.

PS: I put in some more logs around the Wakelocks and they all seem to behave ok on first view:
https://gist.github.com/ganomi/a4c82444b87f5311683b

There also does not seem to be any other missbehavior connected to the Wakelocks. They are only held as long as needed:

11-11 23:09:29.156  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:29.656  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:29.656  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:30.167  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:30.167  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:30.387  19347-19347/org.thoughtcrime.securesms W/WebSocket.PushService﹕ PushService onStartCommand
11-11 23:09:30.387  19347-19347/org.thoughtcrime.securesms W/WebSocket.PushService﹕ PushService Awake
11-11 23:09:30.397  19347-19347/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Ping from PushService
11-11 23:09:30.397  19347-19347/org.thoughtcrime.securesms W/WebSocket.PushService﹕ PushService Release
11-11 23:09:30.407  19347-19374/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ Client sendFrame Awake
11-11 23:09:30.407  19347-19374/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ 1415743770417 sendFrame
11-11 23:09:30.427  19347-19374/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ Client sendFrame Release
11-11 23:09:30.667  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:30.667  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:31.168  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:31.168  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:31.398  19347-19375/org.thoughtcrime.securesms W/HybiParser﹕ Client parseOpcode Awake
11-11 23:09:31.398  19347-19375/org.thoughtcrime.securesms W/HybiParser﹕ Client start Release
11-11 23:09:31.408  19347-19375/org.thoughtcrime.securesms W/WebSocket.PushService﹕ 1415743771415 onPong
11-11 23:09:31.668  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:31.668  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:32.179  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:32.179  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:32.679  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:32.679  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held
11-11 23:09:33.180  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ ClientWakelock is NOT held
11-11 23:09:33.190  19347-19376/org.thoughtcrime.securesms W/WebSocketClientxxxxxx﹕ PushServiceWakelock is NOT held

Maybe this really is a problem with cyanogenmod. i might test with a Google Phone the next days.

@ganomi
ganomi commented Nov 14, 2014

@JavaJens i logged the hell out of the code the last nights and there is absolutely nothing wrong with it 👍
As you guessed it was enough to decrease the ping intervall to save significant battery time. i set it to exactly 55 seconds (60 seconds t-mobile timeout). This resulted in an average battery loss of 3,4 % per hour compared to 4,9%/h at 30 second interval compared to 2,5% without TextSecure. For me this is absolutely acceptable. Of course there is some space for improvement (always reset the ping intervall after! any last data has been received. (pong or messages.)

Interesting reads about radio power consumption of mobile devices:
http://xmpp.org/extensions/xep-0286.html#sect-idp1512240
http://stackoverflow.com/questions/19141050/understanding-the-android-radio-state-machine-for-better-battery-life

@agrajaghh
Contributor

I guess the best way would be to adjust the intervall automatically. Something like this:

  • start with 30 seconds
  • if the ping/pong is successfull increase the intervall by 5s
  • if the ping/pong is not succesfull use the last known working value
  • start over on every network change, but I don't know if we can check this. But especially on Wifi/3G change...

But perhaps thats better to implement after the initial PR to keep it simple for now...

@rehans
rehans commented Nov 26, 2014

Since yesterday evening I do not receive messages any longer (although I can send messages which also get received by other contacts).

The logcat has a suspicious output:

11-26 19:03:17.781    1739-2696/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Disconnected! Code: 0 Reason: EOF
11-26 19:03:17.785    1739-1739/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Connect Client
11-26 19:03:17.799    1739-1739/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Connect Client
11-26 19:03:18.202    1739-2707/org.thoughtcrime.securesms E/NativeCrypto﹕ ssl=0x5f469720 cert_verify_callback x509_store_ctx=0x5ea0da40 arg=0x0
11-26 19:03:18.203    1739-2707/org.thoughtcrime.securesms E/NativeCrypto﹕ ssl=0x5f469720 cert_verify_callback calling verifyCertificateChain authMethod=ECDHE_RSA
11-26 19:03:18.580    1739-2707/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Connected to websocket
11-26 19:04:47.642    1739-2707/org.thoughtcrime.securesms W/WebSocket.PushService﹕ Disconnected! Code: 755 Reason: Server error
11-26 19:04:47.652    1739-2707/org.thoughtcrime.securesms W/WebSocketClient﹕ WebSocket EOF!
    java.io.EOFException
            at java.io.DataInputStream.readByte(DataInputStream.java:98)
            at com.codebutler.android_websockets.HybiParser.start(HybiParser.java:123)
            at com.codebutler.android_websockets.WebSocketClient$1.run(WebSocketClient.java:156)
            at java.lang.Thread.run(Thread.java:838)

Any ideas? Moxie once mentioned that it will break. Probably the time has come now.

@ganomi
ganomi commented Nov 26, 2014

I confirm not to be able to receive messages.

@JavaJens

I guess this is that time, however the code in the server repository hasn't changed. So I don't know what changed exactly and how to solve the issues.

@pejakm
Contributor
pejakm commented Dec 1, 2014

Could moxie help? :)

@JavaJens
JavaJens commented Dec 3, 2014

The Server repo was updated, I will try to get it working again asap.

@pejakm
Contributor
pejakm commented Dec 3, 2014

Looking forward to test it. :)

@moxie0
Member
moxie0 commented Dec 3, 2014

FYI this is going to change again. Also please don't run this code against production, only staging. Thanks.

@JavaJens
JavaJens commented Dec 3, 2014

@moxie0 Is the general structure going to change again or just specific details about the WebSocketResources?
Also did you make changes to the Ping timeout? The iOS version uses 15 secs with that I get a somewhat stable connection, but the 30+ I previously used seems to fail.

@JavaJens
JavaJens commented Dec 4, 2014

I have made some slight modifications to add rudimentary support for Websocket Resources.
Message receiving works again, I haven't tested acknowledgments because I don't have another device and have to test against the browser.

However the code needs heavy refactoring, therefore I have closed the PR for now and ask for all discussion to be moved to my fork of the repo: https://github.com/JavaJens/TextSecure/issues

If the needed changes are made, I will again open up a PR.

@agrajaghh @rehans I have removed your commits for now as it was tedious to keep solving the same merge conflicts. Once we have solved the basic issues, I'd be more than happy to accept them as a new PR.

@agrajaghh
Contributor

@JavaJens no problem, just give me a message

@Selaron
Selaron commented Dec 4, 2014

After pulling and buildingyour sources and hitting Register button, TextSecure "won't run without Google play services" which is missing on my phone :-(

@rehans
rehans commented Dec 4, 2014

@JavaJens Sure, no prob.

@JavaJens
JavaJens commented Dec 4, 2014

@hsunke Could you check if this is the same as described here: JavaJens#5

Also does it not work or is it only saying it is not working?

@Selaron
Selaron commented Dec 4, 2014

@JavaJens (changed my github Name)
No its different: As soon I activate push message in settings and fill in my mobile number, then hit [register] I see the message reading that Google play is missing.
My only choice is "Get Google Play" but nothing happens when choosing that. I can then only skip registration. Should I paste logcat or submit debug log?

@marciomr
marciomr commented Dec 6, 2014

I have got the same message here.

@jensMF
jensMF commented Jan 2, 2015

Is there any progress in websocket implementation since last post / last activity at javajens fork (27 days ago)?
Is it possible to build TextSecure with websocket and communicate with people that have the official gcm build?
If not, is it possible to unregister from TextSecure push with websocket? I tried to unregister, but my friends with official gcm build automatically send me push messages and I can't receive thus...

@JavaJens
JavaJens commented Jan 3, 2015

Sorry, there has been so much silence. I didn't have the time to further work on this.
@Selaron and @marciomr Please open up an issue on my fork, if possible with logcat.

@jensMF As far as I know, it is possible to communicate with the official GCM client, but you are not supposed to run this code against the production server, staging only. At least that is what I read from moxies comment some while back.
Regarding unregistering...make sure that you are running on the same server (production vs. staging) and if that still not works, try to get a debug log and open up an issue on my fork.
To me it wasn't exactly clear how unregistering works, as the Wiki Page only specifies unregistering of GCM or APN codes.

@patcon
patcon commented Jan 3, 2015

@jensMF I'd say just install the official app, re-register, and then unregister for push notifications using the standard advice:
#845 (comment)

@jensMF
jensMF commented Jan 5, 2015

@JavaJens Thanks for your answer, but how can I check the server where I'm running textsecure on? And how can I change it? (I tried to unregister with websocket version 2.2.0 because I registered with that one, that should run on the same server cause I used the same apk) For the last try i used the latest version of your fork (I checked out the repository, installed everything needed via android-studio and did ./gradlew build). I can send masseges, but don't receive messages…

@patcon I don't have gapps and I don't want to install them…

@ddast
ddast commented Jan 5, 2015

@jensMF You can unregister from TextSecure on this website: https://whispersystems.org/textsecure/unregister/

@JavaJens
JavaJens commented Jan 5, 2015

@jensMF If you haven't changed anything then you are using the Staging (aka correct) server.
However I never touched the version number, hence I would look at the commit checksums for verification.

You can check the server in the Release.java file. If you need more help, open a ticket on my fork or send me a mail.

@jensMF
jensMF commented Jan 6, 2015

Thanks @ddast, that worked well.

@javajens is it possible communicate whith playstore-version when being on the staging server?

@JavaJens
JavaJens commented Jan 6, 2015

@jensMF As fas as I know, the two servers are not federated. That is if you
register against the staging server, people on the other server don't know
that you use TS and can't send you messages.

In order to communicate with them, you would need to change the
Release.java file, but shouldn't be using development code against a
production system.

@ThomasWaldmann

+1 for textsecure without google services. can't use it currently due to that.

@jensMF
jensMF commented Feb 14, 2015

Thanks for all the information, it helped me out very well. Are there any news about websocket or plans getting away from gcm? Maybe @moxie0 could say something about it? Or does @JavaJens knows something?
Thanks in advance.

@h-2
h-2 commented Feb 16, 2015

I recently tried the last version in @grote 's repository. It still drains a lot of battery. If there is any testing and reporting that I could do to help, please let me know. All Jabber-Clients I know, Kontalk, Telegram ... all have some mechanism that has only a slight influence on the battery life...

@JavaJens

I don't know which version you were using, but I do know that the server has a connection timeout of 15s. This, in my opinion, is the only reason why there is such an enormous battery drain. Compare it to GCM which is said to have a timeout of 15m [1](however I don't know if this is outdated info). The server side of things will most likely not be updated, due to limited server resources: https://github.com/WhisperSystems/TextSecure-Server/issues/27.

About the progress of integrating WebSocket in TextSecure; I know that my initial PR was not very good and required lots of refactoring, furthermore there were some changes planned that would break the communication again. Now I've lost most interest in it as I don't know if/when it will break again and if it would ever be accepted. I don't know how the crew at Signal deals with battery issues, maybe they have a completely different way (like only establish the connection on APN or if the app is focus), but it might be worth to check that out.

So again, feel free to pick up my work, I'd be more that happy to assist :)

[1] https://productforums.google.com/forum/#!msg/nexus/fslYqYrULto/lU2D3Qe1mugJ

@JavaJens

If I see it right there is support for WebSocket on the roadmap for milestone 2.6 https://github.com/WhisperSystems/TextSecure/milestones/v2.6.0

See PR #2423

@schiessle

This sounds awesome! Is there any estimation when 2.6 will be released?

@jeremymasters

Sometime after 2.5.3, sometime before 2.6.1. :)

@moxie0
Member
moxie0 commented Feb 18, 2015

That is not "websocket support" in the sense you want, which is still a ways off.

@devurandom

@moxie: What is the purpose of #2423? I am unable to make it out from the diffs.

@JavaJens

I believe it is to receive messages but triggered by gcm. This would prevent messages from being lost.

Haven't checked thoroughly though.

@h-2
h-2 commented Feb 19, 2015

So again, feel free to pick up my work, I'd be more that happy to assist :)

Thanks for your work thus far, JavaJens. But to be honest, judging from this thread and other conversations, I feel like the overall climate here is not very friendy towards suggestions or even pull requests, so for now I will spend what free time I have on other Free Software projects like Kontalk [no offence to anyone here, its of course their choice where to put priorities].

@jensschulz

Would it be a possible alternative to implement polling (e.g. every 15 minutes) until websocket support is available? I use this on Threema and it's almost as good as push and there is no heavy battery drainage. The load on the server shouldn't be too heavy as only few people are not using GCM. I really would like to switch to TextSecure, but as many others won't install Google Services for it.

edit: Issue has just been closed (#2530)

@ThomasWaldmann

@jensschulz maybe open a separate ticket for that and link to it from here? would also like such a feature!

@moxie0 moxie0 locked and limited conversation to collaborators Mar 4, 2015
@nrizzio nrizzio referenced this issue Feb 20, 2017
@moxie0 moxie0 Support for using Signal without Play Services
This is now possible with beta calling, so non-GCM users are a
part of beta calling by default.

// FREEBIE
1669731
@moxie0
Member
moxie0 commented Feb 20, 2017

in 3.30.0

@moxie0 moxie0 closed this Feb 20, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.