New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduction of Bandwidth/Traffic Consumption #1251

Closed
srkunze opened this Issue Feb 6, 2015 · 138 comments

Comments

Projects
None yet
@srkunze
Copy link

srkunze commented Feb 6, 2015

Is there something we can do about it?

Antox/Antox#6

@ollieh

This comment has been minimized.

Copy link
Contributor

ollieh commented Feb 9, 2015

Mobile apps aren't really supposed to be constantly connected to servers, but to use the push notification system the OS provides. I can't think of a way to do this without using a centralized server. The best situation I can think of is a centralized server just for sending the push notifications, and then the mobile app sends a message to the central push notification server authorizing the TCP relay it is connected to to send everything it receives for that user to the push notification server. It's not such an issue on Android, because you can have stuff working in the background, and we were getting half-decent battery life at one point so it is possible. You can't get anywhere near the battery efficiency of push notifications though, and for iOS it's a complete showstopper. On iOS you can't have things running in the background for more than a few seconds or maybe a minute, so a messaging app on iOS can't work in the background, and must use push messages if it wants to actually be useful. I think it also means user-written messages would have to go through the push system, not just friend requests and such. I may be wrong but it doesn't look promising. Can anyone see an easier way out?

@srkunze

This comment has been minimized.

Copy link

srkunze commented Feb 9, 2015

@ollieh I think what you said is really important. Not sure how push notifications work. If the app can configure the push server freely, it could be changed to respective peers regularly. Could this work?

I want to emphasize the fact that mobile will be (is already?) the dominant way people use chat/voice/video applications. A solution here is crucial (at least in my opinion). Most people will never (exaggerated of course) use two systems (Skype+Tox) because of the network effect (friends etc.).

@ollieh

This comment has been minimized.

Copy link
Contributor

ollieh commented Feb 9, 2015

From the android docs:

Send data from your server to users' Android-powered devices
This could be a lightweight message telling your app there is new data to be fetched from the server or...

It seems like this approach might work. The TCP relay could hold the messages for a little, send a message to the push server to tell the user's client to query the TCP relay, and then pass the messages on when it does. That way you wouldn't necessarily need to authorize each TCP relay as the worst thing they could do would be constantly send push messages that tell you to query the network, to drain your battery. As it'd be a central server controlling it you could just have a timer on it. I wish there was a way to do this without a central server though.

I totally agree with you about mobile. If Tox is to succeed, it has to be fantastic on mobile. Everyone seems to have their sights set on beating Skype, but we can't beat Whatsapp yet. iOS is impossible at the moment, and I don't know about you, but most of my friends have iPhones, and Tox isn't very useful if your friends can't use it. I know most of us probably prefer using desktop applications, but if it's desktop only it'll never take off. If it just did mobile well it'd stand a much, much better chance, even if the desktop clients were shitty or non-existent. We need to solve push messages, and offline messages.

@tttom

This comment has been minimized.

Copy link

tttom commented Feb 11, 2015

@ollieh, I second that. I'd say that the average internet user does not have a computer anymore, but almost certainly just a smartphone of some kind. Battery life and data usage cost of smartphones practically mandate the use of iOS push notifications or Google Cloud messaging. In principle one could sign up to one or more message services of choice, which would keep encrypted messages and send push notifications to their phone. Although I believe that the push notification service is free of charge, running a message service server to send the push notifications would not be entirely free. Thing is, I don't see any incentive in running such a message service unless it is for oneself or ones friends. Bittorrent works because uploaders get somehow rewarded with faster download speeds. Free e-mail providers get to peek into your messages to increase their ad revenue. But here, message service providers would just pay for the bandwidth of smartphone users. Hence, my honest question, is there anything that smartphone users can offer in return for good battery life and minimal data costs?

@srkunze

This comment has been minimized.

Copy link

srkunze commented Feb 16, 2015

@tttom You are right. Not sure if there could be a charity organisation which can collect some money and distribute it to node operators to cover their expenses.

However, we should give people a sensible option first on mobile before thinking about compensation.

@srkunze

This comment has been minimized.

Copy link

srkunze commented Feb 16, 2015

#on-topic @dubslow estimated here #1254 (comment) that one package/message could be 1-2 KiB.

Unfortunately, the tablet traffic monitors tells us about 50 KiB per message. What is wrong here? Is onion routing increasing the traffic so much?

@dubslow

This comment has been minimized.

Copy link
Contributor

dubslow commented Feb 16, 2015

There are indeed more packets than just the message packets...

@srkunze

This comment has been minimized.

Copy link

srkunze commented Feb 16, 2015

25 fold more?

@Kolargol00

This comment has been minimized.

Copy link

Kolargol00 commented Jun 12, 2015

I'm seeing Antox use a huge amount of data on wifi: in the range of 10MB to 50MB per hour while I'm not using the app at all. Is there any number/log I can report to help you guys diagnose this?

@fcore117

This comment has been minimized.

Copy link

fcore117 commented Jun 12, 2015

i hope there are more TCP optimizations, could be useful especially for 2G, 3G, 4G internet users and for very low bandwith ADSL users.

@dcharles525

This comment has been minimized.

Copy link

dcharles525 commented Jun 12, 2015

This has to be fixed, or Tox on the mobile platform will fail.

@LapFox

This comment has been minimized.

Copy link

LapFox commented Jun 12, 2015

Tox on desktop can also be under threat, for people with data caps on their broadband (it's more common than you think)

@fcore117

This comment has been minimized.

Copy link

fcore117 commented Jun 12, 2015

LapFox: this is very common outside of city like farms and houses in woods, mobile broadband is only option for desktop. I myself even have helped some people to set up usb stick to get better signal. Most common monthly plan is 5-20GB and only very rich could get 100GB to infinite bandwidth.

I really hope this is resolvable without sacrificing p2p, servers should go to hell as they are easily banned, filtered, blocked.

@srkunze

This comment has been minimized.

Copy link

srkunze commented Jun 14, 2015

Is there anything the community can do here?

@Kolargol00

This comment has been minimized.

Copy link

Kolargol00 commented Jun 16, 2015

Disabling UDP and installing a recent Antox build brought down data usage to much lower levels (< 1MB / day). Unfortunately, I can't say which operation changed that since I did both together.
Scrap that :( I opened Antox then sent 2 messages and data usage is now more than 100MB in 4 hours.

@srkunze

This comment has been minimized.

Copy link

srkunze commented Jun 21, 2015

What the hell?

@lovetox

This comment has been minimized.

Copy link

lovetox commented Jul 5, 2015

you should give this issue the highest priority

as some said before, tox can and never will be the future of online communication, if it is unusable on smartphones.

@grinapo

This comment has been minimized.

Copy link

grinapo commented Aug 24, 2015

Maybe it would help to track down the issues to generate stats about the various kinds of traffic tox uses and see what uses how much, and after it could be checked why and whether it can be shrunk or not. My guess (without looking) is that it may be an inherent design problem with the protocol and would go the same as bleep: unusable on mobile due to extreme high protocol traffic overhead.

@optimumtact

This comment has been minimized.

Copy link

optimumtact commented Oct 18, 2015

This issue has become much more important, because the upcoming versions of android will require apps to properly sleep and only wake up on messages from google push notifications.

AFAIK IOS already does this

@ameenross

This comment has been minimized.

Copy link

ameenross commented Nov 12, 2015

They usually parasite on a "geek" and see us as mere tools.

QFT 😆
"Geeks" should probably just charge more! (Sorry for offtopic, btw)

@subliun

This comment has been minimized.

Copy link

subliun commented Nov 13, 2015

There is no p2p solution to this problem. Antox will adopt GCM to send wakeup pings/messages sometime in the future, which should mostly remedy the bandwidth issue. I am sure improvements can be made in toxcore to improve bandwidth usage also.

@quininer

This comment has been minimized.

Copy link

quininer commented Nov 13, 2015

@subliun GCM? Can you describe how it will be implementation? It requires a special server?

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Nov 13, 2015

@subliun irssi has a nice plugin that encrypts the messages using a shared key so that neither the app, nor google can see what's in the message. I see no reason you couldn't just push the encrypted messaged to your app.

@subliun

This comment has been minimized.

Copy link

subliun commented Nov 13, 2015

@GrayHatter everything going through GCM will be encrypted of course.

@timofonic The F-Droid build will likely never contain any GCM-related functionality. Unfortunately, due to restrictions imposed by Google in Android M, Antox is forced to adopt GCM or be left crippled and unable to receive phone calls, file transfers or messages in the background. Because of this, the Google Play build of Antox will be using GCM to deliver messages and updates in the background.

@tttom

This comment has been minimized.

Copy link

tttom commented Nov 13, 2015

@subliun, as sending Google Cloud Messages requires each push server to have a Google API key, how would this be implemented for Antox? Ideally all tox nodes should be able to send GCM to antox users, though I don't see it happen that everybody will set up an account with Google, and later Apple etc. Would each Antox user have to run such a push server themselves or would there be a hard coded antox.net server perhaps?

Alternatively, with modifications to toxcore, one could designate one or more relays to be contacted through. Google Play users could register with a relay that is capable of reaching them by GCM and inform their contacts of this, i.e. an extended toxid. Relays would only see encrypted messages destined for that specific Antox user, hence one doesn't necessarily need to run their own service. The ability to specify multiple relays would enable high availability with low cost hardware, and perhaps even offline messaging. The configuration could remain as simple by setting up a default in the client. Users that don't like relays can specify no relay, and watch their battery drain... The designated relay principle could also be used for iOS as well as any open source alternatives for GCM.

@subliun

This comment has been minimized.

Copy link

subliun commented Nov 13, 2015

Would each Antox user have to run such a push server themselves or would there be a hard coded antox.net server perhaps?

This is accurate. My current plan is for some extension in toxcore to allow telling your friend somehow that you want your messages relayed over GCM. Then when you go offline messages will be sent to an offline messaging server/TCP relay server/to be determined, which will forward to an antox.me server set up to handle rate-limiting and abuse prevention. This server will have the GCM api key and will send the messages/wakeup updates through GCM which will be received in the background by the recipient. When Antox is in the foreground (thus it is online) messages will be sent directly as they are now.

I'm open to future extensions which allow more customisation or specification of custom forwarding servers, though I'd like to get the basics worked out first.

@fcore117

This comment has been minimized.

Copy link

fcore117 commented Nov 13, 2015

so Antox will not work with TCP mode too to send files and so on like toxcore should?

I am user who has no Google services installed, pure slim barebone.

@subliun

This comment has been minimized.

Copy link

subliun commented Nov 13, 2015

@fcore117 it will, but not in the background. Please read https://commonsware.com/blog/2015/06/03/random-musing-m-developer-preview-ugly-part-one.html as it covers the issue very well.

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Jan 18, 2016

So, in 24h (longer if I get distracted) I'm going to close this issue. 3 reasons. first; some of the new DHT commits will have fixed this issue (at least in part). 2nd we've clearly verred way offtrack/topic, so much I don't thin. We're coming back. And 3rd, I don't think anyone wants GCM code in toxcore.

If you still think we need a toxcore solution to this antox problem. Open a new issue here, describe the problem really well, and describe what's required for a fix.

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 18, 2016

@GrayHatter: This ticket is not about gcm - gcm was one proposed
solution.

And the issue is not an Antox-only issue, it is an issue with any mobile
client, be it on Android or iOS. Even desktop user with really low data
caps would benefit.

What would be needed for a fix was already outlined several times:
a mode in which a client maintains only one tcp connection and no
udp connections at all, does no forwarding (I think current tcp-only
mode already does that part) and pings or keepalives at most every ten
minutes.

@xavierle

This comment has been minimized.

Copy link

xavierle commented Jan 18, 2016

thank you for sum up andy

this seems to me to be linked to tow core
so that it is easy to implement for every client later
Le 18 janv. 2016 08:43, "aaannndddyyy" notifications@github.com a écrit :

@GrayHatter: This ticket is not about gcm - gcm was one proposed
solution.

And the issue is not an Antox-only issue, it is an issue with any mobile
client, be it on Android or iOS. Even desktop user with really low data
caps would benefit.

What would be needed for a fix was already outlined several times:
a mode in which a client maintains only one tcp connection and no
udp connections at all, does no forwarding (I think current tcp-only
mode already does that part) and pings or keepalives at most every ten
minutes.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Jan 18, 2016

@aaannndddyyy this would be perfect time for you to open an issue to that effect. This thread has become unmanageable as an issue that can be marked as solved to everyone [agreeability?].

Not to mention the OP and the title mentions neither of those suggestions.

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Jan 18, 2016

My goal is to make issues easy to work on any close. This thread would take longer to read, than to fix at this point, so it's no longer a code issue, but a discussion on what color to paint the bikeshed.

@xavierle

This comment has been minimized.

Copy link

xavierle commented Jan 18, 2016

this is clearly a discussion
but at some points a way have been shown

no problem to close this an recreate a more defined one.

why do you seem borred/ angry about this kind of discussion about the
project ?

would you prefer noone is interest in the project to avoid such debate ?

personally i do not enjoy the way you say thay you will close the issue
(even if i agree the reason)
neither your sarcasm about discussion and painting.

hopefully in the future issue your smile would be back on track
Le 18 janv. 2016 08:53, "GrayHatter" notifications@github.com a écrit :

My goal is to make issues easy to work on any close. This thread would
take longer to read, than to fix at this point, so it's no longer a code
issue, but a discussion on what color to paint the bikeshed.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 18, 2016

@GrayHatter If you think that my description is at least somewhat
reasonable and viable, then I will create a new issue with that content
tonight. Right now am kinda busy.
If you think it's outright bs or unfeasable, then no need to create a
ticket that will immediatelly get closed anyhow.

Feedback appreciated.

@ameenross

This comment has been minimized.

Copy link

ameenross commented Jan 18, 2016

This seems to be a meta issue. There's probably a wide range of possibilities to reduce bandwidth. I think each should be discussed separately as well.

Also, FWIW, I'm getting vastly better battery consumption with Android 6 (cm-13) compared to Android 5.1 (cm-12.1). And I haven't used Google apps for a few years (not installed).

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Jan 18, 2016

@aaannndddyyy I think your description is very workable.

I don't have a problem with this issue, nor with it turning into a
discussion. I'm not going to lock it either. Only going to close it, simply
because the title and OP are actually completed. So now this is simply a
discussion about how best to deal with android M and that's not an issue
for toxcore.
On Jan 18, 2016 12:08 AM, "aaannndddyyy" notifications@github.com wrote:

@GrayHatter If you think that my description is at least somewhat
reasonable and viable, then I will create a new issue with that content
tonight. Right now am kinda busy.
If you think it's outright bs or unfeasable, then no need to create a
ticket that will immediatelly get closed anyhow.

Feedback appreciated.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@srkunze

This comment has been minimized.

Copy link

srkunze commented Jan 18, 2016

@aaannndddyyy @GrayHatter Good suggestion. I am going to close this issue as soon as I see @aaannndddyyy's proposal.

This issue led to a number of possible ways of fixing things. So, now it's time to seize discussion and go coding on specific proposals.

@LuccoJ

This comment has been minimized.

Copy link

LuccoJ commented Jan 18, 2016

@GrayHatter except that toxcore still uses a lot more bandwidth (and hence, on mobile devices, power) than other typical IM systems, even in TCP relaying mode, where it really has no reason to.
I like what @aaannndddyyy is proposing (fewer connections, fewer keepalives), so I'm all for going ahead with it, but why does this issue need to be closed, given all that has only been stated but not done yet?
The bandwidth/traffic consumption issue is all still there, for now.
The DHT fixes only impacted UDP mode (since TCP mode doesn't partake in the DHT at all), so they help mobile devices very little, unfortunately. Antox, for instance, only uses TCP by default, so it isn't helped at all.
The fixes also reduce the number of open connections more than the actual bandwidth taken.

@xavierle

This comment has been minimized.

Copy link

xavierle commented Jan 18, 2016

i was a bit surprised but reasonable reasons were pointed out

this thread became a discussion and not an issue.
as said the text is longer than the issue

it will be closed as an issue but not locked.
discussion will remain open.

as a results of the discussion, more precisely defined issues will have to
be created.

andy made a good sum up of the possible ways.
Le 18 janv. 2016 18:18, "LuccoJ" notifications@github.com a écrit :

@GrayHatter https://github.com/GrayHatter except that toxcore still
uses a lot more bandwidth (and hence, on mobile devices, power) than other
typical IM systems, even in TCP relaying mode, where it really has no
reason to.
I like what @aaannndddyyy https://github.com/aaannndddyyy is proposing
(fewer connections, fewer keepalives), so I'm all for going ahead with it,
but why does this issue need to be closed, given all that has only been
stated but not done yet?
The bandwidth/traffic consumption issue is all still there, for now.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 18, 2016

@ALL FYI: #1501

@srkunze

This comment has been minimized.

Copy link

srkunze commented Jan 18, 2016

Discussion can be continued of course. :)

@srkunze srkunze closed this Jan 18, 2016

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 18, 2016

is it better to discuss here ?
If we discuss in the new one, it might become long again too and then become unreadable again.

@GrayHatter

This comment has been minimized.

Copy link
Collaborator

GrayHatter commented Jan 18, 2016

It's better to discuss either on the ML or IRC. Here is fine, but don't let
any of the other issues get sidetracked because then we'll have the same
problem.

Issues only become too long when other issues are raised and discussed.
On Jan 18, 2016 10:55 AM, "aaannndddyyy" notifications@github.com wrote:

is it better to discuss here ?
If we discuss in the new one, it might become long again too and then
become unreadable again.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 19, 2016

Could someone who has whatsapp and/or signal please profile that app for a week or so, as to see how much bandwidth those use when idle or with only text messages?

Would be nice to have a value to compare with.

@xavierle

This comment has been minimized.

Copy link

xavierle commented Jan 19, 2016

for january

whatsapp 179ko text only
freelab messenger/ conversations : 5mo with some images
signal : 3mo with some images

aml this recquires very few data

Le 19 janv. 2016 12:40, "aaannndddyyy" notifications@github.com a écrit :

Could someone who has whatsapp and/or signal please profile that app for a
week or so, as to see how much bandwidth those use when idle or with only
text messages?

Would be nice to have a value to compare with.


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@aaannndddyyy

This comment has been minimized.

Copy link
Contributor

aaannndddyyy commented Jan 19, 2016

Thank you.
This is SIGNIFICANTLY less than tox uses.
Ideally, tox on mobile should have a traffic consumption of that order...

@srkunze

This comment has been minimized.

Copy link

srkunze commented Jan 19, 2016

@aaannndddyyy That's indeed a very good benchmark test.

@GrayHatter Is there someway reproducibly testing this?

@xavierle

This comment has been minimized.

Copy link

xavierle commented Jan 19, 2016

i do not claim any accuracy on the information i sent.
i took the information from android bandwith app usage

i do not know if i use 3 apps the same way.
just all 3 are all day active and daily used with 5 persones each
Le 19 janv. 2016 22:17, "Sven R. Kunze" notifications@github.com a écrit :

@GrayHatter https://github.com/GrayHatter That's indeed a very good
benchmark test.

@aaannndddyyy https://github.com/aaannndddyyy Is there someway
reproducibly testing this?


Reply to this email directly or view it on GitHub
#1251 (comment)
.

@ghost

This comment has been minimized.

Copy link

ghost commented Mar 23, 2016

I feel foolish to have encouraged others to try Antox. It drained the bandwidth of a whole month within days. this should at least be obvious before using.

@LittleVulpix

This comment has been minimized.

Copy link
Contributor

LittleVulpix commented Mar 23, 2016

Whenever someone asks about a mobile client (for tox), I have to:

a) explain there are no accounts and so they will need to create a new profile and add me again
b) explain it will kill their battery if it is running
c) explain it will eat most of their bandwidth so it is to be used only when connected via WIFI or if they have an unlimited data plan
d) explain that there are no offline messages

Not really antox's fault, but I feel that antox is like trying to make a portable construction for your desktop. Yeah you can make it but you'll need heavy batteries and your back will probably hurt after 15 minutes.

I agree with @mray - users should be clearly told that antox will do all of the above to their device/data plan.

@ProMcTagonist

This comment has been minimized.

Copy link
Contributor

ProMcTagonist commented Mar 26, 2016

Alternatively -- I'm simply not able to evangelize Antox until those things are fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment