Bandwidth Reduction for Mobile Clients through a "Mobile Mode"

[aaannndddyyy]

Currently, despite some attempts to fix it, toxcore still uses way too much bw and connections to be a viable im solution on mobile clients.
This is not an Antox-only issue, it is an issue with any mobile client, be it on Android or iOS. Even desktop users with really low data caps would benefit from a solution.

Therefore, I propose creating a mode in which toxcore works in a reduced manner and uses much less bandwidth. This mode you could call "Mobile Mode" or "Reduced Mode", …

What would be needed for a fix:
A mode in which toxcore

• maintains only one tcp connection and no udp connections at all,
• does no relaying of traffic for others, including gc (I think current tcp-only mode already does that),
• does not participate in the dht, and
• sends pings or keepalives at most every ten minutes (concrete best number to be found it).

The solution could make use of current tcp relays, which should become the standard mode of operation for "full participation mode" toxcore.

This would drastically reduce ressource consumption, and totally do without gcm or even an alternative centralized push server set up by Tox Foundation.

[LuccoJ]

• I think realistically, two TCP connections rather than one would be ideal, because a TCP relay can always go offline (it can be any peer after all), and there should be a backup to avoid going offline while finding a new relay. However, I think with the current relay architecture, you need at least one TCP connection per contact you're communicating with.
• Yes, by default, you don't act as a relay yourself when in TCP mode; only if you change the "TCP port" from 0 to a different number is the relaying activated.
• I don't think you partake in the DHT when you are in TCP mode even now.
• Ten minutes sounds like a bit much to me: actual ISP timeouts may be shorter, and it would mean that if your connection dies, it will take ten minutes before your contacts see you as offline. I think a minute or two should be alright.

Finally, I don't think this deserves a separate new mode: it should be TCP mode, and all efforts to optimize it should go towards one "lightweight" mode, while other clients should be encouraged to use full UDP mode (the DHT is very important after all).

[aaannndddyyy]

@LuccoJ Seeing someone as offline or online is actually not as important, IMHO. If the connection is gone, then your peers either want to send you a message and it fails, so they know you are offline and faux offline messages take care of delivering it once you are back online; or they didn't want to send you anything - in that case they don't really care if you are online or not.

[fcore117]

and i think this optimization would not only benefit mobile users but laptops with bandwidth limited 3G/4G internet and huge family houses where lot of people share one crappy router which shares IPTV boxes too.

[LuccoJ]

@aaannndddyyy I agree that very immediate offline-status detection is not a necessity, but I think 10 minutes is way overkill. And don't forget the other part, i.e. that it can cause timeouts on TCP stacks at the ISP side... it's not just about offline status.

If you ping every minute or two, I think that would be a fine compromise, provided there aren't too many connections (because the pings will multiply). The minimum TCP packet size is 40 bytes, so if you send one ping per minute, you waste only 1.7Mb a month, which won't really be a problem for any current connection.

We are going to have more than one connection, though, and I think it could be an important optimization to have all the connections send and receive pings simultaneously, to only wake up the device once per grou pf pings: if you have 10 connection, and ping once a minute, that could mean 10 pings scattered over 1 minute (which might be bad), or a single group of 10 pings (which could be okay).
How about sending and responding to pings at specific times only? For instance, instead of just pinging "every 2 minutes", we could ping "at every even minute, on the minute". This might help if the clocks are synchronized enough (and these days, clocks tend to be NTP-synchronized).
I'm not sure if it would make a practical difference (there is also random network latency to consider), but maybe it's worth a thought.

[Renha]

I'm not sure it's good idea. How to limit users to don't use mobile-mode too much? Because if everyone in tox network would use mobile mode, there wouldn't be tox network anymore. Why all users would want it? Because it is cheaper, in CPU and network usage.

I think you should use full remote client instead, like one i'm working on. You start client on VPS, and connect to it any way you want. You will get multiple-clients-one-user-problem solution as a bonus, because you will use single client over all range of your devices.

[ProMcTagonist]

You start client on VPS, and connect to it any way you want. You will get multiple-clients-one-user-problem solution as a bonus, because you will use single client over all range of your devices.

Cool, but the only way normal users will use this is if someone sells configured tox vpses as SaaS.

Not what we want to encourage.

We need a good mobile mode.

[Renha]

@ProMcTagonist
so we have unanswered question for both solutions

[ProMcTagonist]

How to limit users to don't use mobile-mode too much?

T-shirts like tor does for supernode runners, and warning modals in desktop clients when you activate mobile mode. I'm certain almost no desktop users will be enabling it, I'm certain qTox will never have it on by default, and I'm certain I will be making people feel guilty if they have it on needlessly.

[Renha]

well, I agree that that might be useful, but details are unclear. For example, i think keepalive should not be too rare, because network on mobile devices is unstable when you are traveling in underground or at train, or anywhere mobile client could be useful, in ten minutes, proposed in /0, i usually switches online/offline op to 10 times.

probably instead of single Mobile Mode there should be some settings like set_network_mode( tcp/udp/both ), set_keepalive_timings( msec ), ask_tcp_relays_to_compress_incoming_files_if_possible_please(), set_dht_participation( true/false ), etc.?

[LuccoJ]

@Renha This mode where people don't partake in the Tox DHT already exists, it's called TCP-only mode, and any client can enable it. People can already elect to use it, both on mobile (like Antox) and fixed clients. Now it needs to be improved.

Running a proxy on a VPS is so totally not a solution. I, for one, am certainly not going to rent a VPS just to do my mobile instant messaging. It would also centralize Tox back, at least for the person involved, to the VPS being used: if the VPS goes down, so does the mobile client.

People using fixed landline clients usually won't have enough motivation to enable TCP-only mode, because the CPU usage is negligible, and the bandwidth usage is only important if you have a cap. This is demonstrated by the fact that the vast majority of users right now are keeping UDP mode enabled, and the Tox network is working, even though they could disable it.

[gianni76]

The majority of people I know (really a lot given I work in the IT field) seek a true p2p solution with a light initial discovery mode. The discovery mode should only let the 2-few peers discover and then allow a direct channelling among them. (of course with a low-rate status check/redescover mode running with pings every minute or so). This is a bit like Ricochet is working (and I would definitely use ricochet if it had p2p file transfer, which is essential to me at present).

The description in the first message of this thread (very useful!) let me understand that currently uTox is instead a decentralised relay network where the traffic itself is relayed through other users. This should be made clear in the documentation, as I honestly never thought it would work like this.

The problem of enormous bandwidth consumption per low payload is a real issue in mobile mode, but is a concern also for standard low-speed ADSL home connections. The UDP mode should definitely turned off by default as this option is not responding to the true p2p (chat tunnelling) concept!!

[Renha]

@LuccoJ the majority of people @gianni76 knows wants to definitely turn off UDP mode by default.

[ProMcTagonist]

if you turn off udp mode all your messages go through a relay and not directly to your friends

[gianni76]

Is there a simple but precise section explaining how Tux is working?
I have been trying to turn UDP off and can guarantee that the bandwidth consumption is reduced significantly, as I was hopying from the start, so I am confident that with this option the relaying (by others on my machine) IS OFF.

Why is ProMcTagonist saying that turning udp mode off, 'all your messages go through a relay and not directly to your friends'? It is counter-intuitive for me to understand why would this be the case...

The most efficient way to channel sequential information from a source to a peer (1-to-1 communication) is to open a single tunnel and pass it trough (encoded and encrypted of course) and not distribute through multiple peers and then recombine it. I honestly don't know how this is done in Tux, but would be surprised to see relays in such case (udp off).

[im-grey]

@gianni76 It's Tox, not Tux...

[LuccoJ]

@gianni76 with UDP mode, you communicate directly with your friends, but you also take part in the DHT, which translates Tox IDs into IP addresses (people must know what their friends' address IS), and means many connection and a lot of used bandwidth.

With TCP-only mode, you don't take part in the DHT, but to communicate with your friends, you don't connect to them directly, but only to other people who are acting as TCP relays.
This is also useful in UDP mode when both parties are behind symmetric NAT and UDP communication won't work.

In neither mode you are being used as a relay by others: that only happens if you enable that explicitly. But in UDP mode, you are helping other people translate IDs into IPs.

[gianni76]

Dear LuccoJ,
thank you for the explanation. I still fail to understand the concept for the following reason:

a) With the UDP mode on, the bandwith consumption is CONTINUOUS and HUGE.
I talk about over 100 MB a day. It looks to me IMPOSSIBLE that we are only talking of peer discovery (i.e. basically a Tox id-to-ip-distributed lookup).
b) I would like to distinguish the two separate temporal windows:

• peer discovery
• peer data exchange (chat and/or file transmissions).
  I do understand that the peer discovery mode will need a bit of bandwith (both in TCP and UDP modes), but it should be limited to the initial part of the connection. For example for most people once every xx minutes will do and after the initial daily full check even a simple reciprocal micro-ping will be sufficient to check no changes have happened. This discovery bandwidth can be theoretically very little a day. (probably 1/100 of the over 100MB consumed now).

Once the two peers have their reciprocal IPs (once the discovery is complete/recheked), I see zero reason to use relays. All the communication should be done via direct encrypted channels peer to peer (without any relays) and the same is valid for files exchange etc.
Why should my data been relayed once I know the IP of my peer and want to communicate only with him?

My idea of p2p chat is that the DHT/relay part should be limited to the discovery mode ONLY and stop there. And this of course, besides optimising the bandwith waste, is also for privacy/security reasons Why should potentially sensible data go trough unsecure relays when it can go directly trough a tunnel to my peer?

[LuccoJ]

@gianni76 as to "Zero reason to user relays": perhaps you've never been under a hard, symmetric, ISP-enforced NAT. I cannot communicate with other similarly NAT'd clients without the help of relays, because the "reciprocal IPs" simply don't exist.
For me (and anyone using similarly set up ISPs), this is a problem with a lot of peer-to-peer pieces of software, and I'm glad Tox makes an effort to solve my use case. If you don't need to use this feature, you don't have to: Tox is supposed to only use it when needed. I need it, please let me use it.

There aren't any additional security concerns with relays, either: the data exchanged are always encrypted, of course, and if you want to take the scenario where encryption is broken, well, that's a scenario that would make Tox in its entirety completely insecure, whether or not you use relays.

Of course, I agree that Tox in normal UDP mode seems to use a lot of data; I'm not entirely sure why that is, but don't forget that aside from the DHT lookups ("ID-to-IP"), there is also the onion routing (for friend requests), and of course, simple keepalives, which I suspect are sent quite often.
The reason you're using so much bandwidth is not that you're acting as a relay for others, just to be painstakingly clear.

[gianni76]

Yes, I do understand that in very peculiar cases (roughly 1%) relaying might be useful.
What I fail to understand is how it is possible that this function is core and default (and not selectable) when in 99% of the cases, after the brief discovery phase, a simple direct source-ip-to-destination-ip tunnelling is the most efficient way forward for p2p communications.

And with regards to security, I am sorry but you are totally wrong.
Yes I appreciate encryption and find it absolutely necessary, but this might not be sufficient (now or in the future).A future-proof design should simply avoid the possibility of data tampering at all times.

If the system did not relay personal data, but send them direct ip-to-ip, the chances of anyone able even to steal it (let alone then to decrypt) decrease significantly (to almost 0% and only with rather challenging criminal activities).

Relay should be an option for those having difficulties or strange network configuration.
At very least the design should make possible to turn off this option and have direct ip-to-ip tunneled encrypted communications (more efficient and more secure) for the 99% of us who have open simple standard non-nat data comms.

[gianni76]

BTW: given you asked, I arrived to uTox while looking for a secure p2p decentralised communication tool. This is a simple way to communicate (almost exclusively text chat) with peers/coworkers and exchange data file on the go. Some of the data are sensitive, hence we would rather have a direct ip-to-ip transmission rather than having the data travel through relays.and be potentially exposed to tampering/external recording.

I am looking for a pure p2p tunnelling solution. The closest to the concept seems to me Ricochet, but the absence of file exchange makes it impossible to use for the moment.
uTox with UDP turned off seems perfect in terms of functionalities, but the technical use of relays (instead of the much simpler and more secure source-ip-to-destination-ip tunnelling) honestly does not leave me happy.

[ProMcTagonist]

What I fail to understand is how it is possible that this function is core and default (and not selectable)

it's not, UDP is default

what you want is UDP without participating in the DHT?

[LuccoJ]

@gianni76 you are misguided and you aren't listening, I am not "totally wrong". If encryption somehow weren't enough to stop "data tampering", then that could happen at any router along the route of your IP packets, not just on your hated TCP relays, so that's a completely moot point.

And as mentioned, TCP relaying is not default and is not done unless necessary (except on mobile, where it clearly is necessary). End of story.

You are also clearly confused about Ricochet, because since Ricochet runs the Tor network, it does not make "simpler and more secure source-ip-to-destination-ip tunnelling", but in fact, it uses a lot more relaying that Tox with TCP relaying does.

[mrkiko]

A mobile mode would be very interesting for sure: still UDP relaying (and even
TCP one) are fundamental if I understood correctly Tox design goals. In regard
to security: I was thinking Tox was designed to try to do its best to guarantee
it even across relays who can't decrypt the data or tamper with it without the
two parties detecting it. Am I wrong? An definitely, "direct" connections are
"direct" only to a certain extent (ISPs infrastructure and so on).
This is only my opinion and I am glad I can learn something from you all.

That said, in my vision Tox may even try to do TCP and UDP relaying when it
detects it can unless the user explicitly asks differently. I would also
tolerate the use of UPNP to open needed ports, still admitting UPNP can be
pretty problematic. I think that making Tox always more efficient is important,
and that there is value in lightweight things, in any case. Still, this is an
open source project and, at least for now, there are no big server farms or
super-nodes around. I am not saying this is going to change in the future. I
think we should definitely help the network run smoothly: even in sight of the
fact that video and audio aren't as lightweight (e.g.: there are timing
constrains for example), and not everyone wanting to use it will be able to
establish direct connections.
This is, however, only my opinion: and for sure I am not an expert in any way
on these topics.

Thank you guys for your great work.

[mrkiko]

... it seems to me the usual tradeoff to meet between collective and individual interests. :D But ok, this is technology so might be I should reason differently based on conditions.

[LuccoJ]

@mrkiko There is no such thing as "UDP relaying" in Tox, only TCP relaying (unless the onion routing counts perhaps, but that is only used for DHT queries).

And yes, of course, "direct" connections aren't direct: any connection on the internet generally goes through a number of routers ("hops") that you don't control, and which could easily be tampered with, making using relays no less secure than not using them.
You're basically already using many relays, they're just called internet routers, and that's how the internet works.

At the same time, Tox relays are only used when necessary and it makes sense for it to stay that way.

[mrkiko]

Thank you.

[RIscRIpt]

I'm certain qTox will never have it on by default, and I'm certain I will be making people feel guilty if they have it on needlessly.

I just want to provide you network usage from last 30 days of my desktop PC.

I played that "Somegame" (online shooter) more often than used qTox, and still the network usage is almost twice as large. IMO it must be fixed...

[fcore117]

RIscRIpt: now that is really overkill, lets hope for more massive UDP optimizations.

[aaannndddyyy]

@LuccoJ: Problem to solve is that even with only one connection, you have to send pings to each of your online contacts through the relay and also through the relay do the dht lookups of offline contacts.
This would be trivial by simply copying the same ping on the relay if there were not the crypto part involved ....

[Green-Sky]

an "easy" way would be to make use of the "user status" by eg. disconnecting one self from the grid for several minutes or decreasing the ping rate ...

[ghost]

I am wondering that as far as "pinging" contacts for availability... does it take into account seeing traffic from that neughbour? I mean if that neighbour has just pinged me then I can avoid pinging it as I know it is alive. The other thing is if that neighbour is sending me a file, a message, has just called etc then we both know we are alive. Then could we have a greater period before the next ping. Because we know that we are alive and in use so we know that state is likely to continue for a while. Then after that time we could reduce that period to say 75% and so on up to a point say for an hour where we increase it to say 200% (of the default ping period) because the device is likely to be inactive as in the user is doing something else or gone to sleep, whatever. This of course needs to be mirrored on all clients so timeouts use the appropriate expected intervals.

Lastly do we even care for this contact is online behaviour? I mean if it is costing us so much data why bother? The phone does not care which contact is on or not until, that is, you make a call or send a text. Then you get a meaningful response from the network, busy, ring etc. This makes much more sense. Maybe there could be a field that shows last active time or the green circle slowly unwinds to grey but turns red should a ping timeout ocour or one is notified by the contact as going offline or busy etc.

And this is it, why arent the peers sending status messages instead? Like when I fire tox app it sends a hello I'm up state message to all my contacts and peers. They don't display anything but not the state. In that message is where I send expect a hello every 10 mins so that peer can set up the timeout at 3 or more times that value. When I exit the app the last thing it does is send goodbye going offline message to all my contacts and peers. This s how link state protocols work (bgp ospf etc) timeouts and hellos are just used as a backup. This is way more cpu and bandwidth efficient imho and does not leave peers waiting for a hello except to clean up their state database in case of an error, crash or dead battery.

What do you all think? Sending thousands up on thousands ie frequent pings is just mad just to display a green ring around a contact. They may be down for ages and everyone is pinging them for nothing. Yes ping irregularly ie every few minutes and then back off algorithmically to a max say 10 mins to discover whensome has come on line and the up message has been lost but that is all. Maybe this should just be done for frequently used contacts anyway. We all have contacts that we may use every blue moon why waste time pinging them for status, meh. The ones we use daily maybe have some value but maybe make that opional too and just do the mobile phone thing....on use only. And rely perhaps on status reports solely on mobile platforms.

[no149]

How's @ghost suggestion? Is it reasonable? I would like to help.