Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use of QUIC for transport #5377

Closed
Avamander opened this issue Dec 15, 2018 · 31 comments

Comments

Projects
None yet
9 participants
@Avamander
Copy link

commented Dec 15, 2018

From the last time a different (transport) protocol was discussed a few years have passed, things (protocols) have moved along. So, it would be really nice if Syncthing could use QUIC to transport files for both speed, transport protocol redundancy, latency and possible security benefits. Actually QUIC has a few other nice features that could be used but just having QUIC would suffice for now.

There are a few nice libraries available for QUIC for Go so it shouldn't be incredibly hard to implement.

@imsodin imsodin added the enhancement label Dec 16, 2018

@jeroen7s

This comment has been minimized.

Copy link

commented Jan 3, 2019

why not migrate to µTP
would add a lot of benefits for performance, scalability and reliability, as demonstrated by the bittorrent network

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

Because all Go implementations of it suck, and it's widely recognised as a protocol and blocked by ISPs.

@jeroen7s

This comment has been minimized.

Copy link

commented Jan 3, 2019

Is blocking the torrent protocol still a thing?

@jeroen7s

This comment has been minimized.

Copy link

commented Jan 3, 2019

also most torrent clients bypass blocking easily using TLS encryption to hide the torrent traffic
so not sure if it's even blocked effectively anywhere

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

TLS can only be within UTP, as TLS requires reliable delivery and reliably delivery only happens once you are using UTP, so UTP is still visible and is blocked in many cases.

@pipboy96

This comment has been minimized.

Copy link

commented Feb 4, 2019

Another option may be Noise. It is used by Wireguard, WhatsApp, Bitcoin Lightning Network and I2P.

@pipboy96

This comment has been minimized.

Copy link

commented Feb 4, 2019

@jeroen7s Yes, torrents are often blocked on mobile networks.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

Noise uses tcp, so not sure how this is relevant.

@pipboy96

This comment has been minimized.

Copy link

commented Feb 4, 2019

@AudriusButkevicius Wireguard uses Noise over UDP.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

Sure, because it's probably packet oriented and cares about encryption of the individual packets and handles loss fine. We need a lossless protocol, and noise is just a framework on top of some protocol.

@Lillecarl

This comment has been minimized.

Copy link

commented Feb 6, 2019

Isn't QUIC and KCP basically doing the same thing? Implementing reliable transport over UDP?

@pipboy96

This comment has been minimized.

Copy link

commented Feb 6, 2019

@Lillecarl KCP is a protocol created to deal with Chinese internet issues (traffic congestion and censorship)

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

Kcp also has no state, such as handshakes etc, all of which has to be implemented on top making it hard to use and mostly unreliable.

@Lillecarl

This comment has been minimized.

Copy link

commented Feb 12, 2019

@AudriusButkevicius doesn't QUIC align quite well with syncthing then?

@calmh

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

It does, in theory at least, which is why this ticket exists and is open.

People keep suggesting alternatives to QUIC, each of which have their advantages/disadvantages, some of which we've tried and failed to use, some of which don't have implementations we can use. What remains to do is for someone to plug in QUIC and get it to work.

I tried that the other day, and as of currently I couldn't get the client cert bit to work in the current version of the QUIC implementation I tried.

So, please, don't argue about other protocols in this issue.

@Lillecarl

This comment has been minimized.

Copy link

commented Feb 12, 2019

I read up on it a bit, and it seems like QUIC and STUN isn't completely thought through yet, which would have implications for the p2p nature of syncthing, can't wait for v6 to be mainstreamed.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

You can work around that as stun packets can clearly be identified.

@dpsenner

This comment has been minimized.

Copy link

commented Feb 16, 2019

I recently worked a lot with grpc at my dayjob. I implemented a quick and dirty test and I was easily able to transfer >400Mbyte/s without much delve into performance improvements like reducing garbage collection. The implementation is available over here. grpc is under the hoods a tcp connection with http2 protocol and payloads are transported as protobuf messages.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 16, 2019

We don't use GRPC, nor do we use http2 so I am not sure how this is relevant. We are talking about the transport here, not the protocol, so comparisons of protocols are irrelevant.

@dpsenner

This comment has been minimized.

Copy link

commented Feb 16, 2019

I didn't want to intrude this discussion but since you are discussing quic as a transport, my suggestion is to evaluate grpc as the transport as well. Since we've adopted grpc and protobuf we haven't looked back at the old days when we tried to reinvent the wheel with udp based transports. Been there, got no t-shirt. We learned that udp should only be considered if latency matters.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 16, 2019

grpc is not a transport, it's a protocol/framework over any reliable transport that ships bytes from A to B.

Edit:

https://godoc.org/google.golang.org/grpc#WithDialer
https://godoc.org/google.golang.org/grpc#Server.Serve

You just need to implement net.Conn and net.Listener

@dpsenner

This comment has been minimized.

Copy link

commented Feb 17, 2019

Your response is a little bit unclear to me, and possibly others. Therefore it would be great if you would expand on the following question:

Do you think that grpc could serve as a transport for syncthing? Furthermore, am I reading your comment correctly in that you think that it could be easily adopted by implementing the two functions mentioned in your previous comment?

PS: I have no intent to start a philosophical discussion about what is a transport and what is not. I write this just so that we can agree on the meaning of the word "transport". In my perception, anything that transports bytes from A to B can serve as a transport (to syncthing). If that "transport" is an rpc framework it is actually just an implementation detail of that "transport". (period)

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 17, 2019

This ticket is about transports, GRPC is not a transport, it's a rpc framework that has pluggable transports, as I've explained above.

Sure, we could use GRPC, XMLRPC, JSONRPC whatever we want as the RPC protocol, but this is not what this ticket is about. This ticket is about transports not protocols, your comment series is a bit unclear, if you are suggesting we move to GRPC, there is no reason to do that as we already have something that works (which is protobuf over wire) and GRPC in itself would bring nothing new to the table.

@dpsenner

This comment has been minimized.

Copy link

commented Feb 18, 2019

Today I am a user of syncthing and I use it also on my android 8+ phone. But I can't use it to sync my sd card bidirectionally because go can't request access to the sd card since it is no native java application. Far in the future I do hope that a java native implementation of the syncthing protocol emerges and I see grpc as way to achieve this goal more quickly because it provides a typesafe api to communicate between one syncthing node and another. Over that api, a syncthing node could stream files or exchange information about other nodes in the same cluster. If that does not fit into the vision, I am sorry and apologize for taking your time.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 18, 2019

Sorry, what does this have to do with this issue where we are talking about adding quic transport?

First you hijack the issue talking about grpc which is mostly irrelevant to this issue, then you start talking about problems on android.

I don't see how any of this is meaningful comments to the in the context of this ticket
which is implementing quic, NOT replacing protocol with GRPC, and NOT rewriting android app.

I understand you are unhappy and want to tell people about it, but doing so on a random issue with talks about transport is not the right place.

Just stop.

@dpsenner

This comment has been minimized.

Copy link

commented Feb 18, 2019

I had hoped to broaden the horizon with an rpc framework that builds on top of the already implemented protobuf messages, allowing me to use that and implement a native android app in the near future, but it looks like I misunderstood what this issue is about and apologize.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Feb 18, 2019

This ticket is about using a udp based transport to perform NAT punch-through to avoid having to use relays, not fixing android sd card issues. You can already implement an android app of you follow the protocol spec. It uses protobuf over tcp mostly. Syncthing-lite already implements the protocol in java/kotlin.

@bt90

This comment has been minimized.

Copy link

commented Feb 26, 2019

Seems like http-over-quic is going to be HTTP/3: https://www.zdnet.com/article/http-over-quic-to-be-renamed-http3/
I think we might see QUIC support in the standard lib because of this.

@calmh calmh closed this in e714df0 May 29, 2019

@calmh calmh added this to the v1.2.0 milestone May 29, 2019

calmh added a commit to calmh/syncthing that referenced this issue May 29, 2019

Merge branch 'master' into crashrep
* master:
  lib/config: Refactor migrations a bit (syncthing#5750)
  lib/connections: Add QUIC protocol support (fixes syncthing#5377) (syncthing#5737)
  github: Create SECURITY.md (syncthing#5749)
  gui, man, authors: Update docs, translations, and contributors
  golang-ci: Disable confused scopelint check
  lib/protocol: Don't send anything else before cluster config (syncthing#5741)
  lib/model: Unflake folder restart with blocking conn test (ref syncthing#5707) (syncthing#5744)
  lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes syncthing#5697) (syncthing#5734)
  lib/protocol: Don't call receiver after calling Closed (fixes syncthing#4170) (syncthing#5742)
  lib/model: Readd special handling of conn close in TestIssue5063 (syncthing#5743)
  lib/model: Close connections when model is stopped (syncthing#5733)
  lib/protocol: Test for Close on blocking send deadlock (ref syncthing#5442) (syncthing#5732)
@pipboy96

This comment has been minimized.

Copy link

commented Jun 4, 2019

@Avamander

This comment has been minimized.

Copy link
Author

commented Jul 9, 2019

How would one go about and specify that a remote device can speak QUIC? Is it quic:// or udp://?

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Jul 9, 2019

quic://

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.