[SPIKE] p2p communication channel

[zolia]

Motivation

There are many cases when we are / will be doing peer-to-peer communication after establishing session between two peers: payment promises, session keep-alives, session disconnects (after tunnel setup, broker may become unreachable), additional monitoring and performance testing services.

Such direct communication also leads to better decentralization. Also this will not increase broker load.

Implementation

Depending on what service is used, communication channel may need additional message encryption. For openvpn and wireguard services we don't need additional encryption.

AC: Messages that are p2p after session is established should be reworked to use such direct communication.

One way to implement such p2p communication is to use plain tcp session with protobuf. This way we reduce overhead introduced by http protocol headers and improve speed.

[anjmao]

After more thinking I would first suggest that we try to implement it with gRPC which already uses Protobuf. gRPC uses HTTP/2 with compressed headers and supports needed features like auth, retries out of the box. Also If we use gRPC we can start using it to call services like Accountant and Quality Oracle.

[zolia]

Requirements

• small footprint, easy to extend and maintain. Easy to add new messages to existing communication channel
• protocol should be lightweight
• messages should be easily accessible from different different node modules ( packages )
• ensure minimal backwards compatibility with already existing messages (such as payments pingpong)
• eventually replace all p2p messages currently on a broker with direct p2p. These can be separate tasks.

gRPC over HTTP/2 looks to be viable option @anjmao suggested. I would go for it. Any other suggestions?

[zolia]

One major issue I see with gRPC over HTTP/2 its that it is TCP only. Meaning it will be ok for our tunnel based services, but will not work for services that are not tunnel based.

My suggestion is to start looking at such transports as QUIC and/or HTTP/3. Currently grpc-go does not support QUIC, but there is such project: https://github.com/lucas-clemente/quic-go

Any1 wants to experiment with that and see if we would be able to hook gRPC over quick?

Reminder why we need UDP: its we only way to have a decent nat hole punching, thus services behind the nat.

[anjmao]

Yes, I looked more at gRPC too, and it looks that for our case it may not fit as it fit mostly client / server . We need something similar to https://github.com/perlin-network/noise which also know how to do other peers discovery by knowing only one peer (in case of multihop?)

https://github.com/lucas-clemente/quic-go Looks interesting, but need to check how stable it as as development is in early stage.

@zolia There is issue for gRPC over UDP grpc/grpc-go#241. Work is being done but nothing working yet. Could we use TCP for p2p messages and UDP proxy? I imagine we could adjust our UDP proxy to support transferring UDP packets TCP quite easy.

[zolia]

I imagine we could adjust our UDP proxy to support transferring UDP packets TCP quite easy.

Packets yes, but not tcp establishment itself. UDP proxy sets up "UDP session" which is not full blown tunnel by itself. You cannot simply establish TCP through it. That requires L3 tunnel.

One way for us is to alway establish tunnel (maybe WG, maybe GRE without encryption), but that would limit portability and flexibility.

With quic or any other udp based protocol, we can pre-punch UDP hole and setup that session on such hole.

UPDATE:
In theory, you could establish TCP listener on localhost, read packet payload from there, send them through our "punched UDP session" and read them back on other side and push to pre-established tcp session with another gRPC peer. But that is a hack and i believe performance of such setup would be extremely bad.

[anjmao]

Ok so there are two cases.

1. p2e communication on established VPN tunnel. Is such case we can communicate via private network using any protocol TCP or UDP.
2. p2p communication on UDP hole for services like SOCKS5 when we we don't have tunnel. Will required UDP.

So it looks like quic is the most promising which will work in both cases.

[chompomonim]

Done in #1797