Two peers maintaining a single, mutual connection

[ppetr]

For my new VPN app I'm building using Iroh, I created a component for maintaining a single mutual connection (irrespective of who listens and who connects). Both peers set it up providing their own Endpoint and the other peer's EndpointId. The component tries to connect to the other peer, as well as receives incoming connections (listened for) from the other peer, until it succeed establishing a connection in either direction.

The component exposes this using 2 methods:

    /// Returns a current, established `Connection`, if any.  Very fast, non-blocking.
    pub fn connection(&self) -> Option<Connection>;

    /// Waits for a new connection to become available.
    /// `None` means that the previous connection has been closed and a new one hasn't been
    /// established yet.
    pub async fn next_connection(&mut self) -> Result<Option<Connection>, watch::error::RecvError>;

If somebody would find it useful, I can contribute this either as a PR somewhere, or a small, dedicated crate.

(Another layer then maintains a set of peers, with such a Connection stream to each of them.)

[matheus23]

Yeah this is tricky to get right. There exists an implementation of something similar in iroh-util, and another similar implementation deeply baked into iroh-gossip: https://github.com/n0-computer/iroh-gossip/blob/2ce78afe09d89d41d123f28eac19bdc831609cc8/src/net.rs#L792

The latter really is not a reusable component, but it shows how difficult this is to get right in practice.
Specifically, thinking around half-open connections where an endpoint crashed ungracefully, then reboots and attempts to re-connect. These cases need to be handled well by the other peer that might suddenly see two connections active at the same time.

Would love if someone figured out all of the rough edges around this. Perhaps iroh-util is the right place to collaborate on something like this :)

[lorenzleutgeb]

Sorry if I am stating the something obvious here, or missing your points, but... For Radicle, we face a similar challenge, where peers may both be configured to connect to each other (and stay connected). So, when they start and stop, connect and reconnect, this process is inherently racy. We use a very simple deterministic tie breaker: Whenever there are two connections between two peers (because both connected, and they raced so that suddenly we have two connections in our hands), view the public key as a tuple of bytes, and compare the public keys of the two endpoints. Whichever public key is lexicographically larger gets to keep the connection that they initiated. The other peer closes the connection that they initiated. Since the public keys are known to both and static, both peers will arrive at the decision on which connection to keep.

[ppetr]

@lorenzleutgeb Yes, I'm using the very same approach to make the choice consistent and deterministic by both peers 👍