feat: Ephemeral peer channels

[Septias]

TLDR: Webxdc API to send ephemeral messages with iroh.

Desktop PR: #3741
Documentation PR: #78
Webxdc Hello PR: #59
Webxdc types PR: #6
Android PR: ?
IOS PR: ?

Gossip Tests: https://github.com/Septias/gossip_tests
Mutliplayer Pong: https://github.com/Septias/multiplayer-pong

Supersedes: #5041

TODO

• How do we handle chat removals?
• Expose gossip APIs in cffi
• polish pong example

WebXDC Realtime API

See Webxdc specs for more info.

Iroh

DeltaChat wants to use the ephemeral-channel provider Iroh, which attempts to establish direct p2p connections between peers (QUIC), using PlumTree as a gossiping protocol for efficient peer sampling. Iroh falls back to relay nodes when a direct connection can not be established. Even a relayed connection is faster than sending single WebXDC messages via mail.

Implementation

Ephemeral channels should be established lazily, to avoid bootstrapping p2p connectivity when it's not required. By asking application developers to explicitly subscribe to ephemeral messages, we know when this should happen.

In any case, we need a way to agree upon a gossip topic ID among all (future) participants, this topic ID should never be guessable or readable to any outsider.

1. We introduce a new GossipTopic message header with a random 32-byte TopicId, securely generated on the initial webxdc sender’s device. This message header is encrypted and sent in the same message as the webxdc application.
2. Whenever joinRealtimeChannel().setListener() or joinRealtimeChannel().send() is called by the webxdc application, we start a routine to establish p2p connectivity and join the gossip swarm with Iroh.
3. The first step of this routine is to introduce yourself with a regular message containing the IrohPublicKey. This message contains the users relay-server and public key. Direct IP address are not included as this information can be persisted by email providers.
4. After the announcement, the sending peer joins the gossip swarm with an empty list of peer IDs (as they don’t know anyone yet).
5. Upon receiving an announcement message, other peers store the sender’s NodeAddr in the database (scoped per WebXDC app instance/message-id). The other peers can then join the gossip with joinRealtimeChannel().setListener() and joinRealtimeChannel().send() just like the other peers.

Testing

The application can be tested using the sk/ephemeral_peer_channels branch on the desktop and core. To setup local core on desktop, see here.

You then need two accounts that know each other and a testing xdc. One option is gossip test which can be downloaded from this repository. Another option is the multiplayer pong game located here.

[link2xt]

Unrelated changes

[Septias]

same as #5548.
Should vanish when we merge the PR and the merge main.

[link2xt]

Let's merge it, but it seems we need to move all the tests from Rust to Python. Current Rust tests only work because of hardcoded relay, and we don't actually want any hardcoded relay, only the one which comes from metadata. iroh.testrun.org is not supported to be running forever.

[link2xt]

Also there is still no way to disable it.

[dignifiedquire]

🎉

[Hocuri]

This test is timing out in #5619 in Rust CI / Rust tests (ubuntu-latest, 1.77.0)