Repository navigation
A NetBird-compatible relay server in Rust, written from a protocol description: why, and what we measured #8624
Replies: 3 comments
|
Hello @dlf-dds this looks very promising. We will comeback soon with a response to your questions. |
|
Hello @dlf-dds, First of all, nice job! It would be interesting to see the comparison on the same machines under the same conditions. I am curious where the differences come from: kTLS, WS + SSL, QUIC, or somewhere else. If you continue your analysis, I’d love to see the results. 1. Is the relay protocol something you would document and version upstream? We would contribute our description as a starting point, and our conformance fixtures, under whatever license you need. I usually prefer keeping the code self-explanatory over maintaining separate docs, but I see the value of a wire-protocol description for independent implementations. A PR with your description is welcome, and we can discuss there how to keep it in sync with protocol changes. 2. Would you consider counting inbound data frames toward liveness in the Go relay? We can open a pull request with the change and the test that reproduces the disconnect. Yes, that makes sense. Please open the PR with the change and the reproducing test. The clock-skew one is also nice, but in the near future we will make changes in this part of the code. If you optimise for speed, maybe this branch is interesting for you. It is an experimental development. As you have probably realised, the Relay itself is protocol agnostic. It does not necessarily need to encapsulate any upper protocol. Based on this property, I tried to exchange the packets with XDP eBPF. Basically, without an exchange between user space and kernel space, and without going through the network stack in the kernel. |
|
Thanks @pappz, that is a clear answer on both. We will open the two pull requests: the wire-protocol description as a document you can keep in sync with protocol changes, and the liveness change with the test that reproduces the disconnect. Understood on the clock-skew part. On the comparison: agreed that the numbers mean more on the same machines under the same conditions. We will rerun both relays side by side and post the results here, split by transport so the kTLS, WebSocket over TLS and QUIC cases are separate. Thanks for the pointer to the XDP branch. We will read it. |
Uh oh!
There was an error while loading. Please reload this page.
goat is an open-source data fabric from the U.S. Army (USAREUR-AF) whose mesh is self-hosted NetBird: code, docs.
We wrote a relay server in Rust that speaks the NetBird relay protocol over WebSocket and QUIC. NetBird clients connect to it without changes. This post says why we wrote it, how, what we measured, and what is not finished. It follows up issue #6021.
Why
Many of our peers sit behind carrier-grade NAT, so a large share of traffic is relayed. In April we measured the Go relay on a 2-vCPU burstable instance: 25 to 30 Mbps for one flow, less in total with 4 or 16 flows, and
strace -cunder load showed 76% of the relay's time infutexandnanosleepand 24% in network I/O. SettingGOMAXPROCS=1raised one flow to 36.6 Mbps. Others in #6021 later measured 200 to 230 Mbps for one flow on a dedicated 3-vCPU host with the relay at 200% CPU, so the absolute ceiling we first reported was a property of our small instances. The CPU cost per Mbps was not.Tuning I/O system calls inside the Go relay would act on the 24%. We wanted to remove the 76%, and that meant a different runtime.
How it was written
We did not port the Go code. We read the upstream source, wrote the protocol down in our own words (framing, message types, handshake, routing decisions, keepalive), and implemented the Rust server against that document. The message definitions under
shared/relay/are BSD-3-Clause; the server underrelay/is AGPL-3.0 and was read only for behavior. Each Rust module's doc comment names the upstream files that were read. The result is Apache-2.0.docs/design/netbird-relay-wire-protocol.md, written against upstream commit53b04e512.goat-relay-protocol/. A conformance tool parses each fixture frame, serializes it again and requires identical bytes.goat-relay/.Design in brief: one single-threaded tokio runtime per pinned core, with no work stealing;
SO_REUSEPORTso the kernel spreads connections across the per-core shards; rustls with the TLS record layer handed to the kernel (kTLS) on the WebSocket path; quinn for QUIC; tokio-tungstenite for WebSocket.What we measured
On one
c7i.2xlarge(8 vCPU), relay in a container, QUIC transport:One instance placement; runs of the same cell varied by 5 to 10%.
These figures are not a same-host comparison with the Go relay. Our Go measurements were on a 2-vCPU burstable instance, and we have not published a run of both relays on the same host.
Where it differs from the upstream relay on purpose
Status
goat-relay/v0.1.0, imageghcr.io/usareur-af/goat-relay:v0.1.0for amd64 and arm64. An image is also built from each commit to main, and our own deployments pin an image by digest.tlskernel module loaded and host networking. Without both, the relay falls back to TLS in user space and still works.Not finished
Questions
The code is Apache-2.0. If you run a self-hosted relay and want to try it, the Ansible role is
ops/ansible/roles/goat-relay/, and we would like to hear what breaks.All reactions