Garnet, on io_uring with kTLS

[mgravell]

So; entirely for fun, I've been playing with a general purpose high performance network layer for .NET, SocketSet - it does not use managed sockets, instead using the raw kernel APIs (ok, with managed sockets as the final fallback for compatibility reasons). Since Garnet has a nice pluggable network layer (kudos for that, btw), I thought I'd have some fun with Garnet!

Currently implemented in SocketSet:

• Linux
  • io_uring
  • epoll
  • both with opensll with full elective kTLS handoff
  • tcp, uds (file and abstract)
  • server and client
• Windows
  • iocp
  • rio
  • both with sspi
  • tcp, uds (non-abstract, because Windows; rio is tcp only)
  • server and client

Technically the design also naturally fits DPDK, but I have not yet explored that (frankly, the "good" hardware for that is far outside my budget), however; key findings so far are that on Linux it is reliably either the same or better than the default managed sockets implementation. I do not have the correct lab setup to properly demonstrate kTLS, which requires at least multiple boxes with fast links, and ideally kTLS-enabled enterprise NICs - with that setup, kTLS should fly (server-to-client, at least - kTLS seems happier with Tx than Rx).

Status

super experimental at the moment - I mostly wanted to put it on your radar as a discussion piece; see https://github.com/mgravell/SocketSet/tree/main/GarnetDemo

Example results

Abstract UDS (measuring raw IO perf):

cell	SAEA (req/s · p99 ms)	SocketSet/io_uring (req/s · p99 ms)	Δ	verdict
-P 1 GET	777,605 · p99 0.143	799,726 · p99 0.111	+2.8%	overlapping
-P 1 SET	717,875 · p99 0.247	799,726 · p99 0.103	+11.4%	DISJOINT
-P 16 GET	9,227,930 · p99 0.239	9,227,930 · p99 0.167	0.0%	IDENTICAL values — generator ceiling (its UDS ceiling is 9.2M vs 7.2M over TCP)
-P 16 SET	4,443,128 · p99 0.807	4,997,502 · p99 0.319	+12.5%	DISJOINT

TLS comparison, without benefitting from kTLS awesomesauce:

cell	SslStream (req/s · p99 ms)	in-transport OpenSSL (req/s · p99 ms)	Δ req/s	Δ p99
-P 1 GET	397,772 · 0.439	444,247 · 0.247	+11.7% DISJOINT	−44%
-P 1 SET	377,858 · 0.663	443,115 · 0.239	+17.3% DISJOINT	−64%
-P 16 GET	5,213,764 · 0.687	5,707,762 · 0.319	+9.5% DISJOINT	−54%
-P 16 SET	3,557,453 · 0.967	4,417,937 · 0.583	+24.2% DISJOINT	−40%

---

And the best bit: at least at the current time, there's no "Garnet needs to take a dependency" here - all the fun is can be achieved via a separate layer like SocketSet.Garnet - although it would need folks to self host rather than using the oh-so-convenient garnet-server global tool.

---

One small upstream UDS note: Garnet's embedding path (GarnetServer with servers == null) requires UnixSocketPath and unconditionally calls File.Delete before binding, which can never wok with abstract names because of the leading NUL. We hacked around it, but it would be super nice if we could just use UnixDomainSocketEndPoint("\0name") and have it do the right thing re chmod and delete. Bonus points if the wider piece also allowed socat/systemd conventions, i.e. @name gets interpreted as \0name, at least on Linux. I'm proposing the same thing over on redis/redis.

[badrishc]

Really nice exploration, thank you Marc! We were hopeful of reaching this at some point (there are internal eRPC and MsQuic stacks we have experimented with Garnet in the past). This is the closest to something that could actually ship cleanly.