Using raw sockets and kernel bypassing to improve performance

[ccaapton]

There are a lot kernel bypassing work in linux network stack, like packet_mmap, dpdk or the recent ebpf/xdp framework. I'm wondering if tinc could take advantage of these tricks, or even use raw sockets instead of the conventional udp+tuntap interface to reduce the kernel overhead?

Wireguard has drawn a lot of attention recently, but I think it is not much superior over tinc other than it seats in the kernel, while only handle a small part of tinc's job. Also, raw socket is not linux-specific, so if tinc could use raw sockets, it could be a state of the art high performance cross platform solution.

[gsliepen]

Tinc already supports raw sockets: set DeviceType = raw_socket in tinc.conf. However, raw sockets are no replacement for tun: packets received and sent using a raw socket will not be seen by the kernel's own IP stack. That means the host that tinc is running on will itself not be able to participate in the VPN. I think DPDK suffers from the same problem.

As for UDP, sendmmsg() and recvmmsg() will reduce system call overhead by processing many packets in one go. However, the tun device does not expose any kind of equivalent functionality to user space, so packets to and from a tun interface still have a significant system call overhead.

The easiest way to improve performance is to make tinc multi-threaded, but that's easier said than done.

[ccaapton]

What about bridging tinc to the host with something like openvswith, or macvtap?

[gsliepen]

I'm not sure that is going to work. The problem is that a raw socket will capture packets coming in to the device and will send them back out to the device, without those packets ever being processed any further by the kernel. In contrast, a tun device will capture anything coming from the kernel and will send packets back to the kernel. But if you are interested in it, just go ahead and try it out. If it works and is usable, then it might be worth spending time implementing PACKET_MMAP.

[ccaapton]

After some thought and experiment, I think the following should work(only linux), as it makes tinc behaves as a vm side by side with the host:

1. setup a separate network namespace.
2. create a veth pair VethA and VethB. VethA connects the main network ns, and VethB connects the new namespace.
3. Add address 10.3.0.1/24 to VethA, and 10.3.0.2/24 to VethB
4. Setup iptables source nat or masquerade for VethA, and allow ip_forward, so the new namespace could communicate with outside world.
5. tinc open raw socket on VethB. Now tinc could inject/recv packets to/from the rest of the system by sending packets to 10.3.0.1

From my test, veth interfaces are quite fast, and transfering data across network namespace does not need user/kernel context switch. What do you think?

[gsliepen]

I tried this, but it doesn't work unfortunately.

[olibrius]

Did you have a look at KNI in DPDK?
I was interested to understand if it could be a way to inteface network kernel interface with DPDK and improve tinc performances that way....
Are you interested in exploring further ?

Olivier

[splitice]

#110 appears to have a possibly workable solution to write() syscall perforamcne that doesnt require DPDK.

Raw sockets (and hence sendmmsg & writev) are not the solution, they are on the wrong "side" of the tun/tap unfortunately.

[gsliepen]

I'm closing this issue, as raw sockets are not a way to improve tinc. However, syscall overhead is quite big in tinc, so if there are other ways to solve this issue, another ticket should be made. And there's also #275 which tries to address this.