Add Support for TUN-based Channels (kqueue/epoll)

[HeikoBornholdt]

Motivation

I want to develop a mesh VPN. Since I love netty, I would love to use it for this project. Therefore I need a TUN-based Channel implementation.

Modifications

• Added native methods to io.netty.channel.kqueue.BsdSocket and io.netty.channel.epoll.LinuxSocket, allowing the creation of TUN devices and setting/getting the network interface MTU on macOS and Linux.
• Implemented io.netty.channel.kqueue.KQueueTunChannel and io.netty.channel.epoll.EpollTunChannel, allowing the creation of TUN-based channels using epoll and kqueue.
• Defined and implemented io.netty.channel.socket.TunChannelConfig, allowing setting/getting the network interface MTU.
• Modified io.netty.channel.unix.Socket to avoid Socket#isIPv6(fd) call that is not supported on TUN sockets.
• Modified io.netty.channel.epoll.AbstractEpollChannel allowing to set AbstractEpollChannel#local in EpollTunChannel#doBind as io.netty.channel.kqueue.Native#epollCtlAdd must be called after TUN device has been bound.
• Renamed io.netty.channel.kqueue.AbstractKQueueDatagramChannel to io.netty.channel.kqueue.AbstractKQueueMessageChannel because KQueueTunChannel needs to implement this class but is not datagram related. As this rename may break the API, I re-added AbstractKQueueDatagramChannel for legacy reasons.
• Added io.netty.channel.socket.Tun4Packet and io.netty.channel.socket.Tun6Packet to help work with IPv4 and IPv6 packets.
• Added io.netty.example.tun.TunPingDevice example creating a TUN device that will reply to IPMC/IPv6-ICMP echo ping requests.
• Added io.netty.example.tun.TunEchoDevice example that echoes all received IP packets (e.g., suitable for performance tests).
• Added native dependencies to example/pom.xml (might require changes to your release workflow).
• Modified run-example.sh to activate platform-dependent native transport maven profile.

Result

Support for TUN-based Channels on kqueue- and epoll-enabled platforms.

Additional Nodes

If desired, I can also provide a TunChannel for Windows based on Wintun. I already prototyped it, but I'm not sure if the Wintun license is compatible with netty.

[hyperxpro]

This is something that I'd love to work on. I'll review in sometime and get back to you.

[hyperxpro]

First round of small changes

[hyperxpro]

@chrisvest Can you please trigger CI?

[hyperxpro]

I'll do benchmarking and testing today.

[hyperxpro]

Can we add test cases that test actual traffic flow because the current one only test packet level? Can we do an ICMP ping or a UDP echo?

[hyperxpro]

This works like charm. Successfully built a bridge-tunnel connection between 2 EC2.

But my main concern is regarding performance since we are only using 1 thread for handling all I/O. Is there any way?

[hyperxpro]

See https://docs.kernel.org/networking/tuntap.html#multiqueue-tuntap-interface

[HeikoBornholdt]

Can we add test cases that test actual traffic flow because the current one only test packet level? Can we do an ICMP ping or a UDP echo?

I will try to implement an ICMP ping test

[...]
But my main concern is regarding performance since we are only using 1 thread for handling all I/O. Is there any way?
See https://docs.kernel.org/networking/tuntap.html#multiqueue-tuntap-interface

The multiqueue feature looks interesting!
However, as netty allocates exactly one thread per channel, this is not compatible with my current approach.
Instead, we would need to switch to a ServerChannel and then create child channels for each thread. This might work.
What concerns me most is the absence of a packet discriminator: We cannot control which packet is read into which channel. Therefore, if we have a communication state, it has to be maintained outside the channel and needs to be synchronized between all channels.

[hyperxpro]

You can use the UDP SO_REUSEPORT approach. We set this socket option and bind on the same port multiple times but the Handler remains the same for all channels. Each time we bind, we use the next EventLoop on EventLoopGroup.

You can use multi-queue to accept multiple packet streams using different channels but on the same Handler.

[HeikoBornholdt]

I followed your advice and added EpollTunChannelOption.IFF_MULTI_QUEUE. If set to true, one can create multiple TunChannels assigned to the same TUN device. I also expanded the TunEchoDevice example so one can define the number of channels to be created.

Earlier, you suggested writing some tests: Have you thought of a fully automated test (e.g., JUnit) or a new testsuite Maven module to help with manually testing?

[hyperxpro]

Looks good but we need more test cases to cover all functionality.

[hyperxpro]

We don't need an extra module I guess.

[HeikoBornholdt]

I am thinking on an integration test creating a TunChannel with Ping4Handler added to it.
In addition, we need a DatagramChannel sending an ICMP Ping Request and waiting for a response.
This must be made for both epoll and kqueue. The kqueue test would require us to create a new macOS-based GitHub Workflows.

Is this in line with your expectation?

[hyperxpro]

Sounds good. Also, a little bit of TCP tunneling tests will be good too.

[hyperxpro]

@normanmaurer Is it fine if we create a workflow for testing this on macOS?

[HeikoBornholdt]

• Added check if IFF_MULTI_QUEUE is available when building netty-transport-native-epoll.
• Added io.netty.channel.epoll.Native.IS_SUPPORTING_MULTI_QUEUE, allowing to check if IFF_MULTI_QUEUE feature is available.
• Added tests for epoll and kqueue (currently, only UDP-based traffic is processed. I don't see any advantages of adding a TCP-based test as well, as it makes no difference for the TUN device. But if you insist, I can add it).
• Adjusted docker/docker-compose.yaml to make TUN tests work in CI.

[hyperxpro]

@chrisvest Please add this to your TO-DO :)

[chrisvest]

This seems a bit exotic, so I'm wondering if we can get by with the minimum JNI changes in core Netty, and have the rest of the implementation in a contrib repository? It's also a big change in general, which makes me wonder if Netty 5 is a better target (fewer versions to maintain; would have to be ported anyway).

[hyperxpro]

Sounds good.

@chrisvest If you can review the code then I can create a contrib repository and begin working with @HeikoBornholdt on porting this.

[HeikoBornholdt]

I would love to see TUN support added to Netty 4.1.
Porting this feature to Netty 5 is already something I want to address next.

[hyperxpro]

I'll be closely maintaining this module with @HeikoBornholdt if merged in 4.1. You can rely on us for maintenance. @chrisvest

[KevinRoebert]

I'll be closely maintaining this module with @HeikoBornholdt if merged in 4.1. You can rely on us for maintenance. @chrisvest

+1 As a co-author, I am also willing to maintain the module.

[hyperxpro]

@normanmaurer @chrisvest Can we do this folks?

[AriseFX]

I would like to know the progress of this thing as I happen to have a need to develop a TUN interface :)

[hyperxpro]

@chrisvest Gentle ping... :)

[HeikoBornholdt]

Moin @normanmaurer 👋,

I remember your delight about similar PR https://twitter.com/normanmaurer/status/830336283683721217.
Any chance you could take a look at this PR?

[KevinRoebert]

@normanmaurer @hyperxpro Is there any update on the review? Anything we can help with?

[normanmaurer]

Unfortunately I dont feel we have the resources at the moment to maintain this and so put it in the main project. The best bet for now would be to either make it part of the Netty-incubator or make it a separate repository under your own username.

[HeikoBornholdt]

Thank you for replying, I understand that.
I prefer to make it part of Netty-incubator.

Would it be acceptable to put the necessary C methods (see changes in netty_epoll_linuxsocket.c, netty_epoll_native.c, netty_kqueue_bsdsocket.c) into the main project?
This should minimize maintenance for you and would help us a lot in providing this feature through Netty-incubator (or any other separate project).
If not, we must build a (parallel) system for integrating native transports.

[KevinRoebert]

@normanmaurer @hyperxpro Do we get the okay for that?

[hyperxpro]

We are having an internal discussion on this. Will get back soon with an update.

[hyperxpro]

There you go folks: https://github.com/netty-contrib/transport-tun

Please do a PR and tag everyone for review.