Program routes and policy rules using netlink, not iptables binary

[danderson]

The linux router currently programs the linux network stack with ip and iptables. iptables is unavoidable because the corresponding netlink APIs are not safe for direct use. However, everything we do with ip we could just do directly over a netlink socket.

This would in particular help embedded systems that ship with the Busybox version of ip, which is stripped down and doesn't understand advanced commands like policy routing rules, which Tailscale requires.

[stapelberg]

iptables is unavoidable because the corresponding netlink APIs are not safe for direct use.

Can you expand on this? Which APIs do you need, and in which way are they unsafe?

I was about to open an issue suggesting to use https://github.com/google/nftables/ where available to avoid the iptables dependency. This would help make tailscale work on https://gokrazy.org/, where only Go is available (no C userland, i.e. no iptables).

[danderson]

For one, we can't depend on nftables, because we need to support OSes that haven't switched to nftables under the hood (e.g. pre-Buster debian), and we have to be cross-compatible with other software that doesn't support nftables (e.g. Docker, Kubernetes). The main way we can do that safely is to continue using the iptables binary, which will do "the right thing" based on OS/administrator policy.

And the legacy iptables is unsafe to use over netlink, because the API does replacement of the entire firewall at once, and doesn't have a way of preventing concurrent writes from squashing each other. The userspace tools work around this (poorly) with file-based locks, which already goes wrong in situations like containers.

If we can (a) positively detect that nftables, and only nftables is in use on the system; (b) we can manipulate the "legacy iptables converted to nftables" rules in nftables without conflicting with iptables itself... Then we could possibly make that work. But it's a bunch of work that, frankly, only makes sense for something like gokrazy, and not really for any other OS. So, that's going to be pretty low priority from my POV.

[stapelberg]

Thanks for the additional details, I agree with your view point.

I expect that using nftables directly might become more attractive in a few years as more and more installations run on the legacy-converted-to-nftables model.

Would an nftables fallback code path be something that you would accept if people contributed it, or is there a lot of churn and hence maintenance burden in that part of the tailscale code base?

[danderson]

I would love to not depend on iptables where possible. The part I'm unsure about is whether we can reliably detect that yes, it's definitely safe to use nftables directly without breaking the system.

If we can figure out how to make that part work reliably, I'd be very happy to have logic that uses nftables directly where possible, and falls back to shelling out to iptables otherwise.

There is definitely a maintenance burden, and given that we support distros like RHEL that take nearly a decade to drop support, I don't think iptables is going anywhere any time soon, sadly. I still think it's worth doing, if you want to open that particular can of worms :). Otherwise, I think we'll get to it "someday", but once we get the iptables logic stable it's going to be low priority.

[bradfitz]

Before I lose this, adding @stapelberg's pointers from https://twitter.com/zekjur/status/1427356540072255501:

https://github.com/rtr7/router7/blob/5869922efbe39c4911b18cb417f9e266458be879/internal/netconfig/netconfig.go#L301

Which uses https://pkg.go.dev/github.com/vishvananda/netlink