Program routes and policy rules using netlink, not iptables binary

[danderson]

The linux router currently programs the linux network stack with ip and iptables. iptables is unavoidable because the corresponding netlink APIs are not safe for direct use. However, everything we do with ip we could just do directly over a netlink socket.

This would in particular help embedded systems that ship with the Busybox version of ip, which is stripped down and doesn't understand advanced commands like policy routing rules, which Tailscale requires.

[stapelberg]

iptables is unavoidable because the corresponding netlink APIs are not safe for direct use.

Can you expand on this? Which APIs do you need, and in which way are they unsafe?

I was about to open an issue suggesting to use https://github.com/google/nftables/ where available to avoid the iptables dependency. This would help make tailscale work on https://gokrazy.org/, where only Go is available (no C userland, i.e. no iptables).

[danderson]

For one, we can't depend on nftables, because we need to support OSes that haven't switched to nftables under the hood (e.g. pre-Buster debian), and we have to be cross-compatible with other software that doesn't support nftables (e.g. Docker, Kubernetes). The main way we can do that safely is to continue using the iptables binary, which will do "the right thing" based on OS/administrator policy.

And the legacy iptables is unsafe to use over netlink, because the API does replacement of the entire firewall at once, and doesn't have a way of preventing concurrent writes from squashing each other. The userspace tools work around this (poorly) with file-based locks, which already goes wrong in situations like containers.

If we can (a) positively detect that nftables, and only nftables is in use on the system; (b) we can manipulate the "legacy iptables converted to nftables" rules in nftables without conflicting with iptables itself... Then we could possibly make that work. But it's a bunch of work that, frankly, only makes sense for something like gokrazy, and not really for any other OS. So, that's going to be pretty low priority from my POV.

[stapelberg]

Thanks for the additional details, I agree with your view point.

I expect that using nftables directly might become more attractive in a few years as more and more installations run on the legacy-converted-to-nftables model.

Would an nftables fallback code path be something that you would accept if people contributed it, or is there a lot of churn and hence maintenance burden in that part of the tailscale code base?

[danderson]

I would love to not depend on iptables where possible. The part I'm unsure about is whether we can reliably detect that yes, it's definitely safe to use nftables directly without breaking the system.

If we can figure out how to make that part work reliably, I'd be very happy to have logic that uses nftables directly where possible, and falls back to shelling out to iptables otherwise.

There is definitely a maintenance burden, and given that we support distros like RHEL that take nearly a decade to drop support, I don't think iptables is going anywhere any time soon, sadly. I still think it's worth doing, if you want to open that particular can of worms :). Otherwise, I think we'll get to it "someday", but once we get the iptables logic stable it's going to be low priority.

[bradfitz]

Before I lose this, adding @stapelberg's pointers from https://twitter.com/zekjur/status/1427356540072255501:

https://github.com/rtr7/router7/blob/5869922efbe39c4911b18cb417f9e266458be879/internal/netconfig/netconfig.go#L301

Which uses https://pkg.go.dev/github.com/vishvananda/netlink

[bradfitz]

I sent vishvananda/netlink#711 to add the support for the unreachable rule we use. After that we're down to "just" iptables/nftables. 🥲

[raggi]

An update on status:

v1.48.1 has been released and will default to iptables again.

You can test out the detection logic which will pick between nftables and iptables based on usage, by setting: TS_DEBUG_FIREWALL_MODE=auto for example in /etc/default/tailscaled. You can also explicitly force the mode with TS_DEBUG_FIREWALL_MODE=nftables or TS_DEBUG_FIREWALL_MODE=iptables.

There is additionally a fix in 1.48.1 that resolves the conflict with ufw that lead to forwarding traffic (subnet routers and exit nodes) from functioning properly under nftables mode. We would appreciate wider testing of the nftables mode to ensure broad coverage, as the integration requires coordination with conventions that are not standards in the platform. As of 1.48.1, in ntables mode Tailscale inserts rules into the tables used by iptables-nft and ufw in order to coordinate with those ecosystems.

We hope to re-enable firewall mode auto-detection in a future release.

[hj-collab]

@hj-collab, we're not able to reproduce. Can you file a new bug with a lot more details?

Not able to reproduce with 1.48.1. I'll report back if I face this issue again.

[zlepper]

It is working for me on version v1.48.1-t0e9f04c83 now, with TS_DEBUG_FIREWALL_MODE=auto env set when running on Azure kubernetes (AKS) and the tailscale k8s operator.

[adrianmihalko]

In https://tailscale.com/kb/1294/firewall-mode/ guide

"When neither the iptables binary nor the nftables Netlink API are available, Tailscale will fall back to a degraded operation that may result in reduced performance or increased CPU usage. "

• this is still true? For me it doesn't fall back to degraded operation, instead it just exiting.

wgengine.NewUserspaceEngine(tun "tailscale0") error: creating router: exec: "iptables": executable file not found in $PATH

[rodrigc]

I ran a quick test.

1. Spun up a Kubernetes cluster in Oracle Cloud

2. Used image: docker.io/tailscale/tailscale:unstable-v1.49.189

3. Created a simple tailscale-test.yaml for testing:

---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: tailscale
  name: tailscale
spec:
  containers:
    - args:
        - /bin/sh
        - -c
        - echo hello;sleep 36000
      image: docker.io/tailscale/tailscale:unstable-v1.49.189
      name: tailscale
      securityContext:
        privileged: true
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

4. Applied the yaml:

kubectl apply -f tailscale-test.yaml

5. Jumped into the container:

kubectl exec -t -i tailscale -- sh

6. Set the heuristic to auto and started tailscaled:

export TS_DEBUG_FIREWALL_MODE=auto
tailscaled

7. I saw that the heurstic correctly identified nftables:

logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
wgengine.NewUserspaceEngine(tun "tailscale0") ...
router: nftables rule count: 0, iptables rule count: 0
router: nftables is available
router: using nftables
router: disabling tunneled IPv6 due to system IPv6 config: %!w(*errors.errorString=&{disable_ipv6 is set})

8. Verified that this image was being used on the Oracle side:
   Oracle-Linux-8.8-2023.06.30-0-OKE-1.26.2-632

[bradfitz]

Sufficiently done.