wgengine/router: add auto selection heuristic for iptables/nftables

[KevinLiang10]

Summary:

This change implements chooseFireWallMode that decides which one of iptables/nftables will be used.

Issue related

Updates: #391
Fix: #8111 #8733

Changes made

1. Added chooseFireWallMode that automatically decides which one of iptables ornftables should be used.
2. Replaced the TS_DEBUG_USE_NETLINK_NFTABLES envknob with TS_DEBUG_FIREWALL_MODE that should be set to either 'iptables' or 'nftables' to select firewall mode manually.
3. Reimplemented DetectIptables that detects iptables binary and returns number of rules in iptables in current network namespace.

Thought Process

1. It is a very edge case when iptables binary is not available, nft is also not available and iptables is in use. So we are not supporting this case for now.
2. It is a very edge case when nftables is only available throught nft binary but not netlinkAPI, so we will skip this case for now also. This is usually due to system configuration.
3. We always prioritize nftables ahead of iptables due to it's excellence in performance, and less likely to fail in containerized environment.
4. When In a container, if there was other app in the same container using iptables, we use iptables since if iptables wouldn't work the container would fail anyways. Otherwise we tend to use nftables since netlinkAPI is less likely to run into incompatibility issues.

Expected Result

1. If either 'iptables' or 'nftables' is set for TS_DEBUG_FIREWALL_MODE use the tool respectively.
2. If nftables is in use already use nftables.
3. if iptables is already in use user iptables, notice that since iptables-nft would write to nftables subsystem, iptables here refer to iptables-legacy.
4. Otherwise if nftables is available throught netlink API, use nftables. This case includes both ipt, nft are in use, neither is used or iptables-nft was used.
5. If only iptables binary is available, use iptables.
6. If neither iptables or nftables is available, return error.

How to test

On linux environment do the following:

1. Bring up tailscaled by running sudo TS_DEBUG_FIREWALL_MODE=iptables ./tool/go run ./cmd/tailscaled
2. Start tailscale by running sudo ./tool/go run ./cmd/tailscale up.
3. Run sudo iptables -L to view the rules added. In log of tailscaled, there should be a line router: router: envknob TS_DEBUG_NETFILTER_MODE=iptables set.
4. Turn everything down.
5. Bring up tailscaled by running sudo TS_DEBUG_FIREWALL_MODE=nftables ./tool/go run ./cmd/tailscaled
6. Start tailscale by running sudo ./tool/go run ./cmd/tailscale up.
7. Run sudo nft list ruleset to view the rules added. In the output log of tailscaled, it should say router: envknob TS_DEBUG_NETFILTER_MODE=nftables set.
8. Turn everything down.
9. Buring up tailscaled by running sudo ./tool/go run ./cmd/tailscaled
10. Start tailscale
11. You should see the count of iptables rules and nftables rule currently in system from the log in the form

router: router: iptables rule count: {number}
router: router: nftables rule count: {number}

if iptables is associated with iptables-nft, or nftables is used, it's likely that bother number are non zero depending on system implementation.
12. Your tailscale should be working, with which ever mode is selected.

[andrew-d]

Looking good! I did a preliminary review, and I'd appreciate if @raggi and/or @catzkorn could do one as well since they're more familiar with this than I am.

[raggi]

Looks good overall, but please adjust the logic so that we can still detect and function correctly on systems that only have iptables and are missing ip6tables, as we are able to function on those configurations today.

[joesankey]

I was getting the following error while trying these changes with the Kubernetes operator to expose a service on a tailnet:

│ modprobe: can't change directory to '/lib/modules': No such file or directory                                                                                               │
│ modprobe: can't change directory to '/lib/modules': No such file or directory                                                                                               │
│ modprobe: can't change directory to '/lib/modules': No such file or directory                                                                                               │
│ iptables v1.8.8 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)                                                              │
│ Perhaps iptables or your kernel needs to be upgraded.                                                                                                                       │
│ boot: 2023/08/11 01:42:37 installing proxy rules: executing iptables failed: exit status 3

Looks like containerboot needs to be updated to use iptables/nftables autoselection this PR introduced. When in nftable mode, the nat table no longer exists for container boot to add rules to.

I added this commit to my fork to fix the issue, at least for nftables.