util/linuxfw: add nftable support for linux router

[KevinLiang10]

Summary:

This change implements netfilterRunner interface for nftables manipulation as a low level nftables runner. This change allows manipulation to netfilter rules via nftables.

Issue related

Updates: #391

Changes made

1. Added nftableRunner that implements netfilterRunner and nftable as a container of the nftables.Table abstraction.
2. Added envknob TS_DEBUG_USE_NETLINK_NFTABLES Which controls using nftables or iptables.
3. Added nftables rule creaters via google/nftables package.
4. Added unit test for nftables, the tests are splitted in to two parts. The first part tests on the correctness of rule we insert against the expected binary representation. The second part tests if the rules are added and deleted by interface methods in a testing namespace.

Thought Process

1. nftablesRunner contains a single connection for all nftables operation. This design was for testing with customized connections.
2. The second part of tests has to run with sudo privilege since we created a new network namespace for testing the rule operations.
3. All ts rules has counter expression for basic debug log purposes provided to user. If we want we can also add log expressions for logging in system
4. Since nftables doesn't have default table, we also created tailscale tables to contain the chains.
5. Since nftables doesn't have default chains, we have promoted our chain to a little bit higher priority than usual (-1 for filter type chains and -100 for NAT type chain) to be processed ahead of "normal fw rules" (basically the ones that follow convensions). The NAT type chain has reached highest possible priority but filter type chain can be promoted more if we want to.
6. Table cleaning was implemented for nftables rules.

Expected Result

tailscaled should be manipulating rules using nftables when it's running with envknob TS_DEBUG_USE_NETLINK_NFTABLES=true set. We expect that the app works the exact same way as we use iptables, and also expect these rules to be present when we run sudo nft list ruleset. Notice that the rule matching iifname "lo*" has local machine address as saddr, and the rules have the bytes displayed in little endian in the file paste file)

As an example for IPv4 rules.

How to test

On linux environment do the following:

1. Bring up tailscaled by running sudo TS_DEBUG_USE_NETLINK_NFTABLES=true ./tool/go run ./cmd/tailscaled
2. Start tailscale with a testing tailnet on corp.
3. Run sudo nft list ruleset to view the rules added.
4. Test the CGNAT rule and ChromeOSVMRange rule by creating packet from corresponding saddr range to a none tailscale network interface.
5. Shut down tailscale
6. Run tailscale as an exit node by running sudo ./tool/go run ./cmd/tailscale up --advertise-exit-node, the way of allowing forward and exit node can be found here.
7. Connect to exit node from another node on tailscale.
8. Check traffic on NAT table and ts-forward chain respectly. (About setting and checking ts marks and controlling outgoing packets)

Note

If we want to build then run, the env var should come with run, not build. So basically something like

go build -o tailscaled ./cmd/tailscaled/ && env TS_DEBUG_USE_NETLINK_NFTABLES=1 sudo -E ./tailscaled --statedir /tmp/ts/ --socket /tmp/ts/ts1.sock --verbose 99

[catzkorn]

A first pass, will need other eyes on too!

[KevinLiang10]

nftables package should work in most cases, the only case wouldn't work is that system has security policy (say using SElinux) that prevents user space program to work with netlink api but still supports system utilities like nft to function. Since nftables rely on golang.org/x/sys/unix, it should compile even when netlink header is missing.

The solution I can think of are :

1. Somehow change security policy, but this is usually done by system admin we might not want to do this.
2. We can use r.cmd.run to use nft in this case.

I'm not sure it if worth talking about this case, since it's kind of like an edge case that usually wouldn't happen? Tho I do recall that @raggi mentioned to me there are distros only giveing us access to nft but not netlink api.

I didn't solve this problem when I implemented this solution because we thought the nftables package would solve this case, but I did some further investigation today and it doesn't seem like the case. We might need @raggi to provide further information

[twitchyliquid64]

Luv it

[xxx@tslptp:~]$ sudo nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		counter packets 4578 bytes 9213931 jump ts-input
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 0 bytes 0 jump ts-forward
	}

	chain ts-input {
		iifname "lo" ip saddr 100.82.9.85 counter packets 0 bytes 0 accept
		iifname != "tailscale0" ip saddr 100.115.92.0/23 counter packets 0 bytes 0 return
		iifname != "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
	}

	chain ts-forward {
		iifname "tailscale0" counter packets 0 bytes 0 meta mark set mark and 0xff00ffff xor 0x40000
		meta mark & 0x00ff0000 == 0x00040000 counter packets 0 bytes 0 accept
		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
		oifname "tailscale0" counter packets 0 bytes 0 accept
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		counter packets 1 bytes 88 jump ts-input
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 0 bytes 0 jump ts-forward
	}

	chain ts-input {
		iifname "lo" ip6 saddr fd7a:115c:a1e0:ab12:4843:cd96:6252:955 counter packets 0 bytes 0 accept
	}

	chain ts-forward {
		iifname "tailscale0" counter packets 0 bytes 0 meta mark set mark and 0xff00ffff xor 0x40000
		meta mark & 0x00ff0000 == 0x00040000 counter packets 0 bytes 0 accept
		oifname "tailscale0" counter packets 0 bytes 0 accept
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		counter packets 423 bytes 90219 jump ts-postrouting
	}

	chain ts-postrouting {
		meta mark & 0x00ff0000 == 0x00040000 counter packets 0 bytes 0 masquerade
	}
}
table ip6 nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		counter packets 0 bytes 0 jump ts-postrouting
	}

	chain ts-postrouting {
		meta mark & 0x00ff0000 == 0x00040000 counter packets 0 bytes 0 masquerade
	}
}

@danderson fyi - is this something you want to take a pass at?

[danderson]

@twitchyliquid64 I don't need to take a pass at this, if it implements the same behavior as the iptables implementation then it's good to go. I also frankly don't know nftables well enough to find subtle issues. Do you want additional scrutiny?

[twitchyliquid64]

Also, what platforms has this been tested on?

[twitchyliquid64]

@twitchyliquid64 I don't need to take a pass at this, if it implements the same behavior as the iptables implementation then it's good to go. I also frankly don't know nftables well enough to find subtle issues. Do you want additional scrutiny?

Gotcha - I'm happy, and ive done some local testing too. I'm feeling good about this going in after 1.46 is cut (for 1.48). We should also try and get it on some subnet routers / exit nodes.

[raggi]

Looking really good Kevin! I've not had a chance to test this out locally yet, but I just did a pass through, some comments included. The key thing I would like to understand is how evolution will work as we change the rules

[raggi]

Related to the comment above on findRule around the longer term lifetime of matching and upgrading these rules, related considerations show up here.

Perhaps what we want to find is a rule that matches a pattern, rather than exact expression list. We might for example look for a rule that matches the core spirit of the operation, rather than an exact expression list. If we do that it may also avoid the need to relax conditions or skip over implementation in the tests. Did you try something like this?

[raggi]

Relatedly, is there an expression we can include that has no runtime effect, but acts as an explicit marker that demonstrates that it is a rule we inserted?

[KevinLiang10]

I think this topic Adrian brought up was the answer for the marker. We can do this by adding comment to rule but don't have a simple way to do this.

[raggi]

You might like https://pkg.go.dev/encoding/hex#Dump which does most of what I think you want here, it takes a []byte and gives you a string containing a hexdump of the format:

00000000  47 6f 20 69 73 20 61 6e  20 6f 70 65 6e 20 73 6f  |Go is an open so|
00000010  75 72 63 65 20 70 72 6f  67 72 61 6d 6d 69 6e 67  |urce programming|
00000020  20 6c 61 6e 67 75 61 67  65 2e                    | language.|

[KevinLiang10]

Tried Hex.Dump, the fact that I can't change number of bytes in single like makes me don't like it. If only one bit goes wrong it takes much more effort to find the mismatch (at least for me)

[sailorfrag]

I feel like there are still some details regarding rule life cycle that we might need to work out further but I also feel like it's probably fine to give this a try for now (especially since it requires an env var to opt in) and we can make incremental improvements later. I suspect it's already more robust than the iptables path.

[sailorfrag]

consider having newSysConn call t.Cleanup(cleanupSysConn) (which will require reworking how t and ns are stored a little, maybe the cleanup would need to be a closure) to make it a bit easier for test writers

[KevinLiang10]

Updated

[raggi]

Looks good, and now had a chance to test on my router and it's working there too!

[raggi]

Please squash when you submit!