Deleted port forwarding rules aren't completely cleaned up on Isolated Network VRs

[kateellinenkoyha]

problem

Summary:
The iptables rules created on the VR in the PREROUTING chain of the nat table aren't properly cleaned up when a previously created port forwarding rule is deleted, and the leftover rules aren't accounted for if a user recreates said rule later.
This results in the new ruleset being generated in the wrong order, breaking routing of return traffic under certain configurations.
In these certain configurations, users can no longer establish working connections for a given combination of public IP address and port, seemingly "tainting" the address for that network even if they release and re-acquire it.

This bug has no apparent adverse security implications, but it is highly disruptive when encountered, and has no user-diagnosable cause (i.e. apart from logging onto the affected VR) or non-disruptive fix for non-admin users.

Details:
As it stands, Cloudstack's Virtual Router can, to an extent, handle user traffic into an Isolated Network arriving over multiple public interfaces. This scenario can appear when multiple public IP address ranges are available to users of a given Zone.
As NAT is involved, and the underlying structure of the public networks available on each interface are unknown to the VR, care must be taken to direct return traffic for any given incoming connection (as permitted by port forwarding rules) back through the interface it first arrived.
A combination of the CONNMARK/MARK extensions of iptables and policy routing rules with fwmark is used to achieve this, and it works great initially, but is prone to breakage due to an apparent issue removing certain iptables rules on the VR when deleting port forwarding rules in ACS, and subsequently failing to check for their existence if a new port forwarding rule (say, an identical one) would recreate them.

More specifically, the CONNMARK rules involved in setting the correct connection mark for a given port forwarding rule aren't removed. That's these guys:
-A PREROUTING -d [public IP] -i eth2 (or greater) -p tcp -m tcp --dport [port] -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
Normally they'd be at the end of a given set of DNAT and MARK/CONNMARK rules, as such (for a given TCP port forwarding rule on eth2 of the VR):
-A PREROUTING -d [Public IP]/32 -i eth2 -p tcp -m tcp --dport ["public" port] -j DNAT --to-destination [Instance IP]:["private" port]
-A PREROUTING -d [Public IP]/32 -i eth0 -p tcp -m tcp --dport ["public" port] -j DNAT --to-destination [Instance IP]:["private" port]
-A PREROUTING -d [Public IP]/32 -i eth2 -p tcp -m tcp --dport ["public" port] -j MARK --set-xmark [Interface fwmark]/0xffffffff
-A PREROUTING -d [Public IP]/32 -i eth2 -p tcp -m tcp --dport ["public" port] -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
When recreating the rules, these ones get skipped due to already existing. This appears to be a Cloudstack-specific effect, as iptables itself will happily add duplicate rules with -A.
As the rule is skipped, but exists higher in the chain already, the result is that rules initially added as [1,2,3,4] end up as [4], then [4,1,2,3] when deleted and re-added.

What then results is a subtly broken configuration; When a port forwarding rule is first added, it works as intended. Everything gets tagged with the correct connection and packet marks, and traffic is sent back on the interface it arrived on based on policy routing rules that match fwmarks to their respective interfaces.
When the rule is recreated, this intricate tagging ceases to work. Traffic that would normally be tagged with, say, 0x67 (mark=103, which is generally the case for the second public interface on a VR, eth3) instead comes out of nat:PREROUTING marked as 0x00 (mark=0). This causes return traffic to fall back to the default route, which is not necessarily the original receiving interface.
What's more, the address translation is handled as normal, and so the correct public IP address is still used, even if it's not local to the now-outgoing interface. This makes troubleshooting rather confusing, but has the fortunate side effect of masking the issue under certain network conditions. See example rules below (more on the networks here under STEPS TO REPRODUCE):
root@r-212-VM:~# ip rule show
0: from all lookup local
32761: from 10.172.103.0/24 lookup Table_eth3
32762: from all fwmark 0x67 lookup Table_eth3
32763: from 10.172.102.0/24 lookup Table_eth2
32764: from all fwmark 0x66 lookup Table_eth2
32765: from all fwmark 0x64 lookup Table_eth0
32766: from all lookup main
32767: from all lookup default

root@r-212-VM:~# ip route show
default via 10.172.102.1 dev eth2
10.70.0.0/24 dev eth0 proto kernel scope link src 10.70.0.1
10.172.102.0/24 dev eth2 proto kernel scope link src 10.172.102.118
10.172.103.0/24 dev eth3 proto kernel scope link src 10.172.103.108
169.254.0.0/16 dev eth1 proto kernel scope link src 169.254.161.96

The severity of this bug is essentially null for what I'd imagine to be the vast majority of Isolated Network setups.
Unless multiple IP ranges behind different VLANs (or other L2/L3 isolation labels, presumably) are available in the Zone and used on the same VR, marking NAT'd packets is, at least to my naïve eye, entirely optional.
The bug still occurs, but traffic tagged 0x00 still gets sent back the right way, as there is only one interface it could've arrived on.

In the event multiple such interfaces are present, the bug still only matters when PF rules on interfaces other than the one used for source NAT break this way. Any rule on eth2 will work fine, rules on eth3 and up are where things break.
Further yet, in certain cases it's possible for network conditions outside of Cloudstack itself to mask the breakage, though I can't imagine any real networks looking like this. If, for some reason, the upstream router local to the VR runs masquerading NAT for the connecting client (i.e. just one hop of client — NAT — VR), return traffic instead looks local to eth3 and gets sent through it. In my test environment, where the public networks for eth2 and eth3 are both routed through the same Linux host, traffic also works fine despite very much going through the wrong interface — not sure what's up with that.
For a sanely routed network, however, the breakage is very real, and not very hard for a user to encounter; "Why wouldn't this IP address and that one work the same?", and so on. Where it gets really annoying is that the issue persists, as mentioned above, even if the user releases and reacquires the address — there is no obvious fix for non-admins, save for restarting the network with cleanup (I think? I don't believe I've tested that yet.) or being told by an administrator to move all their rules to the source NAT address' address range (assuming they fit).

I've attempted to illustrate the routing behavior here, in case the word jumble above was too unclear:

versions

Cloudstack version: 4.18.2.5 - 4.22.1.0 (possibly applicable to versions below 4.18 as well, haven't tested)

Configuration: Advanced Network w/ VLAN isolation.
Currently only tested for Isolated Networks w/ Conserve mode ON and source NAT + port forwarding provided by VR.
Only benign effects observed unless multiple public IP ranges are available in Zone and are in use on the same VR.

OS/Environment: KVM on Ubuntu Server (Various: 18.04-24.04), x86-64. Ditto for management server w/ Mysql 8.0 as DB.

The steps to reproduce the bug

Steps to reproduce:
This is for reproducing the scenario wherein the aforementioned route breakage happens, requiring multiple public interfaces to be added to the affected VR.
The benign case can be observed on pretty much any Isolated Network with multiple public IPs allocated.

• Ensure your Zone has public IPs available from multiple ranges, preferably (Necessarily? Not quite sure.) from multiple VLANs.
  For example:

• Create an Isolated Network with an offering providing source NAT and port forwarding via VR (e.g. DefaultIsolatedNetworkOfferingWithSourceNatService).
  Start an instance in the network to allocate VR resources and for later port forwarding. Ensure you have console or SSH access, or have some daemon listening on the soon-to-be-forwarded port.
• Acquire an additional public IP address for the network from a range other than your source NAT address.
  Example:

• Add a port forwarding rule on the latter IP address pointing to your VM. (e.g. TCP/12345 from 0.0.0.0/0). Also add a corresponding firewall rule for access.

• Log onto the VR, observe source NAT address and the additional public address on separate interfaces.
  Example: root@r-212-VM:~# ip addr show | grep -E eth[0-9]
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  inet 10.70.0.1/24 brd 10.70.0.255 scope global eth0
  3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  inet 169.254.161.96/16 brd 169.254.255.255 scope global eth1
  4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  inet 10.172.102.118/24 brd 10.172.102.255 scope global eth2
  5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  inet 10.172.103.108/24 brd 10.172.103.255 scope global eth3

• Observe the port forwarding rules (particularly in nat:PREROUTING) being created correctly. Note the order of the rules therein.
  Example: root@r-212-VM:~# iptables-save | grep -E '*|PREROUTING'
  [...]
  *nat
  [...]
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -j DNAT --to-destination 10.70.0.245:12345
  -A PREROUTING -d 10.172.103.108/32 -i eth0 -p tcp -m tcp --dport 12345 -j DNAT --to-destination 10.70.0.245:12345
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -j MARK --set-xmark 0x67/0xffffffff
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
  ...wherein 10.70.0.245 is our test instance in this case.

• Generate traffic through the port forwarding rule and observe the conntrack table on the VR. _Note the mark field - this should match set-xmark above (0x67 -> 103).
  Here I used netcat & telnet (ugly, I know), but a webserver works just as well:
  VM: root@test-4-22-iptables-1:~# nc -l 12345
  Workstation: [me]$ telnet 10.172.103.108 12345
  Then while that's running:
  VR: root@r-212-VM:~# grep 12345 /proc/net/nf_conntrack
  ipv4 2 tcp 6 431309 ESTABLISHED src=[workstation] dst=10.172.103.108 sport=44362 dport=12345 src=10.70.0.245 dst=[workstation] sport=12345 dport=44362 [ASSURED] mark=103 zone=0 use=2

• Delete the port forwarding rule in Cloudstack and check the VR's firewall state again.
  root@r-212-VM:~# iptables-save | grep -E '*|PREROUTING'
  [...]
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff

• Re-add the port forwarding rule as it was, check iptables yet again.
  root@r-212-VM:~# iptables-save | grep -E '*|PREROUTING'
  [...]
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -j DNAT --to-destination 10.70.0.245:12345
  -A PREROUTING -d 10.172.103.108/32 -i eth0 -p tcp -m tcp --dport 12345 -j DNAT --to-destination 10.70.0.245:12345
  -A PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp --dport 12345 -j MARK --set-xmark 0x67/0xffffffff

• Try generating traffic again. Depending on your network setup (see DETAILS) you may now end up with a hanging connection. Checking the conntrack table again will show the wrong fwmark being applied.
  root@r-212-VM:~# grep 12345 /proc/net/nf_conntrack
  ipv4 2 tcp 6 431542 ESTABLISHED src=[workstation] dst=10.172.103.108 sport=41816 dport=12345 src=10.70.0.245 dst=[workstation] sport=12345 dport=41816 [ASSURED] mark=0 zone=0 use=2
  Regardless of whether your connection hung, you can also observe the return traffic getting sent on the wrong interface:
  root@r-212-VM:~# tcpdump -vvv -n -i any tcp port 12345 | grep -E eth[0-9]
  tcpdump: data link type LINUX_SLL2
  tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
  16:01:26.632851 eth3 In IP (tos 0x0, ttl 63, id 52994, offset 0, flags [DF], proto TCP (6), length 54)
  16:01:26.632929 eth0 Out IP (tos 0x0, ttl 62, id 52994, offset 0, flags [DF], proto TCP (6), length 54)
  16:01:26.633761 eth0 In IP (tos 0x0, ttl 64, id 23420, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:26.633859 eth2 Out IP (tos 0x0, ttl 63, id 23420, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:30.728622 eth0 In IP (tos 0x0, ttl 64, id 23421, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:30.728725 eth2 Out IP (tos 0x0, ttl 63, id 23421, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:30.733900 eth3 In IP (tos 0x0, ttl 63, id 52995, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:30.733953 eth0 Out IP (tos 0x0, ttl 62, id 52995, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:33.344087 eth3 In IP (tos 0x0, ttl 63, id 52996, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:33.344162 eth0 Out IP (tos 0x0, ttl 62, id 52996, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:33.344878 eth0 In IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:33.344950 eth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 52)
  16:01:34.771509 eth3 In IP (tos 0x0, ttl 63, id 32345, offset 0, flags [DF], proto TCP (6), length 60)
  16:01:34.771615 eth0 Out IP (tos 0x0, ttl 62, id 32345, offset 0, flags [DF], proto TCP (6), length 60)
  16:01:34.772362 eth0 In IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)

Or simply look at eth3, then eth2. Note that 10.172.103.108 is local to eth3, not eth2:
root@r-212-VM:~# tcpdump -vvv -n -i any tcp port 12345 -i eth3
tcpdump: listening on eth3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:03:27.947511 IP (tos 0x0, ttl 63, id 32349, offset 0, flags [DF], proto TCP (6), length 54)
[workstation].60348 > 10.172.103.108.12345: Flags [P.], cksum 0xcd3a (correct), seq 3293823536:3293823538, ack 4164530400, win 63, options [nop,nop,TS val 3532973700 ecr 1048281990], length 2
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
root@r-212-VM:~# tcpdump -vvv -n -i any tcp port 12345 -i eth2
tcpdump: listening on eth2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:03:35.074421 IP (tos 0x0, ttl 63, id 49016, offset 0, flags [DF], proto TCP (6), length 52)
10.172.103.108.12345 > [workstation].60348: Flags [.], cksum 0xd5a5 (incorrect -> 0x7e92), seq 4164530400, ack 3293823540, win 510, options [nop,nop,TS val 1048297899 ecr 3532980824], length 0

For further testing, I found it easiest to use ip route get with the appropriate fwmark, e.g.:
Correct:
root@r-212-VM:~# ip route get from 10.70.0.245 to [workstation] ipproto tcp sport 12345 dport 40443 mark 0x67 iif eth0
[workstation] from 10.70.0.245 via 10.172.103.1 dev eth3 table Table_eth3 mark 0x67
cache iif eth0
Incorrect:
root@r-212-VM:~# ip route get from 10.70.0.245 to [workstation] ipproto tcp sport 12345 dport 40443 mark 0x00 iif eth0
[workstation] from 10.70.0.245 via 10.172.102.1 dev eth2
cache iif eth0

Expected results:

• Deleting port forwarding rules in Cloudstack removes every associated iptables rule on the Virtual Router.
• Failing to do so, an error is thrown and either some failsafe takes place or the leftover rule is removed upon readding the forwarding rule.
• Policy routing remains functional.

Actual results:

• Some rules silently fail to be deleted (well, at least silently to the user; this was actually logged on the VR).
• Recreated rules are not ordered correctly, causing traffic to be routed incorrectly.

What to do about it?

The immediate cause is improper handling of the rule when passing it to be deleted, so the immediate solution would be to fix said handling.

When the VR fails to delete the CONNMARK rule, the following is logged:
Command 'iptables -t nat -D PREROUTING -d 10.172.103.108/32 -i eth3 -p tcp -m tcp -m state --state NEW --dport 12345 -j CONNMARK' returned non-zero exit status 2.
It appears that the rule is being truncated before deletion; the above command is missing the parameters after -j CONNMARK at the end. (I.e. --save-mark --nfmask 0xfff...)

As far as I can tell, this comes down to a logic issue in CsNetfilter.__convert_to_dict @ systemvm/debian/opt/cloud/bin/cs/CsNetfilter.py:357 (@4.22.1.0), where an even amount of parameters is expected but the rule has an odd amount of them. This truncates some of the content, and CsNetfilter.to_str then finally butchers it entirely.

def __convert_to_dict(self, rule):
    [...]
    bits = rule.split(' ')
    rule = dict(list(zip(bits[0::2], bits[1::2]))) ### This seems to be where the problem occurs.
    if "-A" in list(rule.keys()):
        self.chain = rule["-A"]
    return rule

Not sure how that'd be fixed in a smart way, but an "ugly hack" like the one written for --checksum-fill could probably be reused for --save-mark, if nothing else.
Copying said hack, and adding --save-mark (as --save), --nfmask and --ctmask to the "order" list in to_str seems to produce the correct output, but I haven't tested any further than that.
Of course, it'd be better if the parsing here were less fragile overall, but I suppose that'd be more of a maintenance task.

Maybe there should also be more obvious logging and/or errors thrown upon critical errors on the VR (in this case, when rules fail to be applied or deleted correctly). I can imagine there may be more similar issues, given the fragility of the parsing here, that'd be pretty easily caught if the tests or even real production VRs threw harsher errors whenever iptables and similar commands fail.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!