Apparently xfrm lookup not taking place, egress unencrypted packets get blackholed

[jwfloroiu]

Hi All,

My scenario looks like:

                          /----  3.134.14.13   ===== \\       StrongSWAN AWS EC2  
100.64.2.207 --- AWS VPN                                [3.9.219.146 / 172.25.129.204] --- 172.25.131.130
                          \---- 18.220.197.136 ===== //

Ping requests goes left to right, ping responses arrive at StrongSWAN but get blackholed:

13:43:44.322386 06:ae:3f:29:18:94 > 06:b0:89:9c:09:da, ethertype IPv4 (0x0800), length 178: 18.220.197.136.4500 > 172.25.129.204.4500: UDP-encap: ESP(spi=0xc60e1793,seq=0x4f5), length 136
13:43:44.322386 06:ae:3f:29:18:94 > 06:b0:89:9c:09:da, ethertype IPv4 (0x0800), length 98: 100.64.2.207 > 172.25.131.130: ICMP echo request, id 18259, seq 1058, length 64
13:43:44.322514 06:b0:89:9c:09:da > 06:93:1c:24:a9:24, ethertype IPv4 (0x0800), length 98: 100.64.2.207 > 172.25.131.130: ICMP echo request, id 18259, seq 1058, length 64
13:43:44.322886 06:ae:3f:29:18:94 > 06:b0:89:9c:09:da, ethertype IPv4 (0x0800), length 98: 172.25.131.130 > 100.64.2.207: ICMP echo reply, id 18259, seq 1058, length 64
13:43:44.322897 06:b0:89:9c:09:da > 06:ae:3f:29:18:94, ethertype IPv4 (0x0800), length 98: 172.25.131.130 > 100.64.2.207: ICMP echo reply, id 18259, seq 1058, length 64

root@ip-172-25-129-204:~# arp -a
_gateway (172.25.128.1) at 06:ae:3f:29:18:94 [ether] on eth0
? (172.25.131.130) at 06:93:1c:24:a9:24 [ether] on eth0

To the configuration:

root@ip-172-25-129-204:~# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
link/ether 06:b0:89:9c:09:da brd ff:ff:ff:ff:ff:ff
inet 172.25.129.204/22 brd 172.25.131.255 scope global dynamic eth0
valid_lft 1899sec preferred_lft 1899sec
inet6 fe80::4b0:89ff:fe9c:9da/64 scope link
valid_lft forever preferred_lft forever

root@ip-172-25-129-204:~# ip r l t all
default via 172.25.128.1 dev eth0 proto dhcp src 172.25.129.204 metric 100
172.25.128.0/22 dev eth0 proto kernel scope link src 172.25.129.204
172.25.128.1 dev eth0 proto dhcp scope link src 172.25.129.204 metric 100
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 172.25.128.0 dev eth0 table local proto kernel scope link src 172.25.129.204
local 172.25.129.204 dev eth0 table local proto kernel scope host src 172.25.129.204
broadcast 172.25.131.255 dev eth0 table local proto kernel scope link src 172.25.129.204
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::4b0:89ff:fe9c:9da dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium

root@ip-172-25-129-204:~# cat /etc/ipsec.conf

conn %default
ikelifetime=24h
keylife=3h
rekeymargin=5m
authby=secret
type=tunnel
#keyexchange=ikev1
keyexchange=ikev2
mobike=no
ike=aes256-sha256-modp2048
esp=aes256-sha256-modp2048
left=172.25.129.204
leftid=3.9.219.146
auto=start
#leftupdown=/etc/strongswan.d/ipsec-vti.sh
#leftsubnet=0.0.0.0/0
#rightsubnet=0.0.0.0/0
installpolicy=yes
closeaction=restart
dpddelay=10
dpdtimeout=30
dpdaction=restart
#left=%defaultroute
#leftfirewall=yes

include /etc/ipsec.d/peers/*.conf

root@ip-172-25-129-204:~# cat /etc/ipsec.d/peers/peer10.conf
conn aws-11-1-1
right=3.134.14.13
rightsubnet=100.64.0.0/16
leftsubnet=172.25.128.0/22
leftupdown="/etc/strongswan.d/ipsec-vti.sh"
mark=0x0011
reqid=11
conn aws-12-1-1
right=18.220.197.136
rightsubnet=100.64.0.0/16
leftsubnet=172.25.128.0/22
leftupdown="/etc/strongswan.d/ipsec-vti.sh"
mark=0x0012
reqid=12

root@ip-172-25-129-204:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.6.2, Linux 5.4.0-1088-aws, x86_64):
uptime: 28 minutes, since Nov 11 13:00:23 2022
malloc: sbrk 1613824, mmap 0, used 809616, free 804208
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 8
loaded plugins: charon aesni aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
172.25.129.204
Connections:
aws-11-1-1: 172.25.129.204...3.134.14.13 IKEv2, dpddelay=10s
aws-11-1-1: local: [3.9.219.146] uses pre-shared key authentication
aws-11-1-1: remote: [3.134.14.13] uses pre-shared key authentication
aws-11-1-1: child: 172.25.128.0/22 === 100.64.0.0/16 TUNNEL, dpdaction=restart
aws-12-1-1: 172.25.129.204...18.220.197.136 IKEv2, dpddelay=10s
aws-12-1-1: local: [3.9.219.146] uses pre-shared key authentication
aws-12-1-1: remote: [18.220.197.136] uses pre-shared key authentication
aws-12-1-1: child: 172.25.128.0/22 === 100.64.0.0/16 TUNNEL, dpdaction=restart
Security Associations (2 up, 0 connecting):
aws-12-1-1[2]: ESTABLISHED 28 minutes ago, 172.25.129.204[3.9.219.146]...18.220.197.136[18.220.197.136]
aws-12-1-1[2]: IKEv2 SPIs: dec3f1e4b0210c2b_i* e44f36591881c3e8_r, pre-shared key reauthentication in 23 hours
aws-12-1-1[2]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
aws-12-1-1{2}: INSTALLED, TUNNEL, reqid 12, ESP in UDP SPIs: c60e1793_i c9cf994d_o
aws-12-1-1{2}: AES_CBC_256/HMAC_SHA2_256_128, 34776 bytes_i (414 pkts, 1s ago), 0 bytes_o, rekeying in 2 hours
aws-12-1-1{2}: 172.25.128.0/22 === 100.64.0.0/16
aws-11-1-1[1]: ESTABLISHED 28 minutes ago, 172.25.129.204[3.9.219.146]...3.134.14.13[3.134.14.13]
aws-11-1-1[1]: IKEv2 SPIs: 188392191047eb2c_i* 4ece66ec30472962_r, pre-shared key reauthentication in 23 hours
aws-11-1-1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
aws-11-1-1{1}: INSTALLED, TUNNEL, reqid 11, ESP in UDP SPIs: c00a1761_i cdb4b916_o
aws-11-1-1{1}: AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 2 hours
aws-11-1-1{1}: 172.25.128.0/22 === 100.64.0.0/16

root@ip-172-25-129-204:~# iptables-save
*mangle
:PREROUTING ACCEPT [3105:696229]
:INPUT ACCEPT [1919:596605]
:FORWARD ACCEPT [1186:99624]
:OUTPUT ACCEPT [1297:197351]
:POSTROUTING ACCEPT [2483:296975]
-A INPUT -s 18.220.197.136/32 -d 172.25.129.204/32 -p udp -m udp --dport 4500 -j MARK --set-xmark 0x12/0xffffffff
-A INPUT -s 3.134.14.13/32 -d 172.25.129.204/32 -p udp -m udp --dport 4500 -j MARK --set-xmark 0x11/0xffffffff
-A FORWARD -s 172.25.128.0/22 -d 100.64.0.0/16 -j MARK --set-xmark 0x12/0xffffffff
-A FORWARD -s 172.25.128.0/22 -d 100.64.0.0/16 -j MARK --set-xmark 0x11/0xffffffff
-A OUTPUT -s 172.25.128.0/22 -d 100.64.0.0/16 -j MARK --set-xmark 0x12/0xffffffff
-A OUTPUT -s 172.25.128.0/22 -d 100.64.0.0/16 -j MARK --set-xmark 0x11/0xffffffff
COMMIT

The interesting part is that the forward chain is being hit (counters are increasing) and a matching (as far as I can tell) policy is right there:

root@ip-172-25-129-204:~# iptables -v -n -L -t mangle
Chain PREROUTING (policy ACCEPT 3522 packets, 741K bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 2102 packets, 622K bytes)
pkts bytes target prot opt in out source destination
912 139K MARK udp -- * * 18.220.197.136 172.25.129.204 udp dpt:4500 MARK set 0x12
311 34832 MARK udp -- * * 3.134.14.13 172.25.129.204 udp dpt:4500 MARK set 0x11

Chain FORWARD (policy ACCEPT 1420 packets, 119K bytes)
pkts bytes target prot opt in out source destination
710 59640 MARK all -- * * 172.25.128.0/22 100.64.0.0/16 MARK set 0x12
710 59640 MARK all -- * * 172.25.128.0/22 100.64.0.0/16 MARK set 0x11

Chain OUTPUT (policy ACCEPT 1357 packets, 205K bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 172.25.128.0/22 100.64.0.0/16 MARK set 0x12
0 0 MARK all -- * * 172.25.128.0/22 100.64.0.0/16 MARK set 0x11

Chain POSTROUTING (policy ACCEPT 2777 packets, 325K bytes)
pkts bytes target prot opt in out source destination

root@ip-172-25-129-204:~# ip xfrm policy
src 172.25.128.0/22 dst 100.64.0.0/16
dir out priority 380543
mark 0x12/0xffffffff
tmpl src 172.25.129.204 dst 18.220.197.136
proto esp spi 0xc9cf994d reqid 12 mode tunnel
src 100.64.0.0/16 dst 172.25.128.0/22
dir fwd priority 380543
mark 0x12/0xffffffff
tmpl src 18.220.197.136 dst 172.25.129.204
proto esp reqid 12 mode tunnel
src 100.64.0.0/16 dst 172.25.128.0/22
dir in priority 380543
mark 0x12/0xffffffff
tmpl src 18.220.197.136 dst 172.25.129.204
proto esp reqid 12 mode tunnel
src 172.25.128.0/22 dst 100.64.0.0/16
dir out priority 380543
mark 0x11/0xffffffff
tmpl src 172.25.129.204 dst 3.134.14.13
proto esp spi 0xcdb4b916 reqid 11 mode tunnel
src 100.64.0.0/16 dst 172.25.128.0/22
dir fwd priority 380543
mark 0x11/0xffffffff
tmpl src 3.134.14.13 dst 172.25.129.204
proto esp reqid 11 mode tunnel
src 100.64.0.0/16 dst 172.25.128.0/22
dir in priority 380543
mark 0x11/0xffffffff
tmpl src 3.134.14.13 dst 172.25.129.204
proto esp reqid 11 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0

but the policy is not hit (as per ip -s xfrm policy, the counters are 0)

No xfrm errors are reported:

root@ip-172-25-129-204:~# cat /proc/net/xfrm_stat
XfrmInError 0
XfrmInBufferError 0
XfrmInHdrError 0
XfrmInNoStates 0
XfrmInStateProtoError 0
XfrmInStateModeError 0
XfrmInStateSeqError 0
XfrmInStateExpired 0
XfrmInStateMismatch 0
XfrmInStateInvalid 0
XfrmInTmplMismatch 0
XfrmInNoPols 0
XfrmInPolBlock 0
XfrmInPolError 0
XfrmOutError 0
XfrmOutBundleGenError 0
XfrmOutBundleCheckError 0
XfrmOutNoStates 0
XfrmOutStateProtoError 0
XfrmOutStateModeError 0
XfrmOutStateSeqError 0
XfrmOutStateExpired 0
XfrmOutPolBlock 0
XfrmOutPolDead 0
XfrmOutPolError 0
XfrmFwdHdrError 0
XfrmOutStateInvalid 0
XfrmAcquireError 0

And disable_xfrm and disable_policy are set to 0:

root@ip-172-25-129-204:~# cat /proc/sys/net/ipv4/conf/eth0/disable_*
0
0

Any help would be greatly appreciated.

Thank you,
John

[Thermi]

Looks like you don't have any VTIs that could be used.

[jwfloroiu]

The name of the updown script is a bit misleading, sorry for that. I use VTIs for the route-based VPNs. However, this is a policy-based VPN. Therefore the updown script only installs the iptables mangle rules (and no VTIs) in this case.

[Thermi]

For this setup to work you MUST use VTIs or XFRM interfaces. Otherwise you can not have two parallel CHILD_SAs with identical networks in the traffic selectors.

[jwfloroiu]

I will look into that, but shouldn't I see at least some error counters increased in /proc/net/xfrm_stat?

[Thermi]

No because you are setting marks. The marks are being evaluated also as part of the IPsec policies. You most likely did not configure any setting of the right mark values so the policies never match.

[jwfloroiu]

All works fine if the MARK rules in the mangle table for the FORWARD hook go to PREROUTING. Which is a bit of a mystery to me since according to https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg, for instance, xfrm lookup for inside -> outside plaintext packets would take place after FORWARD and even after POSTROUTING. But none of FORWARD or POSTROUTING work. The conclusion is: PREROUTING does the marking for the packets generated by the clients and OUTPUT does the marking for the packets generated locally.

Two distinct marks are required in order to set up two distinct IPsec tunnels for the same IP traffic selector.

And two distinct reqid's are required (iirc) to mitigate the rekeying issue described here: https://wiki.strongswan.org/issues/3268.