VPN does not connect and stuck on "attempting to confirm connection" in IPv6 only environment with NAT64

[pavel-odintsov]

Describe the bug

We run IPv6 only network with NAT64 gateway (TAYGA based) and DNS64 implemented via Unbound to access legacy IPv4 Internet.

Unfortunately, latest version of Mozilla VPN fails to connect with error: "Attempting to confirm connection".

It successfully creates IPv4 routes:

ip r get 8.8.8.8
8.8.8.8 dev moz0 table 51820 src 10.64.15.80 uid 1000 
    cache 

And IPv6 routes:

ip -6 route get 2a03:2880:f158:82:face:b00c:0:25de
2a03:2880:f158:82:face:b00c:0:25de from :: dev moz0 table 51820 src fc00:bbbb:bbbb:bb01::1:f4f metric 1024 pref medium

But traffic is not flowing via IPv4:

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

And IPv6:

ping -6 2a03:2880:f158:82:face:b00c:0:25de
PING 2a03:2880:f158:82:face:b00c:0:25de(2a03:2880:f158:82:face:b00c:0:25de) 56 data bytes
^C
--- 2a03:2880:f158:82:face:b00c:0:25de ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

VPN version:
2.12.0 (2.12.0-jammy1)

Affected Platforms:
Ubuntu 22.04 on x86_64.

Prerequisites:
Nothing special.

Steps to reproduce:
Just click "Connect".

Expected result:
Connected and working fine VPN

Actual result:
It stuck on "attempting to confirm connection" and traffic is not going over VPN.

Additional notes:

mozillavpn_short.txt

┆Issue is synchronized with this Jira Task

[oskirby]

So, the root cause of this issue is that our implementation presumes that IPv4 connectivity exists, and the VPN is attempting to connect to the IPv4 address of the server. When this connection fails, it assumes that there is that the server is unreachable.

This presumption is wrong, and I agree that we ought to have a method to detect whether v4 or v6 connectivity is available.

[pavel-odintsov]

Thank you so much for your prompt reply. That's my normal setup and I'll be more then happy to test developer / beta builds and provide any debug information you need to investigate it.

[oskirby]

Would you be able to share what your device's routing table, and interface configuration looks like with the VPN deactivated? If there is an easy way to detect when in a v6-only environment we could simply add an if statement to WireguardUtilsLinux::updatePeer() in theory is all we need to select IPv6.

I am a little bit worried about clients that have an IPv6 address but no real IPv6 connectivity from their ISP... such a situation might lead to unintended breakage for a large number of users. But if this check could simply look for an obvious absence of IPv4 (eg: no local address, or no default route), then this might be very easy.

[pavel-odintsov]

Hello!

Thank you for prompt answer and sorry about delay.

I've provided output from main network tools for my network.

ifconfig:

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:a4:35:07:19  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp34s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:d8:61:52:bb:0b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp37s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::7d01:cd3a:98a8:7dc8  prefixlen 64  scopeid 0x20<link>
        inet6 xx:xx:xx:xx:1228:ffb6:5b2f:233e  prefixlen 64  scopeid 0x0<global>
        inet6 xx:xx:xx:xx:30bf:4623:1ac2:d6bf  prefixlen 64  scopeid 0x0<global>
        ether a0:36:9f:a3:57:5c  txqueuelen 1000  (Ethernet)
        RX packets 608643  bytes 624605579 (624.6 MB)
        RX errors 0  dropped 19668  overruns 0  frame 0
        TX packets 237477  bytes 89457388 (89.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xc0100000-c01fffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 50015  bytes 4971355 (4.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 50015  bytes 4971355 (4.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip addr show:

ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp34s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:d8:61:52:bb:0b brd ff:ff:ff:ff:ff:ff
3: enp37s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:a3:57:5c brd ff:ff:ff:ff:ff:ff
    inet6 xx:xx:xx:xx:30bf:4623:1ac2:d6bf/64 scope global temporary dynamic 
       valid_lft 86295sec preferred_lft 71895sec
    inet6 xx:xx:xx:xx:1228:ffb6:5b2f:233e/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86295sec preferred_lft 71895sec
    inet6 fe80::7d01:cd3a:98a8:7dc8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp37s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether a0:36:9f:a3:57:5d brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:a4:35:07:19 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

ip route show:

ip route show
169.254.0.0/16 dev docker0 scope link metric 1000 linkdown 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

ip -6 route show:

::1 dev lo proto kernel metric 256 pref medium
64:ff9b::/96 via 2a01:4b00:88b8:3a00:8832:73ff:fe02:edb6 dev enp37s0f0 proto static metric 100 pref medium
xx:xx:xx:xx:3a00:8832:73ff:fe02:edb6 dev enp37s0f0 proto static metric 100 pref medium
xx:xx:xx:xx::/64 dev enp37s0f0 proto ra metric 100 pref medium
xx:xx:xx:xx::/56 via fe80::1 dev enp37s0f0 proto ra metric 100 pref high
fde3:db50:80e2:1::/64 via fe80::daeb:46ff:fea9:b678 dev enp37s0f0 proto ra metric 100 pref low
fe80::/64 dev enp37s0f0 proto kernel metric 1024 pref medium
default via fe80::1 dev enp37s0f0 proto ra metric 100 pref medium

I think connect syscall may be good source of information about lack of global IPv4 connectivity, check how telnet works:

telnet 151.101.62.132
Trying 151.101.62.132...
telnet: Unable to connect to remote host: Network is unreachable

Internally, it's connect syscall returns ENETUNREACH:

strace -s 1024 -f telnet 151.101.62.132 2>&1|grep 'connect'

connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr("151.101.62.132")}, 16) = -1 ENETUNREACH (Network is unreachable)
write(2, "telnet: Unable to connect to remote host: Network is unreachable\n", 65telnet: Unable to connect to remote host: Network is unreachable

ping expresses same behaviour:

ping 151.101.62.132
ping: connect: Network is unreachable

And it's connect too:

trace -f -s 1024  ping 151.101.62.132 2>&1|grep connect
connect(5, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=inet_addr("151.101.62.132")}, 16) = -1 ENETUNREACH (Network is unreachable)

And curl is same:

curl -v http://151.101.62.132 
*   Trying 151.101.62.132:80...
* Immediate connect fail for 151.101.62.132: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

Fails on connect as other:

strace -f -s 1024 curl -v http://151.101.62.132 2>&1|grep connec
connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("151.101.62.132")}, 16) = -1 ENETUNREACH (Network is unreachable)

I think it may be pretty good signal that IPv4 is not available on machine and only thing you need some external IP address as localhost IPv4 connection work in my case:

telnet 127.0.0.1 179
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

I hope it helps.

If you can point me to your connection confirmation logic it will be very helpful.

[oskirby]

Thank you, that is very helpful. It seems that in your situation, there is no default route for IPv4 which is ultimately what causes networking syscalls to report Network is unreachable So we could probably get your use-case working again by doing a route lookup first, and using that to select IPv4 or IPv6 when attempting to start the connection.

[pavel-odintsov]

Hello! That would be great! Thank you.

[pavel-odintsov]

I've tried latest build which is 2.13.0 (2.13.0-jammy1). Unfortunately, it hungs on "Attempting to confirm connection" as previously.

[pavel-odintsov]

I discovered nice way to workaround this issue by using Wireguard directly.

mozillavpn tool got support for command line option to generate Wireguard configuration: #5817

I used it and I was able to run Mozilla VPN service in my IPv6 only environment with NAT64 without any issues: https://www.stableit.blog/2023/05/mozilla-vpn-without-ui-on-ubuntu-linux.html

Would be great to see native solution for this issue.