masque connect-ip failure

[Bfarkiani]

Hi.
I am trying to play with quic using https://www.chromium.org/quic/playing-with-quic/ and connect-ip.
I used chromium source and compiled masque client and server. Then I ran masque server with
./masque_server --masque_mode=open --certificate_file=leaf_cert.pem --key_file=leaf_key_pkcs8.der --port=10000 --allow_unknown_root_cert --v=2

and then connected masque client to it with
./masque_client --disable_certificate_verification=true --certificate_file=leaf_cert.pem --key_file=leaf_key_pkcs8.der --allow_unknown_root_cert --bring_up_tun --v=2 [IPV6]:10000

IPv6 was enabled on both machines. Then I used curl with HTTP/3 to connect to google.

curl --interface tun0 -I -v -g -6 --http3 https://www.google.com

* Host www.google.com:443 was resolved.
* IPv6: 2a00:1450:401b:810::2004
* IPv4: (none)
*   Trying [2a00:1450:401b:810::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2a00:1450:401b:810::2004: Network is unreachable
* Failed to connect to www.google.com port 443 after 1 ms: Could not connect to server
*   Trying [2a00:1450:401b:810::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2a00:1450:401b:810::2004: Network is unreachable
* Failed to connect to www.google.com port 443 after 1 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to www.google.com port 443 after 1 ms: Could not connect to server


curl --interface tun0 -I -v --http3 https://www.google.com

* Host www.google.com:443 was resolved.
* IPv6: 2a00:1450:401b:810::2004
* IPv4: 142.250.203.196
*   Trying [2a00:1450:401b:810::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2a00:1450:401b:810::2004: Network is unreachable
*   Trying 142.250.203.196:443...
* socket successfully bound to interface 'tun0'
* GnuTLS ciphers: NORMAL:-VERS-ALL:+VERS-TLS1.3:-CIPHER-ALL:+AES-128-GCM:+AES-256-GCM:+CHACHA20-POLY1305:+AES-128-CCM:-GROUP-ALL:+GROUP-SECP256R1:+GROUP-X25519:+GROUP-SECP384R1:+GROUP-SECP521R1:%DISABLE_TLS13_COMPAT_MODE
* found 138 certificates in /etc/ssl/certs/ca-certificates.crt
* found 552 certificates in /etc/ssl/certs
*   Trying [2a00:1450:401b:810::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2a00:1450:401b:810::2004: Network is unreachable
*   Trying 142.250.203.196:443...
* socket successfully bound to interface 'tun0'
* ngtcp2_conn_handle_expiry returned error: ERR_HANDSHAKE_TIMEOUT
* Failed to connect to www.google.com port 443 after 10001 ms: Could not connect to server
* connect to 142.250.203.196 port 443 from 10.1.1.2 port 36342 failed: Connection timed out
* Failed to connect to www.google.com port 443 after 129417 ms: Could not connect to server
* closing connection #0
curl: (7) ngtcp2_conn_handle_expiry returned error: ERR_HANDSHAKE_TIMEOUT

As you see connection through IPv6 or v4 both failed. Client and server logs show they were communicating with each other.
Is my configuration correct or something else is wrong?

Thank you for your help.

[DavidSchinazi]

Hi, can you confirm what commit of QUICHE and what build commands you used?
Also, please run both masque_client and masque_server with --stderrthreshold=0 and paste the logs of both here.
You might need to pass in --cache_dir to masque_server as well.

[Bfarkiani]

The commit hash is: 6adc7c3b0bbae3ebb305fdf7735f96f8ea89b2eb and as far as I recall, I used ninja -C out/Debug masque_server masque_client
I attached all log files.
Client IPv6 is 2001:19f0:ac01:1f4f:5400:05ff:fe00:c481
Server IPv6 is 2001:19f0:ac00:4709:5400:05ff:fe00:c46a

Server command:
./masque_server --masque_mode=open --certificate_file=leaf_cert.pem --key_file=leaf_key_pkcs8.der --port=10000 --allow_unknown_root_cert --stderrthreshold=0 --cache_dir=/home/xxx --v=2 2>&1 | tee server.txt

Client command:
./masque_client --disable_certificate_verification=true --certificate_file=leaf_cert.pem --key_file=leaf_key_pkcs8.der --allow_unknown_root_cert --bring_up_tun --stderrthreshold=0 --v=2 [2001:19f0:ac00:4709:5400:5ff:fe00:c46a]:10000 2>&1 | tee client.txt

Curl output + ping6 and normal connection:

root@Test2:~/curl-x86_64/bin# ./curl --interface tun0 -I -v -g -6 --http3 https://www.google.com
* Host www.google.com:443 was resolved.
* IPv6: 2607:f8b0:4007:80f::2004
* IPv4: (none)
*   Trying [2607:f8b0:4007:80f::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2607:f8b0:4007:80f::2004: Network is unreachable
* Failed to connect to www.google.com port 443 after 1 ms: Couldn't connect to server
*   Trying [2607:f8b0:4007:80f::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2607:f8b0:4007:80f::2004: Network is unreachable
* Failed to connect to www.google.com port 443 after 1 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to www.google.com port 443 after 1 ms: Couldn't connect to server
root@Test2:~/curl-x86_64/bin# ./curl --interface tun0 -I -v --http3 https://www.google.com
* Host www.google.com:443 was resolved.
* IPv6: 2607:f8b0:4007:80f::2004
* IPv4: 142.250.72.228
*   Trying [2607:f8b0:4007:80f::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2607:f8b0:4007:80f::2004: Network is unreachable
*   Trying 142.250.72.228:443...
* socket successfully bound to interface 'tun0'
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
*   Trying [2607:f8b0:4007:80f::2004]:443...
* socket successfully bound to interface 'tun0'
* Immediate connect fail for 2607:f8b0:4007:80f::2004: Network is unreachable
*   Trying 142.250.72.228:443...
* socket successfully bound to interface 'tun0'
* QUIC connection has been shut down
* QUIC connect to 142.250.72.228 port 443 failed: Couldn't connect to server
* Failed to connect to www.google.com port 443 after 30001 ms: Couldn't connect to server
* connect to 142.250.72.228 port 443 from 10.1.1.2 port 37328 failed: Connection timed out
* Failed to connect to www.google.com port 443 after 130308 ms: Couldn't connect to server
* Closing connection
curl: (7) QUIC connection has been shut down
root@Test2:~/curl-x86_64/bin# ./curl -I -v --http3 https://www.google.com
* Host www.google.com:443 was resolved.
* IPv6: 2607:f8b0:4007:80f::2004
* IPv4: 142.250.189.4
*   Trying [2607:f8b0:4007:80f::2004]:443...
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* Server certificate:
*  subject: CN=*.google.com
*  start date: Jun 13 15:27:14 2024 GMT
*  expire date: Sep  5 15:27:13 2024 GMT
*  subjectAltName: host "www.google.com" matched cert's "*.google.com"
*  issuer: C=US; O=Google Trust Services; CN=WR2
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
* Connected to www.google.com (2607:f8b0:4007:80f::2004) port 443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://www.google.com/
* [HTTP/3] [0] [:method: HEAD]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: www.google.com]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.8.0]
* [HTTP/3] [0] [accept: */*]
> HEAD / HTTP/3
> Host: www.google.com
> User-Agent: curl/8.8.0
> Accept: */*
>
* Request completely sent off
< HTTP/3 200
HTTP/3 200
< content-type: text/html; charset=ISO-8859-1
content-type: text/html; charset=ISO-8859-1
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-PfxyYCytOLgJ3QOBRQmfgQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-PfxyYCytOLgJ3QOBRQmfgQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< date: Thu, 04 Jul 2024 19:22:38 GMT
date: Thu, 04 Jul 2024 19:22:38 GMT
< server: gws
server: gws
< x-xss-protection: 0
x-xss-protection: 0
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< expires: Thu, 04 Jul 2024 19:22:38 GMT
expires: Thu, 04 Jul 2024 19:22:38 GMT
< cache-control: private
cache-control: private
< set-cookie: AEC=AVYB7crosIkTrgvt8xJvdNEyXEGOj9wujh9dwdjTD5A0sjH1V_n6nG5bpbY; expires=Tue, 31-Dec-2024 19:22:38 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: AEC=AVYB7crosIkTrgvt8xJvdNEyXEGOj9wujh9dwdjTD5A0sjH1V_n6nG5bpbY; expires=Tue, 31-Dec-2024 19:22:38 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
< set-cookie: NID=515=h9_wO1IZHojSiWq9dC2RuGmp-VvfyHj781vKbBi-e7n3ec_oThdHAF94MjAoipHPqWhsYuTMvtlUVpNZLw93LQVkRmFX-zqHwmIdzcZKja00W9sxSkuOQ1wPbtJzL-04oj1iqQvbcdBHV1Zf0IfZ1ZLUTnS-xFbLk9NEPp5KV8M; expires=Fri, 03-Jan-2025 19:22:38 GMT; path=/; domain=.google.com; HttpOnly
set-cookie: NID=515=h9_wO1IZHojSiWq9dC2RuGmp-VvfyHj781vKbBi-e7n3ec_oThdHAF94MjAoipHPqWhsYuTMvtlUVpNZLw93LQVkRmFX-zqHwmIdzcZKja00W9sxSkuOQ1wPbtJzL-04oj1iqQvbcdBHV1Zf0IfZ1ZLUTnS-xFbLk9NEPp5KV8M; expires=Fri, 03-Jan-2025 19:22:38 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<

* Connection #0 to host www.google.com left intact
root@Test2:~/curl-x86_64/bin# ping6 2607:f8b0:4007:814::2004
PING 2607:f8b0:4007:814::2004(2607:f8b0:4007:814::2004) 56 data bytes
64 bytes from 2607:f8b0:4007:814::2004: icmp_seq=1 ttl=118 time=11.4 ms
64 bytes from 2607:f8b0:4007:814::2004: icmp_seq=2 ttl=118 time=11.3 ms
^C


--- 2607:f8b0:4007:814::2004 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 11.278/11.323/11.369/0.045 ms

I attached server and client logs and server pcap. These are freshly installed Ubuntu 22.04.4.
client.txt
server.txt

Thanks @DavidSchinazi

[DavidSchinazi]

Thanks for the logs! Packets are flowing correctly through the tunnel, but the issue is that --bring_up_tun currently only supports IPv4 without NAT. So in your test, IPv6 fails because the TUN interface doesn't have IP addresses on it, and then IPv4 sends out the packets onto the server WAN without rewriting the source IP address. You can use iptables MASQUERADE to enable NAT on your server box once the TUN interface is up. For what it's worth, general-purpose VPN code should totally do that for you, but this MASQUE toy code isn't quite there yet. Alternatively, you could also manually configure some IPv6 addresses on the TUN interfaces on both sides, and then enable routing on the server.

[Bfarkiani]

Thank you @DavidSchinazi for your comment. I tried with
iptables -t nat -A POSTROUTING -j MASQUERADE
on server box and changed client command to use IPv4 without NAT and it worked.