http3 connect failed: Resource temporarily unavailable (with ngtcp2)

[icebluey]

# curl-http3 -vI --http3 https://cloudflare-quic.com
*   Trying 2606:4700:10::6816:926:443...
* Connect socket 5 over QUIC to 2606:4700:10::6816:926:443
* connect to 2606:4700:10::6816:926 port 443 failed: Resource temporarily unavailable
*   Trying 2606:4700:10::6816:826:443...
* Connect socket 6 over QUIC to 2606:4700:10::6816:826:443
* connect to 2606:4700:10::6816:826 port 443 failed: Resource temporarily unavailable
*   Trying 2606:4700:10::ac43:9eb:443...
* Connect socket 7 over QUIC to 2606:4700:10::ac43:9eb:443
* connect to 2606:4700:10::ac43:9eb port 443 failed: Resource temporarily unavailable
*   Trying 172.67.9.235:443...
* Connect socket 8 over QUIC to 172.67.9.235:443
* connect to 172.67.9.235 port 443 failed: Resource temporarily unavailable
*   Trying 104.22.8.38:443...
* Connect socket 9 over QUIC to 104.22.8.38:443
* connect to 104.22.8.38 port 443 failed: Resource temporarily unavailable
*   Trying 104.22.9.38:443...
* Connect socket 10 over QUIC to 104.22.9.38:443
* connect to 104.22.9.38 port 443 failed: Resource temporarily unavailable
* Failed to connect to cloudflare-quic.com port 443: Resource temporarily unavailable
* Closing connection 0
curl: (7) Failed to connect to cloudflare-quic.com port 443: Resource temporarily unavailable

curl version: 7.76.1-DEV (commit: 3be5ebf)

# curl-http3 -V
curl 7.76.1-DEV (x86_64-pc-linux-gnu) libcurl/7.76.1-DEV GnuTLS/3.7.1 zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 c-ares/1.17.1 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 ngtcp2/0.1.0-DEV nghttp3/0.1.0-DEV
Release-Date: XXX
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets zstd

But the old version works.

# curl-http3 -vI --http3 https://cloudflare-quic.com
*   Trying 2606:4700:10::6816:826:443...
* Connect socket 5 over QUIC to 2606:4700:10::6816:826:443
*   Trying 104.22.8.38:443...
* Connect socket 6 over QUIC to 104.22.8.38:443
* QUIC handshake is completed
* Connected to cloudflare-quic.com () port 443 (#0)
* Using HTTP/3 Stream ID: 0 (easy handle 0xadbb50)
> HEAD / HTTP/3
> Host: cloudflare-quic.com
> user-agent: curl/7.74.0-DEV
> accept: */*
> 
* ngh3_stream_recv returns 0 bytes and EAGAIN
* ngh3_stream_recv returns 0 bytes and EAGAIN
< HTTP/3 200 
HTTP/3 200 
...
* Connection #0 to host cloudflare-quic.com left intact

[bagder]

@tatsuhiro-t any chance you can throw a quick eye on this? It looks like one of our more recent adjustments to the changing ngtcp2 APIs broke functionality and you might have a chance to spot that much faster than anyone else...

[tatsuhiro-t]

There are couple of issues, some are client side, others are, well, version negotiation failure.

Curl uses ALPN h3

curl/lib/vquic/ngtcp2.c

Lines 354 to 355 in 0409c12

	alpn = (const uint8_t *)NGHTTP3_ALPN_H3;
	alpnlen = sizeof(NGHTTP3_ALPN_H3) - 1;

and at the same time it uses version 0xff00001d

curl/lib/vquic/ngtcp2.c

Lines 816 to 818 in 0409c12

	rc = ngtcp2_conn_client_new(&qs->qconn, &qs->dcid, &qs->scid, &path,
	NGTCP2_PROTO_VER_MIN, &ng_callbacks,
	&qs->settings, &qs->transport_params, NULL, qs);

In my opinion, we should use version 0x00000001 for ALPN h3.

If we make this change, it can get from ngtcp2 server, but it fails in curl (56), which is probably another issue.

If we change version to 0x00000001, it still fails with cloudflare-quc.com because it does not support QUIC v1 yet.
In order to workaround this, use QUIC version 0xff00001d (as is) and ALPN h3-29 (change required). Some servers are still supporting h3-29, so this might work.

[tatsuhiro-t]

If we make this change, it can get from ngtcp2 server, but it fails in curl (56), which is probably another issue.

This is ngtcp2 server bug.

[icebluey]

@bagder , @tatsuhiro-t
#6886 this fix doesn't seem to work.
Curl v7.76.1
GnuTLS v3.7.1
nghttp3 (commit: 5cccfe32585738b86b3dd8341509273353ae4ca6)
ngtcp2 ( commit: 169c68127b78ea906c96b49b9e18d4f805ab8eda)
Still failed to connect to cloudflare-quic.com port 443: Resource temporarily unavailable.
Failed to connect to www.cloudflare.com port 443: Resource temporarily unavailable.
Failed to connect to www.google.com port 443: Resource temporarily unavailable.

[bagder]

Works for me:

$ ./src/curl -V
curl 7.76.1-DEV (x86_64-pc-linux-gnu) libcurl/7.76.1-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 c-ares/1.17.1 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0_DEV nghttp2/1.42.0-DEV ngtcp2/0.1.0-DEV nghttp3/0.1.0-DEV librtmp/2.3 libgsasl/1.10.0

[icebluey]

Works for me:

$ ./src/curl -V
curl 7.76.1-DEV (x86_64-pc-linux-gnu) libcurl/7.76.1-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 c-ares/1.17.1 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0_DEV nghttp2/1.42.0-DEV ngtcp2/0.1.0-DEV nghttp3/0.1.0-DEV librtmp/2.3 libgsasl/1.10.0

Your situation is with OpenSSL/1.1.1g?

I built ngtcp2 with gnutls.

# curl-http3 -V
curl 7.76.1 (x86_64-pc-linux-gnu) libcurl/7.76.1 GnuTLS/3.7.1 zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 c-ares/1.17.1 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 ngtcp2/0.1.0-DEV nghttp3/0.1.0-DEV
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets zstd

[tatsuhiro-t]

It looks like boringssl backed server does not like ClientHello generated by GnuTLS. It is hard for me to debug this because it is TLS internal stuff.

[bagder]

@icebluey before we landed #6886, this problem existed for ngtcp2 using either openssl or gnutls and I verified the fix with my openssl build, yes.

[ghost]

"Resource temporarily unavailable" still happens to me with GnuTLS build from GitLab as of GnuTLS commit a6a45ad0

[bagder]

Comments on closed issues mostly get lost/ignored. I filed a new issue for this issue for the gnutls flavor.