Skip to content

Commit

Permalink
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
Browse files Browse the repository at this point in the history
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
  • Loading branch information
icing committed Jan 31, 2023
1 parent 91eb197 commit a6a29f1
Show file tree
Hide file tree
Showing 60 changed files with 3,511 additions and 1,206 deletions.
3 changes: 3 additions & 0 deletions .github/scripts/spellcheck.words
Expand Up @@ -335,6 +335,9 @@ IoT
ipadOS
IPCXN
IPv
IPv4
IPv4/6
IPv6
IRIs
IRIX
Itanium
Expand Down
19 changes: 19 additions & 0 deletions configure.ac
Expand Up @@ -311,6 +311,16 @@ AS_HELP_STRING([--with-test-nghttpx=PATH],[where to find nghttpx for testing]),
)
AC_SUBST(TEST_NGHTTPX)

CADDY=caddy
AC_ARG_WITH(test-caddy,dnl
AS_HELP_STRING([--with-test-caddy=PATH],[where to find caddy for testing]),
CADDY=$withval
if test X"$OPT_CADDY" = "Xno" ; then
CADDY=""
fi
)
AC_SUBST(CADDY)

dnl we'd like a httpd+apachectl as test server
dnl
AC_ARG_WITH(test-httpd, [AS_HELP_STRING([--with-test-httpd=PATH],
Expand Down Expand Up @@ -366,6 +376,14 @@ fi
AC_PATH_PROG([APXS], [apxs])
AC_SUBST(HTTPD_NGHTTPX)

dnl the Caddy server we might use in testing
if test "x$TEST_CADDY" != "x"; then
CADDY="$TEST_CADDY"
else
AC_PATH_PROG([CADDY], [caddy])
fi
AC_SUBST(CADDY)

dnl If no TLS choice has been made, check if it was explicitly disabled or
dnl error out to force the user to decide.
if test -z "$TLSCHOICE"; then
Expand Down Expand Up @@ -4646,6 +4664,7 @@ AC_CONFIG_FILES([Makefile \
tests/libtest/Makefile \
tests/unit/Makefile \
tests/tests-httpd/config.ini \
tests/tests-httpd/Makefile \
packages/Makefile \
packages/vms/Makefile \
curl-config \
Expand Down
28 changes: 27 additions & 1 deletion docs/HTTP3.md
Expand Up @@ -239,7 +239,11 @@ directory, or copy `msquic.dll` and `msh3.dll` from that directory to the

# `--http3`

Use HTTP/3 directly:
Use only HTTP/3:

curl --http3-only https://nghttp2.org:4433/

Use HTTP/3 with fallback to HTTP/2 or HTTP/1.1 (see "HTTPS eyeballing" below):

curl --http3 https://nghttp2.org:4433/

Expand All @@ -249,6 +253,28 @@ Upgrade via Alt-Svc:

See this [list of public HTTP/3 servers](https://bagder.github.io/HTTP3-test/)

### HTTPS eyeballing

With option `--http3` curl will attempt earlier HTTP versions as well should the connect
attempt via HTTP/3 not succeed "fast enough". This strategy is similar to IPv4/6 happy
eyeballing where the alternate address family is used in parallel after a short delay.

The IPv4/6 eyeballing has a default of 200ms and you may override that via `--happy-eyeballs-timeout-ms value`.
Since HTTP/3 is still relatively new, we decided to use this timeout also for the HTTP eyeballing - with a slight twist.

The `happy-eyeballs-timeout-ms` value is the **hard** timeout, meaning after that time expired, a TLS connection is opened in addition to negotiate HTTP/2 or HTTP/1.1. At half of that value - currently - is the **soft** timeout. The soft timeout fires, when there has been **no data at all** seen from the server on the HTTP/3 connection.

So, without you specifying anything, the hard timeout is 200ms and the soft is 100ms:

* Ideally, the whole QUIC handshake happens and curl has a HTTP/3 connection in less than 100ms.
* When QUIC is not supported (or UDP does not work for this network path), no reply is seen and the HTTP/2 TLS+TCP connection starts 100ms later.
* In the worst case, UDP replies start before 100ms, but drag on. This will start the TLS+TCP connection after 200ms.
* When the QUIC handshake fails, the TLS+TCP connection is attempted right away. For example, when the QUIC server presents the wrong certificate.

The whole transfer only fails, when **both** QUIC and TLS+TCP fail to handshake or time out.

Note that all this happens in addition to IP version happy eyeballing. If the name resolution for the server gives more than one IP address, curl will try all those until one succeeds - just as with all other protocols. And if those IP addresses contain both IPv6 and IPv4, those attempts will happen, delayed, in parallel (the actual eyeballing).

## Known Bugs

Check out the [list of known HTTP3 bugs](https://curl.se/docs/knownbugs.html#HTTP3).
Expand Down
3 changes: 2 additions & 1 deletion docs/examples/http3.c
Expand Up @@ -40,7 +40,8 @@ int main(void)
/* Forcing HTTP/3 will make the connection fail if the server is not
accessible over QUIC + HTTP/3 on the given host and port.
Consider using CURLOPT_ALTSVC instead! */
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_3);
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION,
(long)CURL_HTTP_VERSION_3ONLY);

/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
Expand Down
6 changes: 4 additions & 2 deletions lib/Makefile.inc
Expand Up @@ -107,7 +107,8 @@ LIB_CFILES = \
base64.c \
bufref.c \
c-hyper.c \
cf-socket.c \
cf-http.c \
cf-socket.c \
cfilters.c \
conncache.c \
connect.c \
Expand Down Expand Up @@ -232,7 +233,8 @@ LIB_HFILES = \
asyn.h \
bufref.h \
c-hyper.h \
cf-socket.h \
cf-http.h \
cf-socket.h \
cfilters.h \
conncache.h \
connect.h \
Expand Down

0 comments on commit a6a29f1

Please sign in to comment.