QUIC disabled for dns.google and www.google.com

[bassosimone]

We have many integration tests that use 8.8.8.8, dns.google, www.google.com with QUIC. This morning I noticed many of them started failing (see, e.g., https://github.com/ooni/probe-cli/runs/4188573018). What's happening?

(Cc: @fortuna)

I also noticed that www.google.com and dns.google do not advertise QUIC using the Alt-Svc header anymore. See:

> curl -svo/dev/null https://www.google.com/
*   Trying 216.58.208.164:443...
* Connected to www.google.com (216.58.208.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[...]
* Using Stream ID: 1 (easy handle 0x144811800)
> GET / HTTP/2
> Host: www.google.com
> user-agent: curl/7.77.0
> accept: */*
> 
< HTTP/2 200 
< date: Fri, 12 Nov 2021 10:58:23 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
[snip]
< alt-svc: clear
< accept-ranges: none
< vary: Accept-Encoding
< 
{ [1388 bytes data]
* Connection #0 to host www.google.com left intact

and

> curl -svo/dev/null https://dns.google/
*   Trying 8.8.8.8:443...
* Connected to dns.google (8.8.8.8) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[snip]
> GET / HTTP/2
> Host: dns.google
> user-agent: curl/7.77.0
> accept: */*
> 
< HTTP/2 200 
< content-security-policy: object-src 'none';base-uri 'self';script-src 'nonce-OwY+olMx2RJt/W8laSqVAA==' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/honest_dns/1_0;frame-ancestors 'none'
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
< content-type: text/html; charset=UTF-8
< date: Fri, 12 Nov 2021 11:20:00 GMT
< server: scaffolding on HTTPServer2
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: clear
< accept-ranges: none
< vary: Accept-Encoding
< 
{ [1383 bytes data]
* Connection #0 to host dns.google left intact

In both cases, alt-svc value is clear.

I don't think this is just an issue with our client implementation, but I'll compile quiche-client and see what happens with it.

So, here are some runs using quiche-client (I compiled cloudflare/quiche@3ea9bf4):

> ./target/debug/quiche-client https://www.cloudflare.com
[works]

> ./target/debug/quiche-client https://www.google.com 
[2021-11-12T11:24:26.847450219Z ERROR quiche_apps::client] connection timed out after 3.04268553s

> ./target/debug/quiche-client https://dns.google
[2021-11-12T11:25:08.935769177Z ERROR quiche_apps::client] connection timed out after 3.038625337s

> ./target/debug/quiche-client https://cloudflare-dns.com
[works]

I see similar errors and successes when using miniooni and forcing HTTP3. See below:

> ./miniooni -OHTTP3Enabled=true -i https://cloudflare-dns.com/ urlgetter
[      0.000461] <info> Current time: 2021-11-12 12:28:18 CET
[...]
[      1.664451] <info> [1/1] running with input: https://cloudflare-dns.com/
[      1.807013] <info> submitting measurement to OONI collector; please be patient...
[      1.963781] <info> New reportID: 20211112T112819Z_urlgetter_IT_30722_n1_veYDYAkvl537m3Cv
[...]

> ./miniooni -OHTTP3Enabled=true -i https://www.cloudflare.com/ urlgetter 
[      0.000491] <info> Current time: 2021-11-12 12:29:37 CET
[...]
[      0.570090] <info> [1/1] running with input: https://www.cloudflare.com/
[      0.778321] <info> submitting measurement to OONI collector; please be patient...
[      0.934527] <info> New reportID: 20211112T112937Z_urlgetter_IT_30722_n1_kfnSV1cwJBSNRLEJ
[...]

> ./miniooni -OHTTP3Enabled=true -i https://www.google.com/ urlgetter
[      0.000230] <info> Current time: 2021-11-12 12:30:47 CET
[...]
[      0.487870] <info> [1/1] running with input: https://www.google.com/
[      5.492121] <warn> measurement failed: map[error:generic_timeout_error]
[      5.492150] <info> submitting measurement to OONI collector; please be patient...
[      5.668001] <info> New reportID: 20211112T113052Z_urlgetter_IT_30722_n1_0G61uAoYblCbqton
[...]

> ./miniooni -OHTTP3Enabled=true -i https://dns.google/ urlgetter 
[      0.000359] <info> Current time: 2021-11-12 12:31:30 CET
[...]
[      0.825489] <info> [1/1] running with input: https://dns.google/
[     10.857773] <warn> measurement failed: map[error:generic_timeout_error]
[     10.857839] <info> submitting measurement to OONI collector; please be patient...
[     11.023574] <info> New reportID: 20211112T113140Z_urlgetter_IT_30722_n1_8f3FIoRqSzDMz1kT
[...]

So, clearly timeouts. Here are the corresponding OONI reports:

1. cloudflare-dns.com

2. www.cloudflare.com

3. www.google.com

4. dns.google

This fact is quite annoying for us, because we have many tests that are failing. I'll try to figure out if I can quickly modify those tests to use cloudflare services instead of Google services.

[bassosimone]

Fixed by ooni/probe-cli#593 and ooni/probe-cli#594.

Follow-up issue: #1874.

The original problem has been addressed, so we can close the issue.