h3, quic upstreams not working in v0.107.39 on FreeBSD #6301

avhm · 2023-10-12T13:28:02Z

Prerequisites

I have checked the Wiki and Discussions and found no answer
I have searched other issues and found no duplicates
I want to report a bug and not ask a question or ask for help
I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)

Platform (OS and CPU architecture)

FreeBSD, AMD64 (aka x86_64)

Installation

Custom package (OpenWrt, HomeAssistant, etc; please mention in the description)

Setup

On a router, DHCP is handled by the router

AdGuard Home version

v0.107.39

Action

I've previously been using upstreams designated as:
h3://unfiltered.adguard-dns.com/dns-query

However, once updating to v0.107.39 these now all fail. Swapping back to:
https://unfiltered.adguard-dns.com/dns-query
fixes the issue, but not ideal.

Happy to dig into this in more detail if required, just lmk what's helpful for diagnosis.

Expected result

I expected forcing http3 to function

Actual result

All requests to these upstreams fail.

Additional information and/or screenshots

No response

The text was updated successfully, but these errors were encountered:

utlimate · 2023-10-12T13:58:54Z

same issue

EugeneOne1 · 2023-10-12T14:01:17Z

@avhm, hello and thanks for the report. Could you please collect a verbose log to help us troubleshoot the issue? You may send it to devteam@adguard.com.

ainar-g · 2023-10-12T14:07:38Z

I cannot reproduce this.

Custom package (OpenWrt, HomeAssistant, etc; please mention in the description)

Is it the same with the default AdGuard Home build?

Also, is it the same for all HTTP/3 upstreams, or only for the unfiltered AdGuard DNS one?

m0d3rnX · 2023-10-12T14:11:46Z

DNS-over-Quic and DNS-over-HTTPS3 are making problems now,
DNS-over-TLS and DNS-over-HTTPS are still working.

avhm · 2023-10-12T14:12:28Z

@ainar-g It's the same for all http3 upstreams (was using three separate upstreams), I've only tested the h3:// designation in the DNS upstream UI, not by changing the http3 config (configuration yml is defaults otherwise)

Platform is OPNsense 23.7.6-amd64 on FreeBSD 13.2-RELEASE-p3 installed via mimugmail community plugin

avhm · 2023-10-12T14:13:29Z

@avhm, hello and thanks for the report. Could you please collect a verbose log to help us troubleshoot the issue? You may send it to devteam@adguard.com.

I'll dump this for you tonight when I can get some downtime on the Opnsense router.

ainar-g · 2023-10-12T14:45:49Z

Thanks, we've received the logs, but unfortunately they show the opposite, that the HTTP/3 is actually used:

[debug] using HTTP/3 for this upstream: QUIC was faster
[debug] REDACTED:443: sending request over udp: A REDACTED.
…
[debug] upstream REDACTED:443 exchanged ;REDACTED.	IN	 A successfully in 879.419711ms

Perhaps the upstream check is faulty? Can you send the logs that appear when you push Test upstreams with an h3:// upstream?

avhm · 2023-10-12T17:05:02Z

It was another user who sent in the logs I assume, but can confirm (for me at least) that this was an issue with resolution, not just the test.

All upstream resolvers were set with h3://

After updating, no requests were returning until I manually set the upstream resolvers to https instead of h3.

I'll run the logging on my side with the test trigger too before sending them in later.

ainar-g · 2023-10-12T18:22:33Z

Seems like there are quite a few people with issues with QUIC and HTTP/3 in v0.107.39. This is most likely related to quic-go/quic-go#4105. We'll try to rollback to a previous version and see if that fixes the situation.

avhm · 2023-10-12T18:25:06Z

Logs sent, only thing jumping out at me in the log is failing IPV6 requests - my ISP doesn't route IPV6, so wondering if this is causing the failure state.

It looks to me like the IPv4 queries were successful and that H3 is actually working as intended, but for some reason the failing IPv6 queries trigger a failure.

ainar-g · 2023-10-12T18:27:35Z

Thanks for the logs, @avhm!

INTERNAL_ERROR (local): write udp [::]:4288->94.140.14.140:443: sendmsg: invalid argument

Looks like it is indeed the quic-go error I've mentioned previously. We'll try a rollback of the library.

Updates #6301. Squashed commit of the following: commit e1a1b03 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Thu Oct 12 21:41:14 2023 +0300 all: rollback quic-go

ainar-g · 2023-10-12T19:04:49Z

We've released v0.108.0-a.729+506d7131 on the Edge channel. Could any of you please try it out and tell us if QUIC and HTTP/3 upstreams work again? (See the Platforms page on the Wiki for links to the builds and follow the manual update instructions with the archive.)

donfeduardo · 2023-10-12T19:35:50Z

Testing upstreams (including QUIC but not H3) is successful with the Edge channel version for me.

marten-seemann · 2023-10-13T07:26:23Z

Does quic-go/quic-go#4110 fix the problem?

Do you have any advice for us how we can find these kind of bugs in the future before a release? This would have been discovered if we had run our test suite on FreeBSD, but unfortunately GitHub Actions doesn't come with any FreeBSD runners.

ainar-g · 2023-10-13T11:50:16Z

@marten-seemann, I've pushed a branch of dnsproxy, which uses the same DNS libraries as AdGuard Home, but is simpler to set up. Running it with:

./dnsproxy -u 'quic://dns.adguard-dns.com' -v

on a FreeBSD VM in Qemu seems to work, unlike the version currently in our master.

(Anyone willing to test that fix on a real FreeBSD can switch to that branch, run make go-build, run the same command, and make queries with something like host 'www.example.com' '127.0.0.1'.)

Do you have any advice for us how we can find these kind of bugs in the future before a release?

I've been thinking about the same thing for a while now. There are GitHub actions that run VMs on one of GitHub's blessed systems, but at that point it's not that different from cross-compiling the test binary with go test -c and hacking together a large qemu call.

If anybody else has any suggestions, please share.

Adgbeta · 2023-10-14T00:41:21Z

I have same problem here, I am using Version: v0.108.0-b.47, I am on FreeBSD too.

In DNS settings, when I test the DNS upstream, I have this erros:

Server "h3://unfiltered.adguard-dns.com/dns-query": could not be used, please check that you've written it correctly
Server "quic://dns.adguard-dns.com": could not be used, please check that you've written it correctly
Server "quic://unfiltered.adguard-dns.com": could not be used, please check that you've written it correctly

I didn't have this issue before. This was after installing v0.108.0-b.47

Thanks

ainar-g · 2023-10-16T13:41:45Z

Did anybody else test the fix by @marten-seemann? We need to decide whether to rollback to the previous version or wait for quic-go to merge the fix.

donfeduardo · 2023-10-16T15:57:34Z

My OPNsense firewall isn't set up to build anything, but I tried this on a different FreeBSD box and it seems to be doing the right thing with commit a27ca72:

dnsproxy_test_log.txt

Adgbeta · 2023-10-16T16:12:10Z

Did anybody else test the fix by @marten-seemann? We need to decide whether to rollback to the previous version or wait for quic-go to merge the fix.

I just tested Version: v0.108.0-a.730+733d6c1f with only quic://unfiltered.adguard-dns.com and quic://dns.adguard-dns.com is working. This is the response:

Response details
Status
Processed
DNS server
quic://dns.adguard-dns.com:853 (served from cache)
Elapsed
0.35 ms
Response code
NOERROR
Response
A: 172.253.122.99 (ttl=213)
A: 172.253.122.147 (ttl=213)
A: 172.253.122.103 (ttl=213)
A: 172.253.122.104 (ttl=213)
A: 172.253.122.105 (ttl=213)
A: 172.253.122.106 (ttl=213)

Response details
Status
Processed
DNS server
quic://dns.adguard-dns.com:853 (served from cache)
Elapsed
0.23 ms
Response code
NOERROR
Response
AAAA: 2607:f8b0:4004:c08::8a (ttl=46)
AAAA: 2607:f8b0:4004:c08::71 (ttl=46)
AAAA: 2607:f8b0:4004:c08::8b (ttl=46)
AAAA: 2607:f8b0:4004:c08::64 (ttl=46)

this is the response for h3://unfiltered.adguard-dns.com/dns-query :

Response details
Status
Processed
DNS server
https://unfiltered.adguard-dns.com:443/dns-query (served from cache)
Elapsed
0.24 ms
Response code
NOERROR
Response
A: 172.253.122.99 (ttl=92)
A: 172.253.122.147 (ttl=92)
A: 172.253.122.106 (ttl=92)
A: 172.253.122.104 (ttl=92)
A: 172.253.122.103 (ttl=92)
A: 172.253.122.105 (ttl=92)

Response details
Status
Processed
DNS server
https://unfiltered.adguard-dns.com:443/dns-query (served from cache)
Elapsed
0.24 ms
Response code
NOERROR
Response
AAAA: 2607:f8b0:4004:c08::64 (ttl=157)
AAAA: 2607:f8b0:4004:c08::8b (ttl=157)
AAAA: 2607:f8b0:4004:c08::71 (ttl=157)
AAAA: 2607:f8b0:4004:c08::8a (ttl=157)

SCPdesign · 2023-10-16T19:59:56Z

Rolling back to v0.107.38 fixed this for me.

quic://94.140.14.14
quic://94.140.15.15
quic://dns.adguard.com
quic://dns.adguard-dns.com

I'm on FreeBDS. Could not use the new version at all. Failure all around.

marten-seemann · 2023-10-17T06:15:29Z

I just released v0.39.1 with the fix for the ECN control message on FreeBSD.

Updates AdguardTeam/AdGuardHome#6301. Squashed commit of the following: commit 2114596 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Oct 17 15:47:52 2023 +0300 all: upd quic-go

Updates #6301. Squashed commit of the following: commit 1199ea2 Merge: 7218fdc f514f36 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Oct 17 16:54:36 2023 +0300 Merge branch 'master' into 6301-upd-dnsproxy-quic-go commit 7218fdc Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Oct 17 16:23:24 2023 +0300 all: upd dnsproxy, quic-go

ainar-g · 2023-10-17T15:26:44Z

The newest Beta release, v0.108.0-b.48, contains the fix from @marten-seemann. If we get a confirmation that it works on FreeBSD for anyone here, we'll be able to ship the final release this week.

donfeduardo · 2023-10-17T15:44:32Z

"Test upstreams" and a host query both work for me with v0.108.0-b.48.

adguard_test_v0.108.0-b.48.txt

ainar-g · 2023-10-18T16:53:15Z

@avhm, this should be fixed in v0.107.40. Can you (or anyone else experiencing the issue in v0.107.39) please check?

avhm · 2023-10-18T16:55:00Z

@ainar-g confirmed, just updated and ran the tests again, all looking good.

Thanks for the fast turn around!

Mirabis · 2023-10-18T16:57:43Z

I can also confirm that the update fixed it, see attached screenshots:

Updates AdguardTeam#6301. Squashed commit of the following: commit e1a1b03 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Thu Oct 12 21:41:14 2023 +0300 all: rollback quic-go

Updates AdguardTeam#6301. Squashed commit of the following: commit 1199ea2 Merge: 7218fdc f514f36 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Oct 17 16:54:36 2023 +0300 Merge branch 'master' into 6301-upd-dnsproxy-quic-go commit 7218fdc Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Oct 17 16:23:24 2023 +0300 all: upd dnsproxy, quic-go

avhm changed the title ~~h3 upstreams longer working after v0.107.39 update~~ h3 upstreams no longer working after v0.107.39 update Oct 12, 2023

EugeneOne1 added the waiting for data Waiting for users to provide more data. label Oct 12, 2023

ainar-g mentioned this issue Oct 12, 2023

Upstream DNS over QUIC has failed in v0.107.39 #6306

Closed

4 tasks

ainar-g self-assigned this Oct 12, 2023

ainar-g added needs investigation Needs to be reproduced reliably. external libs Issues that require changes in external libraries. and removed waiting for data Waiting for users to provide more data. labels Oct 12, 2023

ainar-g added this to the v0.107.40 milestone Oct 12, 2023

adguard pushed a commit that referenced this issue Oct 12, 2023

Pull request 2040: 6301-rollback-quic-go

506d713

Updates #6301. Squashed commit of the following: commit e1a1b03 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Thu Oct 12 21:41:14 2023 +0300 all: rollback quic-go

ainar-g changed the title ~~h3 upstreams no longer working after v0.107.39 update~~ h3, quic upstreams not working in v0.107.39 on FreeBSD Oct 12, 2023

ainar-g pinned this issue Oct 12, 2023

marten-seemann mentioned this issue Oct 17, 2023

Add support for FreeBSD runners ipdxco/custom-github-runners#24

Open

ainar-g modified the milestones: v0.107.41, v0.107.40 Oct 18, 2023

ainar-g added bug P2: High and removed needs investigation Needs to be reproduced reliably. labels Oct 18, 2023

ainar-g closed this as completed Oct 18, 2023

ainar-g unpinned this issue Oct 20, 2023

ainar-g mentioned this issue Oct 23, 2023

h3 URLs no longer work #6335

Closed

4 tasks

Freekers mentioned this issue Nov 14, 2023

DNS-over-QUIC upstream servers no longer work on v0.107.41 #6422

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

h3, quic upstreams not working in v0.107.39 on FreeBSD #6301

h3, quic upstreams not working in v0.107.39 on FreeBSD #6301

avhm commented Oct 12, 2023 •

edited

utlimate commented Oct 12, 2023

EugeneOne1 commented Oct 12, 2023

ainar-g commented Oct 12, 2023

m0d3rnX commented Oct 12, 2023 •

edited

avhm commented Oct 12, 2023 •

edited

avhm commented Oct 12, 2023

ainar-g commented Oct 12, 2023

avhm commented Oct 12, 2023 •

edited

ainar-g commented Oct 12, 2023

avhm commented Oct 12, 2023

ainar-g commented Oct 12, 2023

ainar-g commented Oct 12, 2023

donfeduardo commented Oct 12, 2023

marten-seemann commented Oct 13, 2023

ainar-g commented Oct 13, 2023

Adgbeta commented Oct 14, 2023 •

edited

ainar-g commented Oct 16, 2023

donfeduardo commented Oct 16, 2023

Adgbeta commented Oct 16, 2023

SCPdesign commented Oct 16, 2023 •

edited

marten-seemann commented Oct 17, 2023

ainar-g commented Oct 17, 2023

donfeduardo commented Oct 17, 2023

ainar-g commented Oct 18, 2023

avhm commented Oct 18, 2023 •

edited

Mirabis commented Oct 18, 2023

h3, quic upstreams not working in v0.107.39 on FreeBSD #6301

h3, quic upstreams not working in v0.107.39 on FreeBSD #6301

Comments

avhm commented Oct 12, 2023 • edited

Prerequisites

Platform (OS and CPU architecture)

Installation

Setup

AdGuard Home version

Action

Expected result

Actual result

Additional information and/or screenshots

utlimate commented Oct 12, 2023

EugeneOne1 commented Oct 12, 2023

ainar-g commented Oct 12, 2023

m0d3rnX commented Oct 12, 2023 • edited

avhm commented Oct 12, 2023 • edited

avhm commented Oct 12, 2023

ainar-g commented Oct 12, 2023

avhm commented Oct 12, 2023 • edited

ainar-g commented Oct 12, 2023

avhm commented Oct 12, 2023

ainar-g commented Oct 12, 2023

ainar-g commented Oct 12, 2023

donfeduardo commented Oct 12, 2023

marten-seemann commented Oct 13, 2023

ainar-g commented Oct 13, 2023

Adgbeta commented Oct 14, 2023 • edited

ainar-g commented Oct 16, 2023

donfeduardo commented Oct 16, 2023

Adgbeta commented Oct 16, 2023

SCPdesign commented Oct 16, 2023 • edited

marten-seemann commented Oct 17, 2023

ainar-g commented Oct 17, 2023

donfeduardo commented Oct 17, 2023

ainar-g commented Oct 18, 2023

avhm commented Oct 18, 2023 • edited

Mirabis commented Oct 18, 2023

avhm commented Oct 12, 2023 •

edited

m0d3rnX commented Oct 12, 2023 •

edited

avhm commented Oct 12, 2023 •

edited

avhm commented Oct 12, 2023 •

edited

Adgbeta commented Oct 14, 2023 •

edited

SCPdesign commented Oct 16, 2023 •

edited

avhm commented Oct 18, 2023 •

edited