Skip to content

Custom encrypted DNS upstream servers configured for a client are 4-5 times slower #7769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
4 tasks done
hagezi opened this issue Apr 16, 2025 · 17 comments
Open
4 tasks done

Custom encrypted DNS upstream servers configured for a client are 4-5 times slower #7769

hagezi opened this issue Apr 16, 2025 · 17 comments
Assignees
@schzhn
Labels
bug P2: High
Milestone
v0.107.63

Comments

@hagezi
Copy link

hagezi commented Apr 16, 2025

Prerequisites

Platform (OS and CPU architecture)

Linux, ARM64

Installation

GitHub releases or script from README

Setup

On a router, DHCP is handled by the router

AdGuard Home version

v0.108.0-b.66

Action

If custom encrypted DNS servers are configured for a client, the resolution times for this client are 4-5 times higher than the same standard configured encrypted DNS servers. If legacy DNS via IP is used instead, the problem does not occur.

Settings > Client Settings > Add/Edit Client
Tab Upstream DNS servers > Configure encrypted DoT or DoH Servers for the Client > Save > Test

Expected result

Same resolution times as with the standard configured encrypted upstream DNS servers.

Actual result

Resolution times are 4-5 times higher than the same standard configured encrypted DNS servers.

Additional information and/or screenshots

No response
@bohtho
Copy link

bohtho commented Apr 24, 2025 •
edited
Loading

I have the same observation and problem with lag (actually 50-100 times more lag than standard encrypted requests) in v0.108.0-b.68 and edge as of today (v0.108.0-a.1088+61a1403e), and I think it may be related to this issue, because that's the error I also get in my syslog. But as OP reports it seems to only affect configured persistent clients with custom upstreams, as per the response times visible in the query log (disregarding cached responses).

So as I layman I would suggest there is something in the code for the persistent clients´ custom upstreams that handles quic (and encrypted dns requests) different from the standard upstreams.

@bohtho bohtho mentioned this issue Apr 24, 2025
Open
4 tasks
fbernier added a commit to fbernier/AdGuardHome that referenced this issue Apr 25, 2025
@fbernier
Prevent continuous connection rebuilding in custom upstreams
b7e36b1 
This commit fixes a critical performance issue with custom encrypted DNS
upstreams configured for persistent clients. The bug caused connections
to be unnecessarily closed and rebuilt on every DNS request, making
custom DoH/DoT/DoQ upstreams 4-5 times slower than global upstreams.

The root cause was that after rebuilding a client's upstream connection,
the timestamp (commonConfUpdate) used for detecting configuration changes
was never updated. This resulted in an endless cycle of detecting "changes"
and rebuilding connections on every request, particularly impacting
encryption protocols with expensive connection establishment.

This PR updates the client's configuration timestamp after rebuilding the
connection in upstreamManager.customUpstreamConfig().

Fixes: AdguardTeam#7739, AdguardTeam#7769
@fbernier fbernier mentioned this issue Apr 25, 2025
Open
@fbernier
Copy link

I may have a fix for this but it's still unconfirmed. Please give it a try if you'd like: #7789

@tjharman
Copy link

Thanks @fbernier I'll make the team aware of this.

@bohtho
Copy link

bohtho commented Apr 26, 2025

I may have a fix for this but it's still unconfirmed. Please give it a try if you'd like: #7789

I run adguard home with a certain complicated installer on my router so I think I have to wait for it to get merged into edge unfortunately...

@fbernier
Copy link

fbernier commented Apr 26, 2025 •
edited
Loading

I may have a fix for this but it's still unconfirmed. Please give it a try if you'd like: #7789

I run adguard home with a certain complicated installer on my router so I think I have to wait for it to get merged into edge unfortunately...

I thought it would be a hassle too but turns out that cloning the repo on a linux desktop, running a single command to cross-compile to a single arm64 binary (make GOOS='linux' GOARCH='arm64'), scp-ing it to my router,docker cp` it to my docker container and restarting the container was pretty quick.

Here's an arm64 binary in case anyone's interested: https://drive.google.com/file/d/1NeGDB0tyX6A523Cp0GHC-M-Mqjturwcj/view

@bohtho
Copy link

bohtho commented Apr 27, 2025

I may have a fix for this but it's still unconfirmed. Please give it a try if you'd like: #7789

I run adguard home with a certain complicated installer on my router so I think I have to wait for it to get merged into edge unfortunately...

I thought it would be a hassle too but turns out that cloning the repo on a linux desktop, running a single command to cross-compile to a single arm64 binary (make GOOS='linux' GOARCH='arm64'), scp-ing it to my router,docker cp` it to my docker container and restarting the container was pretty quick.

Here's an arm64 binary in case anyone's interested: https://drive.google.com/file/d/1NeGDB0tyX6A523Cp0GHC-M-Mqjturwcj/view

Thanks, but my router uses armv7 architecture for most binaries, so it would really be much quicker and easier if maintainers would just merge this one-liner PR, which also actually fixes something that made the persistent client part of agh quite unusable since forever.

@hagezi
Copy link
Author

hagezi commented Apr 27, 2025

ping @ainar-g

@tjharman tjharman mentioned this issue Apr 27, 2025
Closed
4 tasks
@tjharman
Copy link

I've made the AGH team aware of this already - no need to @ people.

@ainar-g ainar-g added this to the v0.107.62 milestone Apr 28, 2025
@ainar-g
Copy link
Contributor

ainar-g commented Apr 28, 2025

Thanks for reporting! We've found that bug as well, and we're aiming to fix it in the next beta. Assigning to @schzhn, who is currently working on the fix.

@schzhn
Copy link
Member

schzhn commented Apr 28, 2025

The fix is already in the edge release. You can download the binary for your platform from this page.

@bohtho
Copy link

bohtho commented Apr 28, 2025

Did the fix for this in edge somehow negate the fix for the custom caches that was just merged days ago...?

@ainar-g
Copy link
Contributor

ainar-g commented Apr 29, 2025

@bohtho, I'm not sure what you mean. Do you see any issues on the edge channel currently?

@hagezi
Copy link
Author

hagezi commented Apr 29, 2025 •
edited
Loading

Docker edge version v0.108.0-a.1089+2c46bc92 fixes the problem for me and the custom client cache also seems to work.

@ainar-g
Copy link
Contributor

ainar-g commented Apr 29, 2025

@bohtho, I cannot reproduce that on the current edge release. There is a separate issue where disabling the global cache also disables custom caches, and we're working on that at the moment, and it should be fixed by the next beta release.

@bohtho
Copy link

bohtho commented May 3, 2025 •
edited
Loading

Leaving the global and custom caches at the same level (and turned on) the custom persistent clients definitely misses a lot in the cache even after several days. But suddenly some things are taken from cache too so I can't figure it out or pin it down.

Same edge v0.108.0-a.1089+2c46bc92

@bohtho
Copy link

bohtho commented May 12, 2025

Anyone else still seeing that the custom client caches are not working (perhaps after the custom client upstream timeout fix) ?

@schzhn
Copy link
Member

schzhn commented May 12, 2025

@bohtho, ensure that the custom client upstream DNS server list is not empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@schzhn schzhn

Labels
bug P2: High
Projects
None yet
Milestone
v0.107.63
Development

No branches or pull requests
6 participants
@bohtho @fbernier @tjharman @ainar-g @hagezi @schzhn