Web services (browsing security and parental control) are too slow #525
Comments
zeus10000
changed the title
|
Hi! Can you post the configuration file? I'm interested whether you changed the ratelimit settings. Taking a guess, I think this is because you ran out of vCPU credits (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html). Can you make sure this isn't the case? Also try running |
hmage
added
waiting for data
|
I did not make any changes to the default config apart from adding additional filter lists & blocklist. Also, The CPU is pretty much staying idle =============================
=============================
============================= ; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> google.com @x.x.x.x ;; QUESTION SECTION: ;; Query time: 321 msec real 0m0.336s |
|
I've seen the same thing in AGH since the first 0.9 release and am wondering whether it is a scalability issue. I've run AGH on 2 environments with same results:
Massive amounts of headroom in both CPU and RAM on both devices. In general operation, I've never seen the General Processing time get under 100ms, and it is usually around 130ms. The ONLY time I saw it lower was when I was migrating from environment 1 (Linux image on Win10) to environment 2 (Windows image on Win Server 2012) and had both of them running concurrently and let devices migrate to the new one as they renewed the DHCP IP info to pick up the new DNS server. In the first 24 horus where the new environment only had a handful of devices using it, the general processing time was in the 20-40ms range. Within a couple of days when all of my devices had migrated over, it was back up to the 100+ms average. I've just upgraded from v0.92 to v0.92-hotfix1 in the last hour so I'll give that 24 hours to get current stats given v0.92 had a couple of DNS bugs that might cause bad results. My config, for reference: bind_host: 0.0.0.0
|
|
When a DOT upstream is under load, it operates on open connections so things go fast. Just in case, try using a plain DNS upstream instead, do you see any difference? |
|
I upgraded to v0.92-hotfix1 and for last 10 hours, the response time as reported on web gui went up to 70ms. I changed the upstream to use UDP instead of TLS, restarted the ADH service. After few hours, the DIG response time is now 263 ms vs the 321ms on using tls. ; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> google.com @xxxx ;; OPT PSEUDOSECTION: ;; ANSWER SECTION: ;; Query time: 263 msec real 0m0.279s |
|
@jimitsalvi it'd help if AGH was running with |
|
Hm, on the other hand, we'd better extend the verbose logging before proceeding. In the current implementation, we won't see how much time does every operation take. We should get back to this issue once #531 is done. |
|
just for the reference -- Not sure, how many logs you wanted but here is a few. Looks like the upstream DNS resolver takes no more than 14 ms. Is there some implementation where we could see the RTT downstream from the ADG to clients? 2019/01/05 18:52:12 AdGuard Home web interface backend, version v0.92-hotfix1 ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: ;; QUESTION SECTION: ;; ANSWER SECTION: ;; ADDITIONAL SECTION: ;; OPT PSEUDOSECTION: |
It's unclear how much we spend inside |
|
Sure, will wait for that feature. Ping & traceroute values to the server come to somewhere around 30~34 secs. 64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=42 time=32.9 ms 28: ec2-xx-xx-xx-xx.us-east-2.compute.amazonaws.com 33.675ms reached |
|
I tried adding a TCP upstream DNS (tcp://1.1.1.1) to see if it would improve performance but if anything I've seen average processing times start exceeding 200ms (prev ~130ms). Also, for comparison, GRC.com DNS Benchmark results below. Cached results are obviously going to be in AGH's favour, but for uncached and dom.com results, AGH is 4-5 times slower. |
|
Hi Guys. All too technical for me ;( - however, after lettingAdGuard run for a few days my Average Processing Time is at 400! It starts out at 3 and keeps rising. |
|
I would say benchmark can be misleading too, because you run that benchmark on some specific domain for short period. I would say it should be an option in the Query to see what is the query that is taking long time to run and provide the result in the query log, so we can see what's running slowly. |
|
@dany20mh we already have that info in querylog -- click 'download log' in querylog page and then use json parser to show only long queries |
|
@hmage but we don't have that in the query log view (in dashboard) and that's what I meant. |
|
Sorry for the huge delay, guys. We've just pushed the new release with extended logging which should be enough to figure if there's any issue: https://github.com/AdguardTeam/AdGuardHome/releases/tag/v0.93 Also, it uses a newer version of |
|
I've been running v0.93 for about 12 hours now and my average processing time is 403.11ms I'll turn on verbose logging to capture some data. |
|
Interesting, and in my case avg time dropped to about 15ms. |
|
About 24 hours on and average processing time is 414.92ms Ran verbose logging for about 45 mins. Zipped log attached |
|
You know, the log seems more or less okay. There are 765 requests in the log, the avg processing time of them is 110ms. One thing that I forgot to ask -- do you have "malware protection" or "parental control" services enabled. They both use a remote web service, and might slow things down. |
|
I do have "Use AdGuard browsing security web service" enabled I had wondered if maybe a couple of really slow queries were skewing the average processing time. When I first started v0.93 for the first hour the average processing time was around 50ms. It just seems that in that sustained 24 hour period there are enough slow responses to really skew the averages. If I check my AGH now, the average processing time is 278.74ms |
Uh, we should add it's timing to the log as well, I don't know why we ignored it:( @admitrevskiy could you please extend logging of the dnsFilter module? |
|
Yeah, so the initial lookup is so slow that even with the cache, it hurts the average time. So our options are:
Tbh, I like the second option more. |
Yes, you're right. For example for hub2.corp.adobe.com 2019/03/23 14:20:54 4032#40 [debug] github.com/AdguardTeam/AdGuardHome/dnsfilter.(*Dnsfilter).checkSafeBrowsing(): SafeBrowsing HTTP lookup for hub2.corp.adobe.com; Elapsed time: 331ms The next time you were looking for this host the following log message appeared: 2019/03/23 14:21:02 4032#85 [debug] github.com/AdguardTeam/AdGuardHome/dnsfilter.(*Dnsfilter).lookupCommon(): hub2.corp.adobe.com: found in the lookup cache |
ameshkov
changed the title
ameshkov
added
enhancement
P3: Medium
and removed
waiting for data
|
Reassigned to v0.96, the task is too complicated: Priority is set to medium because both of these options are disabled by default. |
|
So I turned off 'browsing security' yesterday and now my Average Processing time is 47ms That is a pretty significant impact! |
|
Here's what we are going to do:
|
ghost
mentioned this issue
ghost
mentioned this issue
ghost
mentioned this issue
UPDATE by @ameshkov: See the implementation details: #525 (comment)
Hello,
I am running Adguard-home on RHEL7.6 on AWS with 1vCPU and 1GB ram. Although initially, the DNS was resolving quickly, after 24 hours, I started noticing some delays in resolving DNS (upstream being the default cloudflare). The VM seems to be pretty quite.
I tried the same using Ubuntu 16.04 with similar behavior.
Any suggestions?
Environment
| Version of AdGuard Home server:| - v0.92
| How did you setup DNS configuration:| - AWS t2.micro
| Operating system and version:| RHEL 7.6, Ubuntu 16.04.
The text was updated successfully, but these errors were encountered: