v0.107.31 stops responding to DNS requests, does not terminate #5896

timkgh · 2023-06-12T17:07:55Z

Prerequisites

I have checked the Wiki and Discussions and found no answer
I have searched other issues and found no duplicates
I want to report a bug and not ask a question

Operating system type

Other (please mention in the description)

CPU architecture

64-bit MIPS

Installation

Other (please mention in the description)

Setup

On a router, DHCP is handled by the router

AdGuard Home version

v0.107.31

Description

v0.107.31 running on EdgeRouter 12, mips64_softfloat

Sometimes it just hangs, it doesn't respond to DNS requests anymore, 1 CPU is at 100% and it also does not gracefully restart via systemctl restart, it needs to be force killed.

I know this is not a lot to go on, these random hangs are hard to troubleshoot. It was stable for a while, I believe this started with the updates to .30 and .31

Jun 11 21:46:01 router AdGuardHome[15089]: 2023/06/11 21:46:01.705395 [info] Received signal "terminated"
Jun 11 21:46:01 router AdGuardHome[15089]: 2023/06/11 21:46:01.705503 [info] stopping AdGuard Home
Jun 11 21:46:01 router AdGuardHome[15089]: 2023/06/11 21:46:01.705544 [info] stopping http server...
Jun 11 21:46:01 router AdGuardHome[15089]: 2023/06/11 21:46:01.705984 [info] stopped http server
Jun 11 21:47:31 router systemd[1]: AdGuardHome.service: State 'stop-sigterm' timed out. Killing.
Jun 11 21:47:31 router systemd[1]: AdGuardHome.service: Killing process 15089 (AdGuardHome) with signal SIGKILL.
Jun 11 21:47:31 router systemd[1]: AdGuardHome.service: Main process exited, code=killed, status=9/KILL
Jun 11 21:47:31 router systemd[1]: AdGuardHome.service: Unit entered failed state.
Jun 11 21:47:31 router systemd[1]: AdGuardHome.service: Failed with result 'timeout'.

The text was updated successfully, but these errors were encountered:

ainar-g · 2023-06-13T09:22:23Z

Could you enable the verbose logging and see if there are any clues there? Enabling debug profiling (debug_pprof: true in the configuration file), getting the profile from http://127.0.0.1:6060/debug/pprof/goroutine?debug=1, and sending them to devteam@adguard.com could also help us see if there are any connection leaks.

timkgh · 2023-06-15T05:44:39Z

Unfortunately I don't have enough space on my router for verbose logging and I'm also worried about flash wear.

poolski · 2023-06-15T10:42:43Z

Have you checked that it's not a memory issue? If you're on a ER12, that only has 1GB of RAM. It's conceivable that once RAM is used up, it goes to swap.

Would you be able to show the load values from uptime to see whether there's something going on there? When the system starts swapping heavily, it will slow right to a crawl, and services become unresponsive.

As another aside, is the router still behaving as expected? Or is the CPU bottleneck causing router issues too?

timkgh · 2023-06-15T14:05:30Z

It is not a memory problem, the memory use is normally < 300MB (right now 185MB). The router is fine when this happens, except 1 CPU core is at 100% taken by AGH. Under normal use the CPU cores are at < 5%. The AGH web interface works, AGH just stops responding to DNS requests. I don't need to reboot the router to recover, I just need to kill AGH.

# free -h
              total        used        free      shared  buff/cache   available
Mem:           973M        173M        549M         11M        250M        773M
Swap:            0B          0B          0B

uptime: 07:01:30 up 142 days, 19:26, 1 user, load average: 0.25, 0.25, 0.16

I've reverted to .29 for now because that's been stable for me. Problems started with the recent .30

copacetic82 · 2023-06-17T06:09:11Z

I am having a simiular random issue and am running the latest v0.107.32 on centos-release-7-9.2009.1.el7.centos.x86_64. AGH stops responding to DNS requests, and still able to access the GUI. Restarting the AdGuardHome service does not work i.e. have to force a reboot. Was not able to check CPU/Memory usage yet, but never had issues before the last 1-2 releases. I did observe the logs below, and have increaed the UDP receiver buffer as described here (https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size). Am waiting to see if the issue stops...

[root@dns1 ~]# systemctl status AdGuardHome
● AdGuardHome.service - AdGuard Home: Network-level blocker
Loaded: loaded (/etc/systemd/system/AdGuardHome.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-16 06:40:36 CEST; 13h ago
Main PID: 734 (AdGuardHome)
CGroup: /system.slice/AdGuardHome.service
└─734 /root/AdGuardHome/AdGuardHome -s run

Jun 16 17:18:11 dns1 AdGuardHome[734]: 2023/06/16 17:18:11.470190 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:18:14 dns1 AdGuardHome[734]: 2023/06/16 17:18:14.472600 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:21:11 dns1 AdGuardHome[734]: 2023/06/16 17:21:11.454448 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:21:14 dns1 AdGuardHome[734]: 2023/06/16 17:21:14.464525 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:21:40 dns1 AdGuardHome[734]: 2023/06/16 17:21:40.538860 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:21:43 dns1 AdGuardHome[734]: 2023/06/16 17:21:43.802327 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:40:34 dns1 AdGuardHome[734]: 2023/06/16 17:40:34.338152 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 17:40:37 dns1 AdGuardHome[734]: 2023/06/16 17:40:37.342884 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 18:00:19 dns1 AdGuardHome[734]: 2023/06/16 18:00:19.746670 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
Jun 16 18:00:22 dns1 AdGuardHome[734]: 2023/06/16 18:00:22.771022 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
[root@dns1 ~]#

copacetic82 · 2023-06-17T07:10:28Z

I am having a simiular random issue and am running the latest v0.107.32 on centos-release-7-9.2009.1.el7.centos.x86_64. AGH stops responding to DNS requests, and still able to access the GUI. Restarting the AdGuardHome service does not work i.e. have to force a reboot. Was not able to check CPU/Memory usage yet, but never had issues before the last 1-2 releases. I did observe the logs below, and have increaed the UDP receiver buffer as described here (https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size). Am waiting to see if the issue stops...

[root@dns1 ~]# systemctl status AdGuardHome ● AdGuardHome.service - AdGuard Home: Network-level blocker Loaded: loaded (/etc/systemd/system/AdGuardHome.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2023-06-16 06:40:36 CEST; 13h ago Main PID: 734 (AdGuardHome) CGroup: /system.slice/AdGuardHome.service └─734 /root/AdGuardHome/AdGuardHome -s run

Jun 16 17:18:11 dns1 AdGuardHome[734]: 2023/06/16 17:18:11.470190 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:18:14 dns1 AdGuardHome[734]: 2023/06/16 17:18:14.472600 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:21:11 dns1 AdGuardHome[734]: 2023/06/16 17:21:11.454448 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:21:14 dns1 AdGuardHome[734]: 2023/06/16 17:21:14.464525 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:21:40 dns1 AdGuardHome[734]: 2023/06/16 17:21:40.538860 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:21:43 dns1 AdGuardHome[734]: 2023/06/16 17:21:43.802327 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:40:34 dns1 AdGuardHome[734]: 2023/06/16 17:40:34.338152 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 17:40:37 dns1 AdGuardHome[734]: 2023/06/16 17:40:37.342884 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 18:00:19 dns1 AdGuardHome[734]: 2023/06/16 18:00:19.746670 [error] dnsproxy: unpacking udp packet: dns: buffer size too small Jun 16 18:00:22 dns1 AdGuardHome[734]: 2023/06/16 18:00:22.771022 [error] dnsproxy: unpacking udp packet: dns: buffer size too small [root@dns1 ~]#

Did not solve the issue...CPU is at 100%, enough free memory. Reverted back to v0.107.29.

ainar-g · 2023-06-19T11:26:02Z

@copacetic82, those errors are likely signalling that some plain-UDP clients are sending invalid DNS packets. We'd appreciate if you could reproduce the situation with verbose logging and profile collecting, as described in #5896 (comment), if you decide to give it another go.

copacetic82 · 2023-06-19T11:52:13Z

@copacetic82, those errors are likely signalling that some plain-UDP clients are sending invalid DNS packets. We'd appreciate if you could reproduce the situation with verbose logging and profile collecting, as described in #5896 (comment), if you decide to give it another go.

Hi...thanks for replying...unfortunately i cant reproduce it at the moment as I am away on vacation. But a revert to v0.107.29 did solve the issue for me...no problems since, and using the same clients. Maybe I have time to debug once I get back.

Porygon31 · 2023-06-22T22:31:36Z

It's happening to me since the last update.
I will try the verbose logging

TheWoodsy · 2023-06-23T02:05:18Z

Same issue.
AGH on Ubuntu 22.04 snap v0.107.32
Headless 8700k 32gb
Pegged a core at 100%
Nothing in logs around time of death in syslog/journal/AdGuardHome.err
WebUI works, DNS doesn't.
Yikes...

simonregn · 2023-06-23T07:44:38Z

I face the same problem. Never had issues with AdguardHome until a few weeks. I agree this issue must have started with the last releases.

ainar-g · 2023-06-29T14:31:45Z

We are still not able to reproduce this properly, and the data we've received so far unfortunately doesn't shed any light on what is happening and why. Anyone still having issues, please help us collect the data by doing the following:

Update to v0.108.0-b.38, which has more debug logging, or a later Edge release.
Enable the verbose logging.
Enable debug profiling (debug_pprof: true in the configuration file).
Wait for the issue to occur.
Save the profile from http://127.0.0.1:6060/debug/pprof/goroutine?debug=1, the verbose log, and send both devteam@adguard.com with “AdGuard Home Issue 5896” in the subject line.

bartounet16 · 2023-07-01T10:40:43Z

Hello same problem this morning . whith my adguard docker version: v0.108.0-a.586+2069eddf
Stop to listen and respond DNS request.

Web UI is working

This problem is during since 1 month !!!

bigwookie · 2023-07-04T09:36:07Z

#5962 Had the same problem here on 0.107.33 have reverted back to 0.107.29 as suggested.

bigwookie · 2023-07-04T11:16:55Z

We are still not able to reproduce this properly, and the data we've received so far unfortunately doesn't shed any light on what is happening and why. Anyone still having issues, please help us collect the data by doing the following:

Update to v0.108.0-b.38, which has more debug logging, or a later Edge release.

Enable the verbose logging.

Enable debug profiling (debug_pprof: true in the configuration file).

Wait for the issue to occur.

Save the profile from http://127.0.0.1:6060/debug/pprof/goroutine?debug=1, the verbose log, and send both devteam@adguard.com with “AdGuard Home Issue 5896” in the subject line.

Have set this up and left it running.

Updates #5896. Squashed commit of the following: commit 81ac59e Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Jul 4 16:07:33 2023 +0300 hashprefix: fix cache coding

ainar-g · 2023-07-04T13:47:10Z

Thanks to the logs sent by @bigwookie in #5962, we were able to pinpoint a bug in the code which could cause such issues. We've fixed it in version v0.108.0-a.600+c4651647 on the Edge channel. Could you please check if the fix improves the situation?

bigwookie · 2023-07-04T18:22:15Z

Thanks to the logs sent by @bigwookie in #5962, we were able to pinpoint a bug in the code which could cause such issues. We've fixed it in version v0.108.0-a.600+c4651647 on the Edge channel. Could you please check if the fix improves the situation?

@ainar-g I have downloaded this and left it running.

bigwookie · 2023-07-05T08:34:08Z

@ainar-g I can confirm it's running better.

erdmanbr · 2023-07-05T11:01:17Z

I've been having the same issue I believe and managed to sort out how to update to the latest Edge release over the weekend. It stalled again last night (running v0.108.0-a.600+c4651647) and I was able to grab and send a log of the event. Will follow up if it happens again since I neglected to enable the pprof option this time.

bigwookie · 2023-07-05T11:16:19Z

@ainar-g Just failed again will get the logs and send on.

up2you1 · 2023-07-05T13:49:07Z

just to confirm, i am also getting the same on v0.107.33

LXC Container in Proxmox - the CPU randomly spikes to 100% which causes it to malfunction and not pass any dns through
Docker Container in Unraid - CPU also spikes here too and stops it from working

A reboot of both fixes is temporarily

LXC ADH set to Primary DNS and Docker is Secondary in DHCP

Ive had to switch the LXC Container off as ive no idea how to downgrade it or even use the Edge channel, but ive set the Docker to use Edge and its pulled v0.108.0-a.600+c4651647.

il monitor it

ainar-g · 2023-07-10T15:15:40Z

Yes and yes.

Updates #5896. Squashed commit of the following: commit 4934054 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Mon Jul 10 17:59:45 2023 +0300 hashprefix: fix loop pointer

ainar-g · 2023-07-10T16:33:32Z

Version v0.108.0-a.605+c02a1411 on the Edge channel contains the new fix. @bigwookie, could you please try that one out as well?

bigwookie · 2023-07-10T16:51:52Z

Version v0.108.0-a.605+c02a1411 on the Edge channel contains the new fix. @bigwookie, could you please try that one out as well?

I am now running Version:
v0.108.0-a.605+c02a1411

ainar-g · 2023-07-11T14:19:44Z

For those who follow the Beta channel, v0.108.0-b.40 contains the same fixes.

Freebase394 · 2023-07-11T22:43:38Z

Nice Work @ainar-g

monsm · 2023-07-12T00:31:01Z

Version v0.108.0-a.605+c02a1411 on the Edge channel contains the new fix. @bigwookie, could you please try that one out as well?

I am now running Version: v0.108.0-a.605+c02a1411

@bigwookie How does the new version work

bigwookie · 2023-07-12T07:11:13Z

Version v0.108.0-a.605+c02a1411 on the Edge channel contains the new fix. @bigwookie, could you please try that one out as well?

I am now running Version: v0.108.0-a.605+c02a1411

@bigwookie How does the new version work

So far it is running smoothly with no crashes or errors as before. @ainar-g thank you for this, it's looking good.

ainar-g · 2023-07-12T12:50:31Z

The fixes have now been released to v0.107.34. Once again, thanks to @bigwookie, @TheCableGuy99, and everyone else who tried Edge and Beta builds!

Baxxy13 · 2023-07-13T15:04:23Z

Because of the constantly increasing memory-usage with the last versions i rebooted the Pi3b+ almost once a week.
Let's see if this is now history...

s1lviu · 2023-07-15T12:19:09Z

I had the issue too, hope is fixed.

TheCableGuy99 · 2023-07-17T10:39:46Z

That looks to be HomeAssistant?

OlliZabal · 2023-07-17T18:11:42Z

I'm having some error messages frequently in the docker image log using v0.107.34:
2023/07/17 18:51:59.297889 [error] dnsproxy: unpacking udp packet: dns: buffer size too small

TheCableGuy99 · 2023-07-17T18:22:28Z

I'm having some error messages frequently in the docker image log using v0.107.34:

2023/07/17 18:51:59.297889 [error] dnsproxy: unpacking udp packet: dns: buffer size too small

Unrelated! Open your own thread!

bartounet16 · 2023-07-17T19:37:51Z

hello the problem is it resolved.
1 week without problem with version: v0.108.0-a.616+2adc8624

timkgh · 2023-07-17T20:47:31Z

v0.107.34 looks good 👍

bigwookie · 2023-07-17T23:49:45Z

Yes v0.107.34 looks good from my end, it has been running efficiently and without error since the release.

bartounet16 · 2023-07-18T18:35:29Z

WE Can use no edge version now ?

bartounet16 · 2023-07-18T18:37:34Z

The fixes have now been released to v0.107.34. Once again, thanks to @bigwookie, @TheCableGuy99, and everyone else who tried Edge and Beta builds!

Could toi explain basically what was the problem ?

ainar-g added the waiting for data Waiting for users to provide more data. label Jun 13, 2023

timkgh mentioned this issue Jun 13, 2023

Stops responding randomly #5898

Closed

3 tasks

ainar-g mentioned this issue Jun 29, 2023

DNS stopped working in recent update #5944

Closed

4 tasks

ainar-g pinned this issue Jun 29, 2023

ainar-g mentioned this issue Jun 30, 2023

Couple of times a day shoots to 100% cpu and stops working #5952

Closed

4 tasks

bundyland mentioned this issue Jul 3, 2023

Adguard no respond DNS request after certain Time - Docker Contener become Unealthy #5933

Closed

4 tasks

Freebase394 mentioned this issue Jul 4, 2023

BUG - CPU 100% - Randomly #5964

Closed

4 tasks

adguard pushed a commit that referenced this issue Jul 4, 2023

Pull request 1906: 5896-safe-browsing-cpu-ram

c465164

Updates #5896. Squashed commit of the following: commit 81ac59e Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Tue Jul 4 16:07:33 2023 +0300 hashprefix: fix cache coding

ainar-g mentioned this issue Jul 4, 2023

DNS requests stop being processed but GUI responsive with high CPU usage. #5962

Closed

4 tasks

ainar-g self-assigned this Jul 10, 2023

ainar-g added bug P2: High and removed waiting for data Waiting for users to provide more data. labels Jul 10, 2023

ainar-g added this to the v0.107.34 milestone Jul 10, 2023

adguard pushed a commit that referenced this issue Jul 10, 2023

Pull request 1912: 5896-safe-browsing-ptr

c02a141

Updates #5896. Squashed commit of the following: commit 4934054 Author: Ainar Garipov <A.Garipov@AdGuard.COM> Date: Mon Jul 10 17:59:45 2023 +0300 hashprefix: fix loop pointer

ainar-g mentioned this issue Jul 11, 2023

AdGuard thinks all requests are . domains? #5995

Closed

4 tasks

Freebase394 mentioned this issue Jul 12, 2023

adguardhome导致服务器崩溃或过载 #5998

Closed

4 tasks

ainar-g unpinned this issue Jul 12, 2023

ainar-g modified the milestones: v0.107.35, v0.107.34 Jul 12, 2023

ainar-g closed this as completed Jul 18, 2023

v0.107.31 stops responding to DNS requests, does not terminate #5896

v0.107.31 stops responding to DNS requests, does not terminate #5896

Comments

timkgh commented Jun 12, 2023

Prerequisites

Operating system type

CPU architecture

Installation

Setup

AdGuard Home version

Description

ainar-g commented Jun 13, 2023

timkgh commented Jun 15, 2023

poolski commented Jun 15, 2023

timkgh commented Jun 15, 2023 • edited Loading

copacetic82 commented Jun 17, 2023

copacetic82 commented Jun 17, 2023

ainar-g commented Jun 19, 2023

copacetic82 commented Jun 19, 2023 • edited Loading

Porygon31 commented Jun 22, 2023

TheWoodsy commented Jun 23, 2023

simonregn commented Jun 23, 2023

ainar-g commented Jun 29, 2023 • edited Loading

bartounet16 commented Jul 1, 2023

bigwookie commented Jul 4, 2023

bigwookie commented Jul 4, 2023

ainar-g commented Jul 4, 2023

bigwookie commented Jul 4, 2023

bigwookie commented Jul 5, 2023

erdmanbr commented Jul 5, 2023

bigwookie commented Jul 5, 2023

up2you1 commented Jul 5, 2023 • edited Loading

ainar-g commented Jul 10, 2023

ainar-g commented Jul 10, 2023

bigwookie commented Jul 10, 2023

ainar-g commented Jul 11, 2023

Freebase394 commented Jul 11, 2023

monsm commented Jul 12, 2023

bigwookie commented Jul 12, 2023

ainar-g commented Jul 12, 2023

Baxxy13 commented Jul 13, 2023

s1lviu commented Jul 15, 2023

TheCableGuy99 commented Jul 17, 2023

OlliZabal commented Jul 17, 2023

TheCableGuy99 commented Jul 17, 2023

bartounet16 commented Jul 17, 2023

timkgh commented Jul 17, 2023

bigwookie commented Jul 17, 2023

bartounet16 commented Jul 18, 2023

bartounet16 commented Jul 18, 2023

timkgh commented Jun 15, 2023 •

edited

Loading

copacetic82 commented Jun 19, 2023 •

edited

Loading

ainar-g commented Jun 29, 2023 •

edited

Loading

up2you1 commented Jul 5, 2023 •

edited

Loading