When there are multiple upstream servers, the memory of the adguardhome will gradually increase until it goes down

[6aink]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to report a bug and not ask a question or ask for help

• I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)

Platform (OS and CPU architecture)

Linux, AMD64 (aka x86_64)

Installation

GitHub releases or script from README

Setup

On one machine

AdGuard Home version

v0.108.0-a.762+388583ce

Action

When there are multiple upstream servers, the memory of the adguardhome will gradually increase until it goes down

Expected result

Memory no longer rises and can be used normally

Actual result

memory explosion

Additional information and/or screenshots

No response

[ainar-g]

Unfortunately, we cannot reproduce that. Can you provide the upstreams you're using?

[6aink]

You can try replicating it this way by turning off the cache and setting it to any number of upstream streams, with a request rate of 1000qps. At this point, you will see significant memory growth and the increased memory will not be reclaimed

[ainar-g]

We do not see this behaviour in our tests. The only thing that's growing are things like query log and statistics buffers and safe browsing/parental control hash caches.

If you do feel like there could be a resource leak, please collect the memory profile from your AGH by setting http.pprof.enabled to true and saving the contents of http://127.0.0.1:6060/debug/pprof/heap?debug=1.

[6aink]

https://raw.githubusercontent.com/6aink/render/master/debuglog.txt

[6aink]

this is debug log file

[6aink]

Memory leaks occur when parallel requests are used.

[duckxx]

这种情况出现在v0.108.0-b.50，有可能不是多个上游导致的，更新后，服务器4G内存在半个小时内只剩下10%的可用，CPU涨到了50%。并且只是更新，没有修改任何数据。

[6aink]

请问有什么办法避免这个问题吗

[duckxx]

请问有什么办法避免这个问题吗

在v0.108.0-b.49或以下版本未出现这种异常。可以尝试使用v0.108.0-b.49或以下版本。

[6aink]

I didn't find a memory leak with a single upstream. I suspect it has something to do with their new upstream feature in b50. Hopefully the adguardhome team can fix this memory leak.

[ainar-g]

Thank you for the profiles, we've been able to reproduce the goroutine leak.

@EugeneOne1, this is reproduceable using e.g. godnsbench -a udp://127.0.0.1:53 -c 100000 -q '{random}.example.com' with no cache and parallel requests for plain UDP upstreams. Please fix asap.

[6aink]

Okay, thank you very much.

[EugeneOne1]

@6aink, hello again. We've pushed an edge build that should fix this leak. Could you please try it out?

[6aink]

OK, wait a minute.

[6aink]

I probably observed for ten minutes, the memory has no upward trend, the problem has been solved, thank you.

[6aink]

I see that the memory is rising again, but not very fast. I will try to watch it for another hour.

[6aink]

I tested it for a few hours, and the memory stabilized at 300m, and there was no problem.

[EugeneOne1]

@6aink, thanks for testing! We'll include this to the following beta release.

[Adgbeta]

Edge build looks like fixed the issue so far, testing for over 4 hours, no increase of memory use.

[04bondgoods]

seems this issue still persists in 108.b51 version.
I encountered:

1. agh memory cost grew to ~ 2G (~50% of total mem of 4G) after roughly 1 month
2. after restarting agh 1 day ago, memory has grown to ~ 450MB (~12% of total memory) now
3. memory usage should be ~ 60MB normally